[arch-projects] [ABS] [PATCH 5/6] vcs prototypes: more efficient temp build directories

newer
[arch-projects] [ABS] [PATCH v2...

older
[arch-projects] [ABS] [PATCH 4/6]...

Linus Arver

3 Nov 2011 3 Nov '11

12:52 a.m.

The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on. We exclude this folder when creating a temporary build directory to save time and space. CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems. Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-) diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..." rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build" + cd ../"$_bzrmod-build" # # BUILD HERE diff --git a/prototypes/PKGBUILD-darcs.proto b/prototypes/PKGBUILD-darcs.proto index a55a346..affd2c8 100644 --- a/prototypes/PKGBUILD-darcs.proto +++ b/prototypes/PKGBUILD-darcs.proto @@ -45,8 +45,10 @@ build() { fi rm -rf "$_darcsmod-build" - cp -r "$_darcsmod" "$_darcsmod-build" - cd "$_darcsmod-build" + mkdir $_darcsmod-build + cd $_darcsmod + ls -A | grep -v _darcs | xargs -d '\n' cp -r -t ../"$_darcsmod-build" + cd ../"$_darcsmod-build" msg "Starting build..." diff --git a/prototypes/PKGBUILD-git.proto b/prototypes/PKGBUILD-git.proto index 88bc76b..75edb8b 100644 --- a/prototypes/PKGBUILD-git.proto +++ b/prototypes/PKGBUILD-git.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..." rm -rf "$_gitname-build" - git clone "$_gitname" "$_gitname-build" - cd "$_gitname-build" + mkdir $_gitname-build + cd $_gitname + ls -A | grep -v .git | xargs -d '\n' cp -r -t ../"$_gitname-build" + cd ../"$_gitname-build" # # BUILD HERE diff --git a/prototypes/PKGBUILD-hg.proto b/prototypes/PKGBUILD-hg.proto index 6f54af8..8927fbf 100644 --- a/prototypes/PKGBUILD-hg.proto +++ b/prototypes/PKGBUILD-hg.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..." rm -rf "$_hgrepo-build" - cp -r "$_hgrepo" "$_hgrepo-build" - cd "$_hgrepo-build" + mkdir $_hgrepo-build + cd $_hgrepo + ls -A | grep -v .hg | xargs -d '\n' cp -r -t ../"$_hgrepo-build" + cd ../"$_hgrepo-build" # # BUILD HERE -- 1.7.7.1

Show replies by date

Dave Reisner

3 Nov 3 Nov

1:03 a.m.

On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...

The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g. mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build" Note the proper quoting here, too, that you've stripped.

...

+ cd ../"$_bzrmod-build"

# # BUILD HERE diff --git a/prototypes/PKGBUILD-darcs.proto b/prototypes/PKGBUILD-darcs.proto index a55a346..affd2c8 100644 --- a/prototypes/PKGBUILD-darcs.proto +++ b/prototypes/PKGBUILD-darcs.proto @@ -45,8 +45,10 @@ build() { fi

rm -rf "$_darcsmod-build" - cp -r "$_darcsmod" "$_darcsmod-build" - cd "$_darcsmod-build" + mkdir $_darcsmod-build + cd $_darcsmod + ls -A | grep -v _darcs | xargs -d '\n' cp -r -t ../"$_darcsmod-build" + cd ../"$_darcsmod-build"

msg "Starting build..."

diff --git a/prototypes/PKGBUILD-git.proto b/prototypes/PKGBUILD-git.proto index 88bc76b..75edb8b 100644 --- a/prototypes/PKGBUILD-git.proto +++ b/prototypes/PKGBUILD-git.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_gitname-build" - git clone "$_gitname" "$_gitname-build" - cd "$_gitname-build" + mkdir $_gitname-build + cd $_gitname + ls -A | grep -v .git | xargs -d '\n' cp -r -t ../"$_gitname-build" + cd ../"$_gitname-build"

# # BUILD HERE diff --git a/prototypes/PKGBUILD-hg.proto b/prototypes/PKGBUILD-hg.proto index 6f54af8..8927fbf 100644 --- a/prototypes/PKGBUILD-hg.proto +++ b/prototypes/PKGBUILD-hg.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_hgrepo-build" - cp -r "$_hgrepo" "$_hgrepo-build" - cd "$_hgrepo-build" + mkdir $_hgrepo-build + cd $_hgrepo + ls -A | grep -v .hg | xargs -d '\n' cp -r -t ../"$_hgrepo-build" + cd ../"$_hgrepo-build"

# # BUILD HERE -- 1.7.7.1

Linus Arver

1:15 a.m.

On Wed, Nov 02, 2011 at 09:03:37PM -0400, Dave Reisner wrote:

...

On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...
The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g.

mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build"

Note the proper quoting here, too, that you've stripped.

Ah, thanks for that! I'll change it to your version.

Linus Arver

2:13 a.m.

On Wed, Nov 02, 2011 at 09:03:37PM -0400, Dave Reisner wrote:

...

On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...
The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g.

mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build"

Note the proper quoting here, too, that you've stripped.

Hmm, I just did some tests, but the "!(.bzr)" syntax that you used gives me a parse error.

Dave Reisner

2:21 a.m.

On Wed, Nov 02, 2011 at 07:13:48PM -0700, Linus Arver wrote:

...

On Wed, Nov 02, 2011 at 09:03:37PM -0400, Dave Reisner wrote:

...
On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...
The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g.

mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build"

Note the proper quoting here, too, that you've stripped.

Hmm, I just did some tests, but the "!(.bzr)" syntax that you used gives me a parse error.

Yes, it's a syntax error when extglob isn't enabled. We provide this by default in makepkg, so PKGBUILDs will inherit it.

Linus Arver

2:35 a.m.

On Wed, Nov 02, 2011 at 10:21:04PM -0400, Dave Reisner wrote:

...

On Wed, Nov 02, 2011 at 07:13:48PM -0700, Linus Arver wrote:

...
On Wed, Nov 02, 2011 at 09:03:37PM -0400, Dave Reisner wrote:

...
On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...
The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g.

mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build"

Note the proper quoting here, too, that you've stripped.

Hmm, I just did some tests, but the "!(.bzr)" syntax that you used gives me a parse error.

Yes, it's a syntax error when extglob isn't enabled. We provide this by default in makepkg, so PKGBUILDs will inherit it.

Strange -- the test I did was from inside a PKGBUILD (and I just did a "pacman --Syu" an hour ago). Perhaps there's a bug in pacman/scripts/makepkg.sh.in. I skimmed the code in there and there's a bit of shell option wizardry going on at different points. I guess I'll file a bug report for pacman... Meanwhile, I'll use the "!(foo)" syntax, which is certainly cleaner and much more elegant.

Dave Reisner

2:45 a.m.

On Wed, Nov 02, 2011 at 07:35:40PM -0700, Linus Arver wrote:

...

On Wed, Nov 02, 2011 at 10:21:04PM -0400, Dave Reisner wrote:

...
On Wed, Nov 02, 2011 at 07:13:48PM -0700, Linus Arver wrote:

...
On Wed, Nov 02, 2011 at 09:03:37PM -0400, Dave Reisner wrote:

...
On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...
The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g.

mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build"

Note the proper quoting here, too, that you've stripped.

Hmm, I just did some tests, but the "!(.bzr)" syntax that you used gives me a parse error.

Yes, it's a syntax error when extglob isn't enabled. We provide this by default in makepkg, so PKGBUILDs will inherit it.

Strange -- the test I did was from inside a PKGBUILD (and I just did a "pacman --Syu" an hour ago).

Sorry, I should have been clearer. It's in testing. Line 84 if you're looking at makepkg.sh.in in the git repo.

...

Perhaps there's a bug in pacman/scripts/makepkg.sh.in. I skimmed the code in there and there's a bit of shell option wizardry going on at different points.

I guess I'll file a bug report for pacman... Meanwhile, I'll use the

No. There is no bug.

...

"!(foo)" syntax, which is certainly cleaner and much more elegant.

Linus Arver

2:47 a.m.

On Wed, Nov 02, 2011 at 10:45:05PM -0400, Dave Reisner wrote:

...

On Wed, Nov 02, 2011 at 07:35:40PM -0700, Linus Arver wrote:

...
On Wed, Nov 02, 2011 at 10:21:04PM -0400, Dave Reisner wrote:

...
On Wed, Nov 02, 2011 at 07:13:48PM -0700, Linus Arver wrote:

...
On Wed, Nov 02, 2011 at 09:03:37PM -0400, Dave Reisner wrote:

...
On Wed, Nov 02, 2011 at 05:52:37PM -0700, Linus Arver wrote:

...
The bzr, darcs, git, and hg version control systems use a single internal folder at the root to store all VCS-related data (commits, history, etc.). This folder can get quite large (hundreds of MiB) for big projects and continues to grow as the project lives on.

We exclude this folder when creating a temporary build directory to save time and space.

CVS and SVN pollute the source repo with "CVS" and ".svn" directories recursively for every single directory, so there is no simple one-liner solution to exclude the VCS data for these systems.

Signed-off-by: Linus Arver --- prototypes/PKGBUILD-bzr.proto | 6 ++++-- prototypes/PKGBUILD-darcs.proto | 6 ++++-- prototypes/PKGBUILD-git.proto | 6 ++++-- prototypes/PKGBUILD-hg.proto | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/prototypes/PKGBUILD-bzr.proto b/prototypes/PKGBUILD-bzr.proto index a01eea8..ef7187d 100644 --- a/prototypes/PKGBUILD-bzr.proto +++ b/prototypes/PKGBUILD-bzr.proto @@ -47,8 +47,10 @@ build() { msg "Starting build..."

rm -rf "$_bzrmod-build" - cp -r "$_bzrmod" "$_bzrmod-build" - cd "$_bzrmod-build" + mkdir $_bzrmod-build + cd $_bzrmod + ls -A | grep -v .bzr | xargs -d '\n' cp -r -t ../"$_bzrmod-build"

Oh hell no. We are not parsing ls here. This can be done without the need for grep and findutils as well, e.g.

mkdir "$_bzrmod-build" cp -a "$_bzrmod"/!(.bzr) "$_bzrmod-build"

Note the proper quoting here, too, that you've stripped.

Hmm, I just did some tests, but the "!(.bzr)" syntax that you used gives me a parse error.

Yes, it's a syntax error when extglob isn't enabled. We provide this by default in makepkg, so PKGBUILDs will inherit it.

Strange -- the test I did was from inside a PKGBUILD (and I just did a "pacman --Syu" an hour ago).

Sorry, I should have been clearer. It's in testing. Line 84 if you're looking at makepkg.sh.in in the git repo.

...
Perhaps there's a bug in pacman/scripts/makepkg.sh.in. I skimmed the code in there and there's a bit of shell option wizardry going on at different points.

I guess I'll file a bug report for pacman... Meanwhile, I'll use the

No. There is no bug.

...
"!(foo)" syntax, which is certainly cleaner and much more elegant.

Ah, that makes sense. Yes, I was looking at line 84 myself and scratching my head. Thanks. -Linus

4566

Age (days ago)

4566

Last active (days ago)

List overview

Download

7 comments

2 participants

participants (2)

Dave Reisner
Linus Arver