[aur-general] [PATCH] initial support for any architecture.
This patch adds initial support for any architecture. It assumes that there'll be a repo-any directory in repos/ from which it'll pull the PKGBUILDs. Also the architectureo independent packages are assumed to be in $reponame/os/any. This commit does not fix anything going wrong in the cron-jobs or misc-scripts folder. Also this commit contain any helper scripts for the any architecture corresponding to db-core or testing2x. Signed-off-by: Abhishek Dasgupta <abhidg@gmail.com> --- db-move | 48 ++++++++++++++++++++++++---------------- db-remove | 33 +++++++++++++++++---------- db-update | 72 +++++++++++++++++++++++++++++++++++++++++++++--------------- 3 files changed, 104 insertions(+), 49 deletions(-) diff --git a/db-move b/db-move index ff9677b..aac0e4e 100755 --- a/db-move +++ b/db-move @@ -1,6 +1,27 @@ #!/bin/bash # Originally from Pierre's testing2extra script +process_db_move() { + + # copy the db file into our working area + if [ -f "$ftppath_from/$1/$repofrom.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_from/$1/$repofrom.db.tar.$DB_COMPRESSION" . + /usr/bin/repo-remove -q "$repofrom.db.tar.$DB_COMPRESSION" $packagename || die "Error in repo-remove" + #use '*' to move the old DB too + mv $repofrom.db.tar.$DB_COMPRESSION* $ftppath_from/$1 + echo " Package files will be cleaned up automatically" + fi + + if [ -f "$ftppath_to/$1/$repoto.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_to/$1/$repoto.db.tar.$DB_COMPRESSION" . + fi + + /bin/cp "$ftppath_from/$1/$_pkgfile" . + /usr/bin/repo-add -q "$repoto.db.tar.$DB_COMPRESSION" $_pkgfile || die "Error in repo-add" + #use '*' to move the old DB too + mv $repoto.db.tar.$DB_COMPRESSION* $_pkgfile $ftppath_to/$1 + +} if [ $# -ne 4 ]; then echo "usage: $(basename $0) <pkgname> <repo-from> <repo-to> <arch>" exit 1 @@ -17,8 +38,8 @@ _arch="$4" export CARCH="$_arch" -ftppath_from="$FTP_BASE/$repofrom/os/$_arch/" -ftppath_to="$FTP_BASE/$repoto/os/$_arch/" +ftppath_from="$FTP_BASE/$repofrom/os/" +ftppath_to="$FTP_BASE/$repoto/os/" svnrepo_from="$repofrom-$_arch" svnrepo_to="$repoto-$_arch" @@ -62,7 +83,7 @@ if [ -d "$packagename/repos/$svnrepo_from" ]; then . "$packagename/repos/$svnrepo_from/$BUILDSCRIPT" _pkgfile="$pkgname-$pkgver-$pkgrel-$_arch$PKGEXT" - if [ ! -f "$ftppath_from/$_pkgfile" ]; then + if [ ! -f "$ftppath_from/$_arch/$_pkgfile" ]; then die "error: package file '$_pkgfile' not found in repo '$repofrom'" fi @@ -81,23 +102,12 @@ if [ -d "$packagename/repos/$svnrepo_from" ]; then [ -d build/ ] || mkdir build cd build/ - # copy the db file into our working area - if [ -f "$ftppath_from/$repofrom.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath_from/$repofrom.db.tar.$DB_COMPRESSION" . - /usr/bin/repo-remove -q "$repofrom.db.tar.$DB_COMPRESSION" $packagename || die "Error in repo-remove" - #use '*' to move the old DB too - mv $repofrom.db.tar.$DB_COMPRESSION* $ftppath_from - echo " Package files will be cleaned up automatically" + if [ "$_arch" == "any" ]; then + process_db_move i686 + process_db_move x86_64 + else + process_db_move $_arch fi - - if [ -f "$ftppath_to/$repoto.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath_to/$repoto.db.tar.$DB_COMPRESSION" . - fi - - /bin/cp "$ftppath_from/$_pkgfile" . - /usr/bin/repo-add -q "$repoto.db.tar.$DB_COMPRESSION" $_pkgfile || die "Error in repo-add" - #use '*' to move the old DB too - mv $repoto.db.tar.$DB_COMPRESSION* $_pkgfile $ftppath_to else die "Error: $packagename is not in repo $repofrom" fi diff --git a/db-remove b/db-remove index acc849d..6432d94 100755 --- a/db-remove +++ b/db-remove @@ -1,5 +1,21 @@ #!/bin/bash +process_db_files() { + # copy the db file into our working area + if [ -f "$ftppath/$1/$reponame.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath/$1/$reponame.db.tar.$DB_COMPRESSION" build/ + else + echo "No database found at '$ftpdir', nothing more to do" + exit 0 + fi + + echo " Removing from $reponame DB file" + cd build/ + /usr/bin/repo-remove -q "$reponame.db.tar.$DB_COMPRESSION" $packagename + + /bin/cp "$reponame.db.tar.$DB_COMPRESSION" "$ftppath/$1/" +} + if [ $# -ne 3 ]; then echo "usage: $(basename $0) <pkgname> <reponame> <arch>" exit 1 @@ -15,7 +31,7 @@ arch="$3" export CARCH="$arch" -ftppath="$FTP_BASE/$reponame/os/$arch/" +ftppath="$FTP_BASE/$reponame/os/" svnrepo="$reponame-$arch" [ "$UID" = "" ] && UID=$(uid) @@ -64,20 +80,13 @@ fi cd "$WORKDIR" [ -d build/ ] || mkdir build -# copy the db file into our working area -if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +if [ "$arch" == "any" ]; then + process_db_files i686 + process_db_files x86_64 else - echo "No database found at '$ftpdir', nothing more to do" - exit 0 + process_db_files $arch fi -echo " Removing from $reponame DB file" -cd build/ -/usr/bin/repo-remove -q "$reponame.db.tar.$DB_COMPRESSION" $packagename - -/bin/cp "$reponame.db.tar.$DB_COMPRESSION" "$ftppath/" - echo "Package files will be cleaned up automatically" cleanup diff --git a/db-update b/db-update index 77af39f..f51f470 100755 --- a/db-update +++ b/db-update @@ -15,6 +15,7 @@ _arch="$2" export CARCH="$_arch" ftppath="$FTP_BASE/$reponame/os/$_arch/" +ftppath_base="$FTP_BASE/$reponame/os/" svnrepo="$reponame-$_arch" stagedir="$STAGING/$reponame" @@ -72,10 +73,21 @@ repo_lock $reponame $_arch cd "$WORKDIR" # copy the db file into our working area -if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +if [ "$_arch" == "any" ]; then + for architecture in i686 x86_64; do + if [ -f "$ftppath_base/$architecture/$reponame.db.tar.gz.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_base/$architecture/$reponame.db.tar.gz.$DB_COMPRESSION" build/$reponame-i686.db.tar.gz + else + touch "build/$reponame-$architecture.db.tar.$DB_COMPRESSION" + fi + done +else + if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ + else + touch "build/$reponame.db.tar.$DB_COMPRESSION" + fi fi - echo "Updating DB for $reponame $_arch" if [ -d "${stagedir}64" ]; then @@ -106,11 +118,11 @@ fi if [ -n "$ADDPKGS" ]; then echo "==> Copying DB file from '$reponame'..." >&2 - if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ - else - touch "build/$reponame.db.tar.$DB_COMPRESSION" - fi +# if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then +# /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +# else +# touch "build/$reponame.db.tar.$DB_COMPRESSION" +# fi echo "==> Processing new/updated packages for repository '$reponame'..." >&2 @@ -147,10 +159,19 @@ if [ -n "$ADDPKGS" ]; then pkgs="" for pkg in $to_add; do pkgs="$pkgs $(basename $pkg)"; done - - /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs + if [ "$_arch" == "any" ]; then + /usr/bin/repo-add -q "$reponame-i686.db.tar.$DB_COMPRESSION" $pkgs + /usr/bin/repo-add -q "$reponame-x86_64.db.tar.$DB_COMPRESSION" $pkgs + else + /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs + fi else - rm -f "build/$reponame.db.tar.$DB_COMPRESSION" + if [ "$_arch" == "any" ]; then + rm -f "build/$reponame-i686.db.tar.gz" + rm -f "build/$reponame-x86_64.db.tar.gz" + else + rm -f "build/$reponame.db.tar.$DB_COMPRESSION" + fi echo "Errors found when adding packages" fi else @@ -159,13 +180,28 @@ fi # if non empty, move all build dirs if [ $(/bin/ls "$WORKDIR/build/" 2>/dev/null | wc -l) != 0 ]; then - echo "Copying new files to '$ftppath'" - for f in "$WORKDIR/build/"*; do - if ! /bin/cp "$f" "$ftppath"; then - die "error: failure while copying files to $ftppath" - fi - done - + if [ "$_arch" != "any" ]; then + echo "Copying new files to '$ftppath'" + for f in "$WORKDIR/build/"*; do + if ! /bin/cp "$f" "$ftppath"; then + die "error: failure while copying files to $ftppath" + fi + done + else + echo "Copying architecture-independent packages..." + for f in "$WORKDIR/build/"*.pkg.tar.gz; do + if ! /bin/cp "$f" "$ftppath"; then + die "error: failure while copying $f to $ftppath" + fi + # Make link to arch-any package from os/{i686,x86_64} directories + ln -s "$ftppath/$f" "$ftppath_base/i686/" + ln -s "$ftppath/$f" "$ftppath_base/x86_64/" + for architecture in i686 x86_64; do + if ! /bin/cp "$reponame-$architecture.db.tar.$DB_COMPRESSION" "$ftppath_base/$architecture/$reponame.db.tar.$DB_COMPRESSION"; then + die "error: failed to transfer $architecture repository database." + fi + done + fi else echo "Nothing to copy, no work done" fi -- 1.6.2 -- Abhishek Dasgupta <http://abhidg.mine.nu> GPG 67972DOF pgp.mit.edu
2009/3/11 Abhishek Dasgupta <abhidg@gmail.com>:
@@ -106,11 +118,11 @@ fi if [ -n "$ADDPKGS" ]; then
echo "==> Copying DB file from '$reponame'..." >&2 - if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ - else - touch "build/$reponame.db.tar.$DB_COMPRESSION" - fi +# if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then +# /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +# else +# touch "build/$reponame.db.tar.$DB_COMPRESSION" +# fi
Clarification: The part of the code I've commented out seemed to have been already invoked at the beginning of the script itself. Also, in devtools, there are not many changes needed. In 'commitpkg' I had to add the following lines after source PKGBUILD at line 28: if [ "$arch" == "any" ]; then CARCH="$arch" fi Other than that (I've not checked makechrootpkg and mkarchroot), devtools seems to be ready for any-architecture. -- Abhishek
I've also written a script, convert-to-any which takes the name of a package like this: convert-to-any community/pitivi and converts it into an arch-independent package. It also updates the repository databases accordingly, creating the 'any' folder in the repository if needed. This script requires dbscripts. However, this script does not make the repo-any dir in SVN. #!/bin/bash # Converts an existing package into an # architecture-independent package and updates # the repositories accordingly. # -- Abhishek Dasgupta <abhidg@gmail.com> . "$(dirname $0)/db-functions" [ "$UID" = "" ] && UID=$(uid) if [ $# -ne 1 ]; then echo "Syntax: $(basename $0) <reponame/pkgname>" exit 1 fi repo=$(echo $1 | sed "s#\(.*\)/.*#\1#g"N pkg=$(echo $1 | sed "s#.*/\(.*\)#\1#g") if [ -f "$(dirname $0)/config" ]; then . "$(dirname $0)/config" else TMPDIR=/srv/tmp FTP_BASE=/srv/ftp fi if [ -f /etc/makepkg.conf ]; then . /etc/makepkg.conf else echo "W: /etc/makepkg.conf does not exist." DB_COMPRESSION=gz PKGEXT=".pkg.tar.gz" fi repo_lock $repo any WORKDIR="$TMPDIR/convert-to-any.$pkg.$UID" ftppath="$FTP_BASE/$repo/os" cleanup() { trap '' 0 2 # unlock repo_unlock $repo any rm -rf "$WORKDIR" [ "$1" ] && exit $1 } ctrl_c() { echo "Interrupted" >&2 cleanup 0 } die() { echo "$*" >&2 cleanup 1 } # Enter the temporary build directory # and convert the i686 package into an # architecture-independent package. mkdir -p "$WORKDIR/build"; cd "$WORKDIR/build" oldpkgname=$ftppath/i686/$pkg* if [ -f "$oldpkgname" ]; then cp "$oldpkgname" . else die "E: Package $oldpkgname not found in $ftppath/i686" fi for architecture in i686 x86_64; do if [ -f "$ftppath/$architecture/$repo.db.tar.$DB_COMPRESSION" ]; then cp $ftppath/$architecture/$repo.db.tar.$DB_COMPRESSION \ $repo-$architecture.db.tar.$DB_COMPRESSION else touch $repo-$architecture.db.tar.$DB_COMPRESSION fi done if [ ! -d "$ftppath/any" ]; then mkdir -p "$ftppath/any"; fi # Conversion of i686 package into "any" package. echo -n "Extracting $pkg..." mkdir -p package tar zxf $pkg*i686$PKGEXT -C package echo " done." sed -i "s/arch = i686/arch = any/g" package/.PKGINFO newpkgname=$(ls $pkg*i686$PKGEXT | sed "s/i686/any/g") cd package; tar czf "$newpkgname" * .PKGINFO cd ..; mv "package/$newpkgname" . echo "Created $newpkgname." # New package is ready, move it into place and update db. mv "$newpkgname" "$ftppath/any/" for architecture in i686 x86_64; do # rm -f $ftppath/$architecture/$pkg*$PKGEXT ln -s "$ftppath/any/$newpkgname" "$ftppath/$architecture/$newpkgname" repo-remove -q $repo-$architecture.db.tar.$DB_COMPRESSION $pkg repo-add -q $repo-$architecture.db.tar.$DB_COMPRESSION $newpkgname mv $repo-$architecture.db.tar.$DB_COMPRESSION "$ftppath/os/$architecture" echo "Updated $repo-$architecture for $pkg." done repo_unlock echo -n "Cleaning up..." cd ../.. cleanup echo " done." -- Abhishek
done repo_unlock Sorry for the typo, this repo_unlock should not be there. -- Abhishek
2009/3/11 Abhishek Dasgupta <abhidg@gmail.com>:
done repo_unlock
Sorry for the typo, this repo_unlock should not be there.
-- Abhishek
Pretty sure attached diffs would be more appreciated :)
On Wed, Mar 11, 2009 at 10:14 AM, Phil Dillon-Thiselton <dibblethewrecker@gmail.com> wrote:
2009/3/11 Abhishek Dasgupta <abhidg@gmail.com>:
done repo_unlock
Sorry for the typo, this repo_unlock should not be there.
-- Abhishek
Pretty sure attached diffs would be more appreciated :)
Nope. Inline is preferred as it lets us more easily comment on the patch via email
On Wed, Mar 11, 2009 at 10:24:17AM -0500, Aaron Griffin wrote:
On Wed, Mar 11, 2009 at 10:14 AM, Phil Dillon-Thiselton <dibblethewrecker@gmail.com> wrote:
2009/3/11 Abhishek Dasgupta <abhidg@gmail.com>:
done repo_unlock
Sorry for the typo, this repo_unlock should not be there.
-- Abhishek
Pretty sure attached diffs would be more appreciated :)
Nope. Inline is preferred as it lets us more easily comment on the patch via email
Well, you can always have inline attachments. :D
On Wed, Mar 11, 2009 at 10:10 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
I've also written a script, convert-to-any which takes the name of a package like this: convert-to-any community/pitivi and converts it into an arch-independent package. It also updates the repository databases accordingly, creating the 'any' folder in the repository if needed. This script requires dbscripts. However, this script does not make the repo-any dir in SVN.
Committed to my any-arch dbscripts branch. Thanks
On Wed, Mar 11, 2009 at 8:27 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
This patch adds initial support for any architecture. It assumes that there'll be a repo-any directory in repos/ from which it'll pull the PKGBUILDs. Also the architectureo independent packages are assumed to be in $reponame/os/any.
This commit does not fix anything going wrong in the cron-jobs or misc-scripts folder. Also this commit contain any helper scripts for the any architecture corresponding to db-core or testing2x.
Signed-off-by: Abhishek Dasgupta <abhidg@gmail.com> --- db-move | 48 ++++++++++++++++++++++++---------------- db-remove | 33 +++++++++++++++++---------- db-update | 72 +++++++++++++++++++++++++++++++++++++++++++++--------------- 3 files changed, 104 insertions(+), 49 deletions(-)
diff --git a/db-move b/db-move index ff9677b..aac0e4e 100755 --- a/db-move +++ b/db-move @@ -1,6 +1,27 @@ #!/bin/bash # Originally from Pierre's testing2extra script
+process_db_move() { + + # copy the db file into our working area + if [ -f "$ftppath_from/$1/$repofrom.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_from/$1/$repofrom.db.tar.$DB_COMPRESSION" . + /usr/bin/repo-remove -q "$repofrom.db.tar.$DB_COMPRESSION" $packagename || die "Error in repo-remove" + #use '*' to move the old DB too + mv $repofrom.db.tar.$DB_COMPRESSION* $ftppath_from/$1 + echo " Package files will be cleaned up automatically" + fi + + if [ -f "$ftppath_to/$1/$repoto.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_to/$1/$repoto.db.tar.$DB_COMPRESSION" . + fi + + /bin/cp "$ftppath_from/$1/$_pkgfile" . + /usr/bin/repo-add -q "$repoto.db.tar.$DB_COMPRESSION" $_pkgfile || die "Error in repo-add" + #use '*' to move the old DB too + mv $repoto.db.tar.$DB_COMPRESSION* $_pkgfile $ftppath_to/$1 + +} if [ $# -ne 4 ]; then echo "usage: $(basename $0) <pkgname> <repo-from> <repo-to> <arch>" exit 1 @@ -17,8 +38,8 @@ _arch="$4"
export CARCH="$_arch"
-ftppath_from="$FTP_BASE/$repofrom/os/$_arch/" -ftppath_to="$FTP_BASE/$repoto/os/$_arch/" +ftppath_from="$FTP_BASE/$repofrom/os/" +ftppath_to="$FTP_BASE/$repoto/os/" svnrepo_from="$repofrom-$_arch" svnrepo_to="$repoto-$_arch"
@@ -62,7 +83,7 @@ if [ -d "$packagename/repos/$svnrepo_from" ]; then . "$packagename/repos/$svnrepo_from/$BUILDSCRIPT" _pkgfile="$pkgname-$pkgver-$pkgrel-$_arch$PKGEXT"
- if [ ! -f "$ftppath_from/$_pkgfile" ]; then + if [ ! -f "$ftppath_from/$_arch/$_pkgfile" ]; then die "error: package file '$_pkgfile' not found in repo '$repofrom'" fi
@@ -81,23 +102,12 @@ if [ -d "$packagename/repos/$svnrepo_from" ]; then [ -d build/ ] || mkdir build cd build/
- # copy the db file into our working area - if [ -f "$ftppath_from/$repofrom.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath_from/$repofrom.db.tar.$DB_COMPRESSION" . - /usr/bin/repo-remove -q "$repofrom.db.tar.$DB_COMPRESSION" $packagename || die "Error in repo-remove" - #use '*' to move the old DB too - mv $repofrom.db.tar.$DB_COMPRESSION* $ftppath_from - echo " Package files will be cleaned up automatically" + if [ "$_arch" == "any" ]; then + process_db_move i686 + process_db_move x86_64 + else + process_db_move $_arch fi - - if [ -f "$ftppath_to/$repoto.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath_to/$repoto.db.tar.$DB_COMPRESSION" . - fi - - /bin/cp "$ftppath_from/$_pkgfile" . - /usr/bin/repo-add -q "$repoto.db.tar.$DB_COMPRESSION" $_pkgfile || die "Error in repo-add" - #use '*' to move the old DB too - mv $repoto.db.tar.$DB_COMPRESSION* $_pkgfile $ftppath_to else die "Error: $packagename is not in repo $repofrom" fi diff --git a/db-remove b/db-remove index acc849d..6432d94 100755 --- a/db-remove +++ b/db-remove @@ -1,5 +1,21 @@ #!/bin/bash
+process_db_files() { + # copy the db file into our working area + if [ -f "$ftppath/$1/$reponame.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath/$1/$reponame.db.tar.$DB_COMPRESSION" build/ + else + echo "No database found at '$ftpdir', nothing more to do" + exit 0 + fi + + echo " Removing from $reponame DB file" + cd build/ + /usr/bin/repo-remove -q "$reponame.db.tar.$DB_COMPRESSION" $packagename + + /bin/cp "$reponame.db.tar.$DB_COMPRESSION" "$ftppath/$1/" +} + if [ $# -ne 3 ]; then echo "usage: $(basename $0) <pkgname> <reponame> <arch>" exit 1 @@ -15,7 +31,7 @@ arch="$3"
export CARCH="$arch"
-ftppath="$FTP_BASE/$reponame/os/$arch/" +ftppath="$FTP_BASE/$reponame/os/" svnrepo="$reponame-$arch"
[ "$UID" = "" ] && UID=$(uid) @@ -64,20 +80,13 @@ fi cd "$WORKDIR" [ -d build/ ] || mkdir build
-# copy the db file into our working area -if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +if [ "$arch" == "any" ]; then + process_db_files i686 + process_db_files x86_64 else - echo "No database found at '$ftpdir', nothing more to do" - exit 0 + process_db_files $arch fi
-echo " Removing from $reponame DB file" -cd build/ -/usr/bin/repo-remove -q "$reponame.db.tar.$DB_COMPRESSION" $packagename - -/bin/cp "$reponame.db.tar.$DB_COMPRESSION" "$ftppath/" - echo "Package files will be cleaned up automatically"
cleanup diff --git a/db-update b/db-update index 77af39f..f51f470 100755 --- a/db-update +++ b/db-update @@ -15,6 +15,7 @@ _arch="$2" export CARCH="$_arch"
ftppath="$FTP_BASE/$reponame/os/$_arch/" +ftppath_base="$FTP_BASE/$reponame/os/" svnrepo="$reponame-$_arch" stagedir="$STAGING/$reponame"
@@ -72,10 +73,21 @@ repo_lock $reponame $_arch cd "$WORKDIR"
# copy the db file into our working area -if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +if [ "$_arch" == "any" ]; then + for architecture in i686 x86_64; do + if [ -f "$ftppath_base/$architecture/$reponame.db.tar.gz.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_base/$architecture/$reponame.db.tar.gz.$DB_COMPRESSION" build/$reponame-i686.db.tar.gz + else + touch "build/$reponame-$architecture.db.tar.$DB_COMPRESSION" + fi + done +else + if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ + else + touch "build/$reponame.db.tar.$DB_COMPRESSION" + fi fi - echo "Updating DB for $reponame $_arch"
if [ -d "${stagedir}64" ]; then @@ -106,11 +118,11 @@ fi if [ -n "$ADDPKGS" ]; then
echo "==> Copying DB file from '$reponame'..." >&2 - if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ - else - touch "build/$reponame.db.tar.$DB_COMPRESSION" - fi +# if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then +# /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +# else +# touch "build/$reponame.db.tar.$DB_COMPRESSION" +# fi
echo "==> Processing new/updated packages for repository '$reponame'..." >&2
@@ -147,10 +159,19 @@ if [ -n "$ADDPKGS" ]; then
pkgs="" for pkg in $to_add; do pkgs="$pkgs $(basename $pkg)"; done - - /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs + if [ "$_arch" == "any" ]; then + /usr/bin/repo-add -q "$reponame-i686.db.tar.$DB_COMPRESSION" $pkgs + /usr/bin/repo-add -q "$reponame-x86_64.db.tar.$DB_COMPRESSION" $pkgs + else + /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs + fi else - rm -f "build/$reponame.db.tar.$DB_COMPRESSION" + if [ "$_arch" == "any" ]; then + rm -f "build/$reponame-i686.db.tar.gz" + rm -f "build/$reponame-x86_64.db.tar.gz" + else + rm -f "build/$reponame.db.tar.$DB_COMPRESSION" + fi echo "Errors found when adding packages" fi else @@ -159,13 +180,28 @@ fi
# if non empty, move all build dirs if [ $(/bin/ls "$WORKDIR/build/" 2>/dev/null | wc -l) != 0 ]; then - echo "Copying new files to '$ftppath'" - for f in "$WORKDIR/build/"*; do - if ! /bin/cp "$f" "$ftppath"; then - die "error: failure while copying files to $ftppath" - fi - done - + if [ "$_arch" != "any" ]; then + echo "Copying new files to '$ftppath'" + for f in "$WORKDIR/build/"*; do + if ! /bin/cp "$f" "$ftppath"; then + die "error: failure while copying files to $ftppath" + fi + done + else + echo "Copying architecture-independent packages..." + for f in "$WORKDIR/build/"*.pkg.tar.gz; do + if ! /bin/cp "$f" "$ftppath"; then + die "error: failure while copying $f to $ftppath" + fi + # Make link to arch-any package from os/{i686,x86_64} directories + ln -s "$ftppath/$f" "$ftppath_base/i686/" + ln -s "$ftppath/$f" "$ftppath_base/x86_64/" + for architecture in i686 x86_64; do + if ! /bin/cp "$reponame-$architecture.db.tar.$DB_COMPRESSION" "$ftppath_base/$architecture/$reponame.db.tar.$DB_COMPRESSION"; then + die "error: failed to transfer $architecture repository database." + fi + done + fi else echo "Nothing to copy, no work done" fi
-- Abhishek Dasgupta <http://abhidg.mine.nu> GPG 67972DOF pgp.mit.edu
Hmm the aur-general list is a weird place for this patch :) The only thing that makes me hesitant about this is that I was going a slightly different route. This patch here: http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=any-arch&id=1ef058dad91a5ca82db2744776251a2ab68bcf90 changes the dbscripts to loop over all architectures for each run. I wanted to simply add "any" into the file detection part, but not delete them until the very end So this way, if I have 3 packages in my staging dir: foo(i686), bar(x86_64), and baz(any), running db-update does the following: for A in ARCHES (where ARCHES=(i686 x86_64)): get all files with "A" arch _or_ "any" arch do loop as normal delete all files we processed correctly
On Wed, Mar 11, 2009 at 10:32 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Wed, Mar 11, 2009 at 8:27 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
This patch adds initial support for any architecture. It assumes that there'll be a repo-any directory in repos/ from which it'll pull the PKGBUILDs. Also the architectureo independent packages are assumed to be in $reponame/os/any.
This commit does not fix anything going wrong in the cron-jobs or misc-scripts folder. Also this commit contain any helper scripts for the any architecture corresponding to db-core or testing2x.
Signed-off-by: Abhishek Dasgupta <abhidg@gmail.com> --- db-move | 48 ++++++++++++++++++++++++---------------- db-remove | 33 +++++++++++++++++---------- db-update | 72 +++++++++++++++++++++++++++++++++++++++++++++--------------- 3 files changed, 104 insertions(+), 49 deletions(-)
diff --git a/db-move b/db-move index ff9677b..aac0e4e 100755 --- a/db-move +++ b/db-move @@ -1,6 +1,27 @@ #!/bin/bash # Originally from Pierre's testing2extra script
+process_db_move() { + + # copy the db file into our working area + if [ -f "$ftppath_from/$1/$repofrom.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_from/$1/$repofrom.db.tar.$DB_COMPRESSION" . + /usr/bin/repo-remove -q "$repofrom.db.tar.$DB_COMPRESSION" $packagename || die "Error in repo-remove" + #use '*' to move the old DB too + mv $repofrom.db.tar.$DB_COMPRESSION* $ftppath_from/$1 + echo " Package files will be cleaned up automatically" + fi + + if [ -f "$ftppath_to/$1/$repoto.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_to/$1/$repoto.db.tar.$DB_COMPRESSION" . + fi + + /bin/cp "$ftppath_from/$1/$_pkgfile" . + /usr/bin/repo-add -q "$repoto.db.tar.$DB_COMPRESSION" $_pkgfile || die "Error in repo-add" + #use '*' to move the old DB too + mv $repoto.db.tar.$DB_COMPRESSION* $_pkgfile $ftppath_to/$1 + +} if [ $# -ne 4 ]; then echo "usage: $(basename $0) <pkgname> <repo-from> <repo-to> <arch>" exit 1 @@ -17,8 +38,8 @@ _arch="$4"
export CARCH="$_arch"
-ftppath_from="$FTP_BASE/$repofrom/os/$_arch/" -ftppath_to="$FTP_BASE/$repoto/os/$_arch/" +ftppath_from="$FTP_BASE/$repofrom/os/" +ftppath_to="$FTP_BASE/$repoto/os/" svnrepo_from="$repofrom-$_arch" svnrepo_to="$repoto-$_arch"
@@ -62,7 +83,7 @@ if [ -d "$packagename/repos/$svnrepo_from" ]; then . "$packagename/repos/$svnrepo_from/$BUILDSCRIPT" _pkgfile="$pkgname-$pkgver-$pkgrel-$_arch$PKGEXT"
- if [ ! -f "$ftppath_from/$_pkgfile" ]; then + if [ ! -f "$ftppath_from/$_arch/$_pkgfile" ]; then die "error: package file '$_pkgfile' not found in repo '$repofrom'" fi
@@ -81,23 +102,12 @@ if [ -d "$packagename/repos/$svnrepo_from" ]; then [ -d build/ ] || mkdir build cd build/
- # copy the db file into our working area - if [ -f "$ftppath_from/$repofrom.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath_from/$repofrom.db.tar.$DB_COMPRESSION" . - /usr/bin/repo-remove -q "$repofrom.db.tar.$DB_COMPRESSION" $packagename || die "Error in repo-remove" - #use '*' to move the old DB too - mv $repofrom.db.tar.$DB_COMPRESSION* $ftppath_from - echo " Package files will be cleaned up automatically" + if [ "$_arch" == "any" ]; then + process_db_move i686 + process_db_move x86_64 + else + process_db_move $_arch fi - - if [ -f "$ftppath_to/$repoto.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath_to/$repoto.db.tar.$DB_COMPRESSION" . - fi - - /bin/cp "$ftppath_from/$_pkgfile" . - /usr/bin/repo-add -q "$repoto.db.tar.$DB_COMPRESSION" $_pkgfile || die "Error in repo-add" - #use '*' to move the old DB too - mv $repoto.db.tar.$DB_COMPRESSION* $_pkgfile $ftppath_to else die "Error: $packagename is not in repo $repofrom" fi diff --git a/db-remove b/db-remove index acc849d..6432d94 100755 --- a/db-remove +++ b/db-remove @@ -1,5 +1,21 @@ #!/bin/bash
+process_db_files() { + # copy the db file into our working area + if [ -f "$ftppath/$1/$reponame.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath/$1/$reponame.db.tar.$DB_COMPRESSION" build/ + else + echo "No database found at '$ftpdir', nothing more to do" + exit 0 + fi + + echo " Removing from $reponame DB file" + cd build/ + /usr/bin/repo-remove -q "$reponame.db.tar.$DB_COMPRESSION" $packagename + + /bin/cp "$reponame.db.tar.$DB_COMPRESSION" "$ftppath/$1/" +} + if [ $# -ne 3 ]; then echo "usage: $(basename $0) <pkgname> <reponame> <arch>" exit 1 @@ -15,7 +31,7 @@ arch="$3"
export CARCH="$arch"
-ftppath="$FTP_BASE/$reponame/os/$arch/" +ftppath="$FTP_BASE/$reponame/os/" svnrepo="$reponame-$arch"
[ "$UID" = "" ] && UID=$(uid) @@ -64,20 +80,13 @@ fi cd "$WORKDIR" [ -d build/ ] || mkdir build
-# copy the db file into our working area -if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +if [ "$arch" == "any" ]; then + process_db_files i686 + process_db_files x86_64 else - echo "No database found at '$ftpdir', nothing more to do" - exit 0 + process_db_files $arch fi
-echo " Removing from $reponame DB file" -cd build/ -/usr/bin/repo-remove -q "$reponame.db.tar.$DB_COMPRESSION" $packagename - -/bin/cp "$reponame.db.tar.$DB_COMPRESSION" "$ftppath/" - echo "Package files will be cleaned up automatically"
cleanup diff --git a/db-update b/db-update index 77af39f..f51f470 100755 --- a/db-update +++ b/db-update @@ -15,6 +15,7 @@ _arch="$2" export CARCH="$_arch"
ftppath="$FTP_BASE/$reponame/os/$_arch/" +ftppath_base="$FTP_BASE/$reponame/os/" svnrepo="$reponame-$_arch" stagedir="$STAGING/$reponame"
@@ -72,10 +73,21 @@ repo_lock $reponame $_arch cd "$WORKDIR"
# copy the db file into our working area -if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +if [ "$_arch" == "any" ]; then + for architecture in i686 x86_64; do + if [ -f "$ftppath_base/$architecture/$reponame.db.tar.gz.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath_base/$architecture/$reponame.db.tar.gz.$DB_COMPRESSION" build/$reponame-i686.db.tar.gz + else + touch "build/$reponame-$architecture.db.tar.$DB_COMPRESSION" + fi + done +else + if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then + /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ + else + touch "build/$reponame.db.tar.$DB_COMPRESSION" + fi fi - echo "Updating DB for $reponame $_arch"
if [ -d "${stagedir}64" ]; then @@ -106,11 +118,11 @@ fi if [ -n "$ADDPKGS" ]; then
echo "==> Copying DB file from '$reponame'..." >&2 - if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then - /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ - else - touch "build/$reponame.db.tar.$DB_COMPRESSION" - fi +# if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then +# /bin/cp "$ftppath/$reponame.db.tar.$DB_COMPRESSION" build/ +# else +# touch "build/$reponame.db.tar.$DB_COMPRESSION" +# fi
echo "==> Processing new/updated packages for repository '$reponame'..." >&2
@@ -147,10 +159,19 @@ if [ -n "$ADDPKGS" ]; then
pkgs="" for pkg in $to_add; do pkgs="$pkgs $(basename $pkg)"; done - - /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs + if [ "$_arch" == "any" ]; then + /usr/bin/repo-add -q "$reponame-i686.db.tar.$DB_COMPRESSION" $pkgs + /usr/bin/repo-add -q "$reponame-x86_64.db.tar.$DB_COMPRESSION" $pkgs + else + /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs + fi else - rm -f "build/$reponame.db.tar.$DB_COMPRESSION" + if [ "$_arch" == "any" ]; then + rm -f "build/$reponame-i686.db.tar.gz" + rm -f "build/$reponame-x86_64.db.tar.gz" + else + rm -f "build/$reponame.db.tar.$DB_COMPRESSION" + fi echo "Errors found when adding packages" fi else @@ -159,13 +180,28 @@ fi
# if non empty, move all build dirs if [ $(/bin/ls "$WORKDIR/build/" 2>/dev/null | wc -l) != 0 ]; then - echo "Copying new files to '$ftppath'" - for f in "$WORKDIR/build/"*; do - if ! /bin/cp "$f" "$ftppath"; then - die "error: failure while copying files to $ftppath" - fi - done - + if [ "$_arch" != "any" ]; then + echo "Copying new files to '$ftppath'" + for f in "$WORKDIR/build/"*; do + if ! /bin/cp "$f" "$ftppath"; then + die "error: failure while copying files to $ftppath" + fi + done + else + echo "Copying architecture-independent packages..." + for f in "$WORKDIR/build/"*.pkg.tar.gz; do + if ! /bin/cp "$f" "$ftppath"; then + die "error: failure while copying $f to $ftppath" + fi + # Make link to arch-any package from os/{i686,x86_64} directories + ln -s "$ftppath/$f" "$ftppath_base/i686/" + ln -s "$ftppath/$f" "$ftppath_base/x86_64/" + for architecture in i686 x86_64; do + if ! /bin/cp "$reponame-$architecture.db.tar.$DB_COMPRESSION" "$ftppath_base/$architecture/$reponame.db.tar.$DB_COMPRESSION"; then + die "error: failed to transfer $architecture repository database." + fi + done + fi else echo "Nothing to copy, no work done" fi
-- Abhishek Dasgupta <http://abhidg.mine.nu> GPG 67972DOF pgp.mit.edu
Hmm the aur-general list is a weird place for this patch :)
The only thing that makes me hesitant about this is that I was going a slightly different route.
This patch here: http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=any-arch&id=1ef058dad91a5ca82db2744776251a2ab68bcf90 changes the dbscripts to loop over all architectures for each run. I wanted to simply add "any" into the file detection part, but not delete them until the very end
So this way, if I have 3 packages in my staging dir: foo(i686), bar(x86_64), and baz(any), running db-update does the following: for A in ARCHES (where ARCHES=(i686 x86_64)): get all files with "A" arch _or_ "any" arch do loop as normal delete all files we processed correctly
I guess the only thing this DOESN'T take into account is the symlinking part, but that would just be a single "special case" when copying
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
I guess the only thing this DOESN'T take into account is the symlinking part, but that would just be a single "special case" when copying
Modified patch based on the any-arch branch: (I've added SVN check for ANYPKGS separately and symlinks) --- db-update 2009-03-11 21:52:58.000000000 +0530 +++ db-update 2009-03-11 21:46:46.000000000 +0530 @@ -79,6 +79,7 @@ current_arch="$A" ftppath="$FTP_BASE/$reponame/os/$current_arch/" + ftppath_any="$FTP_BASE/$reponame/os/any/" if [ ! -d "$ftppath" ]; then echo "FTP path for this repo ($reponame) is missing" @@ -105,6 +106,7 @@ to_add="" if [ -d "$stagedir" ]; then ADDPKGS="$(/bin/ls $stagedir/*-${current_arch}$PKGEXT 2>/dev/null)" + ANYPKGS="$(/bin/ls $stagedir/*-any$PKGEXT 2>/dev/null)" fi if [ -n "$ADDPKGS" ]; then @@ -145,6 +147,30 @@ fi done + for pkg in $ANYPKGS; do + _pkgfile=$(basename $pkg) + _pkgname="$(getpkgname $pkg)" + svnrepo="$reponame-any" + echo " Validating package arch (any) $_pkgname" + if ! check_pkg_arch "$pkg" "any"; then + echo " ERROR: $_pkgfile is not architecture independent!" + else + echo " Checking SVN for $_pkgname" + /usr/bin/svn up -q $_pkgname + if [ -d "$_pkgname/repos/$svnrepo" ]; then + . "$_pkgname/repos/$svnrepo/$BUILDSCRIPT" + if [ "$_pkgfile" = "$pkgname-$pkgver-$pkgrel-any$PKGEXT" ]; then + to_add="$to_add $pkg" + else + echo " WARNING: $_pkgfile does not match $BUILDSCRIPT in $svnrepo" + fi + else + echo " WARNING: Package $_pkgname not found in $svnrepo" + fi + fi + done + + if [ -n "$to_add" ]; then cd "$WORKDIR/build/" for f in $to_add; do /bin/cp "$f" .; done @@ -164,12 +190,16 @@ # if non empty, move all build dirs if [ $(/bin/ls "$WORKDIR/build/" 2>/dev/null | wc -l) != 0 ]; then echo "Copying new files to '$ftppath'" - for f in "$WORKDIR/build/"*; do + for f in "$WORKDIR/build/"*$current_arch.pkg.tar.gz; do if ! /bin/cp "$f" "$ftppath"; then die "error: failure while copying files to $ftppath" fi done - + for f in "$WORKDIR/build/"*any.pkg.tar.gz; do + /bin/cp "$f" "$ftppath_any" || die "error: failure while copying files to $ftppath_any" + ln -s "$ftppath_any/$f" "$ftppath/$f" || die "error: failed to make link for $f." + done + /bin/cp "$WORKDIR/build/$reponame.db"* "$ftppath" || die "failed to move repository $reponame-$A". else echo "Nothing to copy, no work done" fi -- Abhishek
On Wed, Mar 11, 2009 at 11:26 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
I guess the only thing this DOESN'T take into account is the symlinking part, but that would just be a single "special case" when copying
Modified patch based on the any-arch branch: (I've added SVN check for ANYPKGS separately and symlinks)
--- db-update 2009-03-11 21:52:58.000000000 +0530 +++ db-update 2009-03-11 21:46:46.000000000 +0530 @@ -79,6 +79,7 @@ current_arch="$A"
ftppath="$FTP_BASE/$reponame/os/$current_arch/" + ftppath_any="$FTP_BASE/$reponame/os/any/"
if [ ! -d "$ftppath" ]; then echo "FTP path for this repo ($reponame) is missing" @@ -105,6 +106,7 @@ to_add="" if [ -d "$stagedir" ]; then ADDPKGS="$(/bin/ls $stagedir/*-${current_arch}$PKGEXT 2>/dev/null)" + ANYPKGS="$(/bin/ls $stagedir/*-any$PKGEXT 2>/dev/null)" fi
if [ -n "$ADDPKGS" ]; then @@ -145,6 +147,30 @@ fi done
+ for pkg in $ANYPKGS; do + _pkgfile=$(basename $pkg) + _pkgname="$(getpkgname $pkg)" + svnrepo="$reponame-any" + echo " Validating package arch (any) $_pkgname" + if ! check_pkg_arch "$pkg" "any"; then + echo " ERROR: $_pkgfile is not architecture independent!" + else + echo " Checking SVN for $_pkgname" + /usr/bin/svn up -q $_pkgname + if [ -d "$_pkgname/repos/$svnrepo" ]; then + . "$_pkgname/repos/$svnrepo/$BUILDSCRIPT" + if [ "$_pkgfile" = "$pkgname-$pkgver-$pkgrel-any$PKGEXT" ]; then + to_add="$to_add $pkg" + else + echo " WARNING: $_pkgfile does not match $BUILDSCRIPT in $svnrepo" + fi + else + echo " WARNING: Package $_pkgname not found in $svnrepo" + fi + fi + done + + if [ -n "$to_add" ]; then cd "$WORKDIR/build/" for f in $to_add; do /bin/cp "$f" .; done @@ -164,12 +190,16 @@ # if non empty, move all build dirs if [ $(/bin/ls "$WORKDIR/build/" 2>/dev/null | wc -l) != 0 ]; then echo "Copying new files to '$ftppath'" - for f in "$WORKDIR/build/"*; do + for f in "$WORKDIR/build/"*$current_arch.pkg.tar.gz; do if ! /bin/cp "$f" "$ftppath"; then die "error: failure while copying files to $ftppath" fi done - + for f in "$WORKDIR/build/"*any.pkg.tar.gz; do + /bin/cp "$f" "$ftppath_any" || die "error: failure while copying files to $ftppath_any" + ln -s "$ftppath_any/$f" "$ftppath/$f" || die "error: failed to make link for $f." + done + /bin/cp "$WORKDIR/build/$reponame.db"* "$ftppath" || die "failed to move repository $reponame-$A". else echo "Nothing to copy, no work done" fi
Committed. One problem I see is that the "rm $to_add" will delete the any packages for the first arch, so they won't be processed for the next arch in the loop. I committed the patch anyway, but that will still need to be fixed
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
Committed.
One problem I see is that the "rm $to_add" will delete the any packages for the first arch, so they won't be processed for the next arch in the loop. I committed the patch anyway, but that will still need to be fixed
Thanks. I've moved processing of architecture-independent packages out of the loop and am now storing the any packages list in to_add_any. That should fix the problem. BTW, trying to git pull from http://code.phraktured.net/git/arch/dbscripts/.git gives the error: fatal: http://code.phraktured.net/git/arch/dbscripts/.git/info/refs not found: did you run git update-server-info on the server? --- db-update 2009-03-11 22:32:39.000000000 +0530 +++ db-update 2009-03-11 23:09:30.000000000 +0530 @@ -76,6 +76,42 @@ trap ctrl_c 2 trap cleanup 0 +# Process architecture-independent packages first. +if [ -d "$stagedir" ]; then + ANYPKGS="$(/bin/ls $stagedir/*-any$PKGEXT 2>/dev/null)" +fi + +echo "==> Processing new/updated arch-independent packages for '$reponame'..." >&2 +cd "$WORKDIR" +svnpath="$(get_svnpath $reponame)" +/usr/bin/svn checkout -N $svnpath checkout +cd checkout +to_add_any="" +if [ -n "$ANYPKGS" ]; then + for pkg in $ANYPKGS; do + _pkgfile=$(basename $pkg) + _pkgname="$(getpkgname $pkg)" + svnrepo="$reponame-any" + echo " Validating package arch (any) $_pkgname" + if ! check_pkg_arch "$pkg" "any"; then + echo " ERROR: $_pkgfile is not architecture independent!" + else + echo " Checking SVN for $_pkgname" + /usr/bin/svn up -q $_pkgname + if [ -d "$_pkgname/repos/$svnrepo" ]; then + . "$_pkgname/repos/$svnrepo/$BUILDSCRIPT" + if [ "$_pkgfile" = "$pkgname-$pkgver-$pkgrel-any$PKGEXT" ]; then + to_add_any="$to_add_any $pkg" + else + echo " WARNING: $_pkgfile does not match $BUILDSCRIPT in $svnrepo" + fi + else + echo " WARNING: Package $_pkgname not found in $svnrepo" + fi + fi + done +fi + for A in ${ARCHES[@]}; do current_arch="$A" @@ -107,10 +143,9 @@ to_add="" if [ -d "$stagedir" ]; then ADDPKGS="$(/bin/ls $stagedir/*-${current_arch}$PKGEXT 2>/dev/null)" - ANYPKGS="$(/bin/ls $stagedir/*-any$PKGEXT 2>/dev/null)" fi - if [ -n "$ADDPKGS" ]; then + if [ -n "$ADDPKGS" || -n "$ANYPKGS" ]; then echo "==> Copying DB file from '$reponame'..." >&2 if [ -f "$ftppath/$reponame.db.tar.$DB_COMPRESSION" ]; then @@ -124,59 +159,38 @@ cd "$WORKDIR" /usr/bin/svn checkout -N $svnpath checkout cd checkout - - for pkg in $ADDPKGS; do - _pkgfile=$(basename $pkg) - _pkgname="$(getpkgname $pkg)" - - echo " Validating package arch ($current_arch) $_pkgname" - if ! check_pkg_arch "$pkg" "$current_arch"; then - echo " ERROR: $_pkgfile was built for the wrong architecture" - else - echo " Checking SVN for $_pkgname" - /usr/bin/svn up -q $_pkgname - if [ -d "$_pkgname/repos/$svnrepo" ]; then - . "$_pkgname/repos/$svnrepo/$BUILDSCRIPT" - if [ "$_pkgfile" = "$pkgname-$pkgver-$pkgrel-$current_arch$PKGEXT" ]; then - to_add="$to_add $pkg" - else - echo " WARNING: $_pkgfile does not match $BUILDSCRIPT in $svnrepo" - fi + + if [ -n "$ADDPKGS" ]; then + for pkg in $ADDPKGS; do + _pkgfile=$(basename $pkg) + _pkgname="$(getpkgname $pkg)" + + echo " Validating package arch ($current_arch) $_pkgname" + if ! check_pkg_arch "$pkg" "$current_arch"; then + echo " ERROR: $_pkgfile was built for the wrong architecture" else - echo " WARNING: Package $_pkgname not found in $svnrepo" - fi - fi - done - - for pkg in $ANYPKGS; do - _pkgfile=$(basename $pkg) - _pkgname="$(getpkgname $pkg)" - svnrepo="$reponame-any" - echo " Validating package arch (any) $_pkgname" - if ! check_pkg_arch "$pkg" "any"; then - echo " ERROR: $_pkgfile is not architecture independent!" - else - echo " Checking SVN for $_pkgname" - /usr/bin/svn up -q $_pkgname - if [ -d "$_pkgname/repos/$svnrepo" ]; then - . "$_pkgname/repos/$svnrepo/$BUILDSCRIPT" - if [ "$_pkgfile" = "$pkgname-$pkgver-$pkgrel-any$PKGEXT" ]; then - to_add="$to_add $pkg" + echo " Checking SVN for $_pkgname" + /usr/bin/svn up -q $_pkgname + if [ -d "$_pkgname/repos/$svnrepo" ]; then + . "$_pkgname/repos/$svnrepo/$BUILDSCRIPT" + if [ "$_pkgfile" = "$pkgname-$pkgver-$pkgrel-$current_arch$PKGEXT" ]; then + to_add="$to_add $pkg" + else + echo " WARNING: $_pkgfile does not match $BUILDSCRIPT in $svnrepo" + fi else - echo " WARNING: $_pkgfile does not match $BUILDSCRIPT in $svnrepo" + echo " WARNING: Package $_pkgname not found in $svnrepo" fi - else - echo " WARNING: Package $_pkgname not found in $svnrepo" fi - fi - done + done + fi - if [ -n "$to_add" ]; then + if [ -n "$to_add" || -n "$to_add_any" ]; then cd "$WORKDIR/build/" - for f in $to_add; do /bin/cp "$f" .; done + for f in $to_add $to_add_any; do /bin/cp "$f" .; done pkgs="" - for pkg in $to_add; do pkgs="$pkgs $(basename $pkg)"; done + for pkg in $to_add $to_add_any; do pkgs="$pkgs $(basename $pkg)"; done /usr/bin/repo-add -q "$reponame.db.tar.$DB_COMPRESSION" $pkgs else @@ -218,6 +232,7 @@ repo_unlock $reponame $current_arch done +/bin/rm $to_add_any cleanup # vim: set ts=4 sw=4 noet ft=sh: -- Abhishek
On Wed, Mar 11, 2009 at 12:48 PM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
Committed.
One problem I see is that the "rm $to_add" will delete the any packages for the first arch, so they won't be processed for the next arch in the loop. I committed the patch anyway, but that will still need to be fixed
Thanks. I've moved processing of architecture-independent packages out of the loop and am now storing the any packages list in to_add_any. That should fix the problem. BTW, trying to git pull from http://code.phraktured.net/git/arch/dbscripts/.git gives the error: fatal: http://code.phraktured.net/git/arch/dbscripts/.git/info/refs not found: did you run git update-server-info on the server?
Fixed that. Do you have a public git repo anywhere where I could pull patches from? It would make this easier. If not, that's fine
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
Fixed that. Do you have a public git repo anywhere where I could pull patches from? It would make this easier. If not, that's fine
http://abhidg.mine.nu/gitweb/?p=dbscripts.git;a=summary -- Abhishek
2009/3/12 Abhishek Dasgupta <abhidg@gmail.com>:
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
Fixed that. Do you have a public git repo anywhere where I could pull patches from? It would make this easier. If not, that's fine
My home server is sometimes down due to network outage, so I've pushed the repository to github as well: http://github.com/abhidg/dbscripts -- Abhishek
On Thu, Mar 12, 2009 at 1:45 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/12 Abhishek Dasgupta <abhidg@gmail.com>:
2009/3/11 Aaron Griffin <aaronmgriffin@gmail.com>:
Fixed that. Do you have a public git repo anywhere where I could pull patches from? It would make this easier. If not, that's fine
My home server is sometimes down due to network outage, so I've pushed the repository to github as well: http://github.com/abhidg/dbscripts
Finally pulled in the last patch there into my any-arch branch. I'm not sure how I feel about it, seeing as it duplicates a lot of work (I was hoping to reuse the loop somehow), but it seems ok otherwise. Now we just need to make sure db-move and db-remove can handle "any" packages
2009/3/17 Aaron Griffin <aaronmgriffin@gmail.com>:
Finally pulled in the last patch there into my any-arch branch.
I'm not sure how I feel about it, seeing as it duplicates a lot of work (I was hoping to reuse the loop somehow), but it seems ok otherwise.
Now we just need to make sure db-move and db-remove can handle "any" packages
I could have added the ANYPKGS stuff in the main loop itself, but that was cluttering the code with a lot of if calls. I'm now working on db-{move, remove}. Would it be better if I refactored it to move packages in all archs possible, rather than specifying the arch? Then we could get rid of the testing2core64 and other *64 scripts. -- Abhishek.
On Thu, Mar 19, 2009 at 1:54 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/17 Aaron Griffin <aaronmgriffin@gmail.com>:
Finally pulled in the last patch there into my any-arch branch.
I'm not sure how I feel about it, seeing as it duplicates a lot of work (I was hoping to reuse the loop somehow), but it seems ok otherwise.
Now we just need to make sure db-move and db-remove can handle "any" packages
I could have added the ANYPKGS stuff in the main loop itself, but that was cluttering the code with a lot of if calls. I'm now working on db-{move, remove}. Would it be better if I refactored it to move packages in all archs possible, rather than specifying the arch? Then we could get rid of the testing2core64 and other *64 scripts.
I thought of that. The reason I decided not to was because some packages could move, say, out of testing for i686 and NOT x86_64. I don't know if that's a common case though. Perhaps we can do this: move all packages by default. if the arch is specified on the command line, move only that arch. or in code - looparches="${ARCHES[@]}" [ -n "$2" ] && looparches="$2" for arch in $looparches; do .... do move logic here done
2009/3/19 Aaron Griffin <aaronmgriffin@gmail.com>:
I thought of that. The reason I decided not to was because some packages could move, say, out of testing for i686 and NOT x86_64. I don't know if that's a common case though.
Perhaps we can do this: move all packages by default. if the arch is specified on the command line, move only that arch. or in code -
looparches="${ARCHES[@]}" [ -n "$2" ] && looparches="$2"
for arch in $looparches; do .... do move logic here done
I just committed a patch to github for adding any architecture support to db-move and db-remove: http://github.com/abhidg/dbscripts/commit/67e112955482397bae1971fa60e1634a92... -- Abhishek
On Fri, Mar 20, 2009 at 11:04 AM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/19 Aaron Griffin <aaronmgriffin@gmail.com>:
I thought of that. The reason I decided not to was because some packages could move, say, out of testing for i686 and NOT x86_64. I don't know if that's a common case though.
Perhaps we can do this: move all packages by default. if the arch is specified on the command line, move only that arch. or in code -
looparches="${ARCHES[@]}" [ -n "$2" ] && looparches="$2"
for arch in $looparches; do .... do move logic here done
I just committed a patch to github for adding any architecture support to db-move and db-remove: http://github.com/abhidg/dbscripts/commit/67e112955482397bae1971fa60e1634a92...
I'm trying to figure out if it would make sense to also remove the old symlinks, or if the cleanup scripts would handle it just fine... I think the cleanup scripts would work The last thing on the list would be adding a block to the cleanup scripts to cleanup 'any' packages that don't have remaining symlinks... seeing as the 'any' dir doesn't have a real DB, we can't clean it up like normal, but if it symlinks to nothing, it can be deleted. Thanks for all your work on this, you're my hero - and the FIRST outside contributor to the db-scripts :)
2009/3/20 Aaron Griffin <aaronmgriffin@gmail.com>:
I'm trying to figure out if it would make sense to also remove the old symlinks, or if the cleanup scripts would handle it just fine... I think the cleanup scripts would work
The cleanup scripts should be deleting the symlinks properly.
The last thing on the list would be adding a block to the cleanup scripts to cleanup 'any' packages that don't have remaining symlinks... seeing as the 'any' dir doesn't have a real DB, we can't clean it up like normal, but if it symlinks to nothing, it can be deleted.
This should do this: http://github.com/abhidg/dbscripts/commit/0959e8914c08042b1a8d1227c33f4ec77e...
Thanks for all your work on this, you're my hero - and the FIRST outside contributor to the db-scripts :)
Thank you and all the other wonderful people too who have worked on this and other backend stuff which helps everything run smoothly :) -- Abhishek
On Fri, Mar 20, 2009 at 1:31 PM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/20 Aaron Griffin <aaronmgriffin@gmail.com>:
I'm trying to figure out if it would make sense to also remove the old symlinks, or if the cleanup scripts would handle it just fine... I think the cleanup scripts would work
The cleanup scripts should be deleting the symlinks properly.
The last thing on the list would be adding a block to the cleanup scripts to cleanup 'any' packages that don't have remaining symlinks... seeing as the 'any' dir doesn't have a real DB, we can't clean it up like normal, but if it symlinks to nothing, it can be deleted.
This should do this: http://github.com/abhidg/dbscripts/commit/0959e8914c08042b1a8d1227c33f4ec77e...
Thanks for all your work on this, you're my hero - and the FIRST outside contributor to the db-scripts :)
Thank you and all the other wonderful people too who have worked on this and other backend stuff which helps everything run smoothly :)
Thanks! Patches pulled into my any-arch branch. I think we're good to merge them into master and push them live... do you see any other issues? Once this is live, I'll updated devtools to support the any arch as well.
2009/3/21 Aaron Griffin <aaronmgriffin@gmail.com>:
Thanks! Patches pulled into my any-arch branch.
I think we're good to merge them into master and push them live... do you see any other issues?
Once this is live, I'll updated devtools to support the any arch as well.
There should not be any problems since the changes are backward compatible (except db-update, of course). I didn't understand what the purpose of setting CARCH in db-move and db-remove is; when we use any packages, then CARCH becomes 'any' which isn't really valid. Also calling db-remove archindeppkg repo i686 (or x86_64) will remove the arch independent package only from that particular architecture. I don't know whether this should be a bug or a feature :) I can't see any bugs which would crop up because of the absence of the 'any' directory on the ftp, but it'd be better to create those directories anyway. -- Abhishek
On Fri, Mar 20, 2009 at 11:25 PM, Abhishek Dasgupta <abhidg@gmail.com> wrote:
2009/3/21 Aaron Griffin <aaronmgriffin@gmail.com>:
Thanks! Patches pulled into my any-arch branch.
I think we're good to merge them into master and push them live... do you see any other issues?
Once this is live, I'll updated devtools to support the any arch as well.
There should not be any problems since the changes are backward compatible (except db-update, of course). I didn't understand what the purpose of setting CARCH in db-move and db-remove is; when we use any packages, then CARCH becomes 'any' which isn't really valid.
I don't know either. I think it was a safety net... or maybe it was in there for a reason once and isn't anymore. It could probably be removed.
Also calling db-remove archindeppkg repo i686 (or x86_64) will remove the arch independent package only from that particular architecture. I don't know whether this should be a bug or a feature :)
That sounds fine to me. It's always possible we have edge cases like this.
participants (4)
-
Aaron Griffin
-
Abhishek Dasgupta
-
Loui Chang
-
Phil Dillon-Thiselton