March 2010 - Arch-dev-public - lists.archlinux.org

[arch-dev-public] [PATCH 4/4] ftpdir-cleanup: lock repo before doing the cleanup
by Eric Bélanger 02 Mar '10

02 Mar '10

This should fix the problem of the ftpdir-cleanup script removing the new package instead of the old one (FS#17058). The script makes $LOCK_TRIAL attempts, each separated by $LOCK_DELAY seconds, in getting the repo lock. If the lock is unsuccessful, the cleanup is skipped for that particular repo. Signed-off-by: Eric Bélanger <snowmaniscool(a)gmail.com> BTW, I didn't really tested this except the while loop logic. --- misc-scripts/ftpdir-cleanup | 28 +++++++++++++++++++++++++--- 1 files changed, 25 insertions(+), 3 deletions(-) diff --git a/misc-scripts/ftpdir-cleanup b/misc-scripts/ftpdir-cleanup index a185090..77f79e6 100755 --- a/misc-scripts/ftpdir-cleanup +++ b/misc-scripts/ftpdir-cleanup @@ -10,15 +10,35 @@ dest=$2 ############################################################ -. "$(dirname $0)/../db-functions" +. "$(dirname $0)/../db-functions" . "$(dirname $0)/../config" ${CLEANUP_DRYRUN} && echo 'dry run mode is active' ftppath_base="$FTP_BASE/$reponame/$FTP_OS_SUFFIX" +LOCK_DELAY=10 +LOCK_TRIAL=6 + for arch in ${ARCHES[@]}; do + IS_LOCKED=0 + count=0 + while [ $count -le $LOCK_TRIAL ]; do + if repo_lock $reponame $arch ; then + IS_LOCKED=1 + let count=$LOCK_TRIAL+1 + continue + fi + sleep $LOCK_DELAY + let count=$count+1 + done + + if [ $IS_LOCKED -eq 0 ]; then + echo "Failed to lock $reponame $arch repo" + exit 1 + fi + TMPDIR=$(mktemp -d /tmp/cleanup-XXXXXX) || exit 1 ftppath="$ftppath_base/$arch" MISSINGFILES="" @@ -80,10 +100,10 @@ for arch in ${ARCHES[@]}; do dbpkgname=$(grep -A1 '^%FILENAME%$' "${p}/desc" 2>/dev/null| tail -n1) if [ "${dbpkgname}" = "${pkgname}" ]; then continue 2 - fi + fi done EXTRAFILES="$EXTRAFILES $pkg" - done + done rm -rf ${TMPDIR} @@ -100,6 +120,8 @@ for arch in ${ARCHES[@]}; do fi done + repo_unlock $reponame $arch + #Make sure we've done *something* before outputting anything if [ -z "$DELETEFILES$DELETESYMLINKS$MISSINGFILES$EXTRAFILES" ]; then continue -- 1.7.0

4 6

[arch-dev-public] [PATCH 4/4] Make repo locking an atomic process and added timeout argument
by Eric Bélanger 02 Mar '10

02 Mar '10

The repo locking function now use a lock directory instead of a lock file. This makes the lock checking and creation an atomic process. The repo_lock function will now try to obtain a lock every $LOCK_DELAY seconds until it is successful. An optional third argument can be use to give a timeout in seconds; this is intended for scripts that runs unattended. Repo locking is now used in the ftpdir-cleanup script. This should fix the problem of the ftpdir-cleanup script removing the new package instead of the old one (FS#17058). Signed-off-by: Eric Bélanger <snowmaniscool(a)gmail.com> --- config | 3 ++ db-functions | 48 ++++++++++++++++++++++++++++-------------- misc-scripts/ftpdir-cleanup | 10 ++++++-- 3 files changed, 42 insertions(+), 19 deletions(-) diff --git a/config b/config index 8d04c60..85a225a 100644 --- a/config +++ b/config @@ -9,6 +9,9 @@ CLEANUP_DESTDIR="/srv/package-cleanup" CLEANUP_DRYRUN=false SOURCE_CLEANUP_DESTDIR="/srv/source-cleanup" +LOCK_DELAY=10 +LOCK_TIMEOUT=300 + STAGING="$HOME/staging" TMPDIR="/srv/tmp" ARCHES=(i686 x86_64) diff --git a/db-functions b/db-functions index e756368..d688df6 100644 --- a/db-functions +++ b/db-functions @@ -15,27 +15,43 @@ restore_umask () { umask $UMASK } -repo_lock () { #repo_lock repo-name arch - LOCKFILE="$TMPDIR/.repolock.$1.$2" - if [ -f "$LOCKFILE" ]; then - owner="$(/usr/bin/stat -c %U $LOCKFILE)" - echo "error: db generation is already in progress (started by $owner)" >&2 - exit 1 - else - /bin/touch "$LOCKFILE" - if [ ! -f "$LOCKFILE" ]; then - echo "error: could not create repo lock... something went wrong!" >&2 - fi - set_umask +repo_lock () { #repo_lock <repo-name> <arch> [timeout] + LOCKDIR="$TMPDIR/.repolock.$1.$2" + local _count + local _trial + local _timeout + local _lockblock + local _owner + + if [ $# -eq 2 ]; then + _lockblock=true + _trial=0 + elif [ $# -eq 3 ]; then + _lockblock=false + _timeout=$3 + let _trial=$_timeout/$LOCK_DELAY fi + + _count=0 + while [ $_count -le $_trial ] || $_lockblock ; do + if ! mkdir "$LOCKDIR" >/dev/null 2>&1 ; then + _owner="$(/usr/bin/stat -c %U $LOCKDIR)" + echo "error: Repo $1-$2 is already locked by $_owner. Retrying in $LOCK_DELAY seconds..." >&2 + else + set_umask + break + fi + sleep $LOCK_DELAY + let _count=$_count+1 + done } -repo_unlock () { #repo_unlock repo-name arch - LOCKFILE="$TMPDIR/.repolock.$1.$2" - if [ ! -f "$LOCKFILE" ]; then +repo_unlock () { #repo_unlock <repo-name> <arch> + LOCKDIR="$TMPDIR/.repolock.$1.$2" + if [ ! -d "$LOCKDIR" ]; then echo "error: repo lock doesn't exist... something went terribly wrong!" >&2 else - rm -f "$LOCKFILE" + rmdir "$LOCKDIR" fi restore_umask } diff --git a/misc-scripts/ftpdir-cleanup b/misc-scripts/ftpdir-cleanup index a185090..4c559b9 100755 --- a/misc-scripts/ftpdir-cleanup +++ b/misc-scripts/ftpdir-cleanup @@ -10,7 +10,7 @@ dest=$2 ############################################################ -. "$(dirname $0)/../db-functions" +. "$(dirname $0)/../db-functions" . "$(dirname $0)/../config" ${CLEANUP_DRYRUN} && echo 'dry run mode is active' @@ -19,6 +19,8 @@ ftppath_base="$FTP_BASE/$reponame/$FTP_OS_SUFFIX" for arch in ${ARCHES[@]}; do + repo_lock $reponame $arch $LOCK_TIMEOUT + TMPDIR=$(mktemp -d /tmp/cleanup-XXXXXX) || exit 1 ftppath="$ftppath_base/$arch" MISSINGFILES="" @@ -80,10 +82,10 @@ for arch in ${ARCHES[@]}; do dbpkgname=$(grep -A1 '^%FILENAME%$' "${p}/desc" 2>/dev/null| tail -n1) if [ "${dbpkgname}" = "${pkgname}" ]; then continue 2 - fi + fi done EXTRAFILES="$EXTRAFILES $pkg" - done + done rm -rf ${TMPDIR} @@ -100,6 +102,8 @@ for arch in ${ARCHES[@]}; do fi done + repo_unlock $reponame $arch + #Make sure we've done *something* before outputting anything if [ -z "$DELETEFILES$DELETESYMLINKS$MISSINGFILES$EXTRAFILES" ]; then continue -- 1.7.0.1

1 0

[arch-dev-public] archboot release homepage created
by Tobias Powalowski 01 Mar '10

01 Mar '10

Hi guys, just created a archboot release homepage, would it be possible to link it somewhere on DOWNLOAD on our main page? ftp://ftp.archlinux.org/iso/archboot/index.html is the adress. thanks in advance greetings tpowa -- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa(a)archlinux.org

4 5

[arch-dev-public] [signoff] m4-1.4.14
by Allan McRae 01 Mar '10

01 Mar '10

Upstream update. * Noteworthy changes in Version 1.4.14 (2010-02-24) [stable] Released by Eric Blake, based on git version 1.4.13.* ** Fix regression introduced in 1.4.12 where executing with stdout closed could crash m4 on exit on some platforms. ** Fix regressions introduced in 1.4.13 in the `esyscmd' builtin, where closed file descriptors could interfere with child execution, and where a child status of 127 made m4 print a spurious message to stderr. ** Fix a security hole in 'make dist', present since at least M4 1.4, that could affect anybody attempting to redistribute modified sources (see Automake CVE-2009-4029). ** A number of portability improvements inherited from gnulib. Signoff both, Allan

2 2