[arch-dev-public] [PATCH 1/4] Moved ALLOWED_LICENSES declaration to config file
Signed-off-by: Eric B�langer <snowmaniscool@gmail.com> --- config | 3 +++ misc-scripts/make-sourceball | 3 --- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/config b/config index 86a5eca..24ec37a 100644 --- a/config +++ b/config @@ -13,3 +13,6 @@ BUILDSCRIPT="PKGBUILD" DBEXT=".db.tar.gz" PKGEXT=".pkg.tar.gz" SRCEXT=".src.tar.gz" + +# Allowed licenses: get sourceballs only for licenses in this array +ALLOWED_LICENSES=('GPL' 'GPL1' 'GPL2' 'LGPL' 'LGPL1' 'LGPL2') diff --git a/misc-scripts/make-sourceball b/misc-scripts/make-sourceball index 865e0e7..462549f 100755 --- a/misc-scripts/make-sourceball +++ b/misc-scripts/make-sourceball @@ -1,8 +1,5 @@ #!/bin/bash -# Allowed licenses: build only for licenses in this array -ALLOWED_LICENSES=('GPL' 'GPL1' 'GPL2' 'LGPL' 'LGPL1' 'LGPL2') - if [ $# -ne 3 -a $# -ne 4 ]; then echo "usage: $(basename $0) [-f] <packagename> <repo> <arch>" echo " -f Force building. Skip license checks" -- 1.6.5
Signed-off-by: Eric B�langer <snowmaniscool@gmail.com> --- db-functions | 16 ++++++++++++++++ misc-scripts/make-sourceball | 24 +----------------------- 2 files changed, 17 insertions(+), 23 deletions(-) diff --git a/db-functions b/db-functions index f4f9ee9..6611a92 100644 --- a/db-functions +++ b/db-functions @@ -110,4 +110,20 @@ get_repos_for_host() { fi } +#usage: chk_license ${license[@]}" +chk_license() { + local l + for l in "$@"; do + l="$(echo $l | tr '[:upper:]' '[:lower:]')" + for allowed in ${ALLOWED_LICENSES[@]}; do + allowed="$(echo $allowed | tr '[:upper:]' '[:lower:]')" + if [ "$l" = "$allowed" ]; then + return 0 + fi + done + done + + return 1 +} + # vim: set ts=4 sw=4 noet ft=sh: diff --git a/misc-scripts/make-sourceball b/misc-scripts/make-sourceball index 462549f..1e7eabc 100755 --- a/misc-scripts/make-sourceball +++ b/misc-scripts/make-sourceball @@ -61,33 +61,11 @@ pkgver_from_src() { echo $tmp | sed 's|.*-\(.*-.*\)$|\1|g' } -#usage: chk_license ${license[@]}" -chk_license() { - #The -f flag skips this check - if [ $FORCE = 1 ]; then - return 0 - fi - - local l - for l in "$@"; do - l="$(echo $l | tr '[:upper:]' '[:lower:]')" - for allowed in ${ALLOWED_LICENSES[@]}; do - allowed="$(echo $allowed | tr '[:upper:]' '[:lower:]')" - if [ "$l" = "$allowed" ]; then - return 0 - fi - done - done - - return 1 -} - create_srcpackage() { if [ -d "$1" ]; then pushd "$1" >/dev/null . "$BUILDSCRIPT" - - if ! chk_license ${license[@]}; then + if ! [ $FORCE == 1 ] && ! chk_license ${license[@]} ; then #Removed so as not to clutter failed.txt #echo -e "\t$packagename license (${license[@]}) does not require source tarballs" >&2 cleanup 0 -- 1.6.5
Signed-off-by: Eric B�langer <snowmaniscool@gmail.com> --- config | 1 + misc-scripts/make-sourceball | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-) diff --git a/config b/config index 24ec37a..92def37 100644 --- a/config +++ b/config @@ -5,6 +5,7 @@ FTP_OS_SUFFIX="os" SVNREPO="file:///srv/svn-packages" CLEANUP_DESTDIR="/srv/package-cleanup" +SOURCE_CLEANUP_DESTDIR="/srv/source-cleanup" STAGING="$HOME/staging" TMPDIR="/srv/tmp" diff --git a/misc-scripts/make-sourceball b/misc-scripts/make-sourceball index 1e7eabc..8899fca 100755 --- a/misc-scripts/make-sourceball +++ b/misc-scripts/make-sourceball @@ -117,7 +117,7 @@ remove_old() { fi done if [ $skip -ne 1 ]; then - rm -f "$srcpkg" + mv "$srcpkg" $SOURCE_CLEANUP_DESTDIR fi fi done -- 1.6.5
This script is intended to be a weekly cron-job. It supplements the current sourceballs cleanup in make-sourceball that only removes the old sourceballs. This script removes the sourceballs of packages that were either removed completely from the repo or moved to a splitted package. It also checks the license of the packages which have a sourceball to see, in case of a license change, if the hosting of the sources is still necessary. Signed-off-by: Eric B�langer <snowmaniscool@gmail.com> --- cron-jobs/sourceballs-cleanup | 61 +++++++++++++++++++++++++++++++++++++++++ 1 files changed, 61 insertions(+), 0 deletions(-) create mode 100755 cron-jobs/sourceballs-cleanup diff --git a/cron-jobs/sourceballs-cleanup b/cron-jobs/sourceballs-cleanup new file mode 100755 index 0000000..c8d7b15 --- /dev/null +++ b/cron-jobs/sourceballs-cleanup @@ -0,0 +1,61 @@ +#!/bin/bash + +. "$(dirname $0)/../db-functions" +. "$(dirname $0)/../config" + +srcpath="$FTP_BASE/sources/" +logfile="$srcpath/cleanup.txt" + +LOCKFILE="/tmp/.sourceball-cleanup.lock" +WORKDIR="/tmp/sourceball-cleanup.$packagename.$UID" + +cleanup () { + restore_umask + rm -rf "$WORKDIR" + rm -f "$LOCKFILE" + exit 0 +} + +ctrl_c() { + echo "Interrupted" >&2 + cleanup 0 +} + +if [ -f "$LOCKFILE" ]; then + owner="$(/usr/bin/stat -c %U $LOCKFILE)" + echo "error: source tarball generation is already in progress (started by $owner)" + exit 1 +fi + +trap cleanup 0 +trap ctrl_c 2 + +/bin/touch "$LOCKFILE" + +#adjust the nice level to run at a lower priority +/usr/bin/renice +10 -p $$ > /dev/null + +set_umask +/bin/mkdir -p "$WORKDIR" +cd "$WORKDIR" + +[ -e "$logfile" ] && /bin/mv "$logfile" "$logfile.old" +echo "Orphaned sourceballs:" > "$logfile" + +for sourceball in "$srcpath"/*$SRCEXT; do + packagename=$(basename $sourceball) + packagename=${packagename%-*-*$SRCEXT} + + if ! /usr/bin/svn export -q --force "$SVNREPO/$packagename" "$packagename" >/dev/null 2>&1 ; then + echo "$packagename : no longer in svn. Removing sourceball." >> "$logfile" + mv $sourceball $SOURCE_CLEANUP_DESTDIR + elif [ -z "$(ls -A "$packagename/repos")" ]; then + echo "$packagename : no longer in repos but trunk is still in svn. Removing sourceball." >> "$logfile" + mv $sourceball $SOURCE_CLEANUP_DESTDIR + elif ! source "$packagename/trunk/$BUILDSCRIPT" && chk_license ${license[@]}; then + echo "$packagename : source hosting no longer required by license. Removing sourceball." >> "$logfile" + mv $sourceball $SOURCE_CLEANUP_DESTDIR + fi +done + +cleanup 0 -- 1.6.5
Thanks for the patches, they all look good so I'll merge them in soon. After this, I think it's finally time to release the new db-scripts. They've been sitting in /arch-new/ for a while now and most of the running cron jobs (source balls, integrity check, etc) are already running from there. Any complaints?
On Wed, Oct 14, 2009 at 3:29 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Thanks for the patches, they all look good so I'll merge them in soon.
After this, I think it's finally time to release the new db-scripts. They've been sitting in /arch-new/ for a while now and most of the running cron jobs (source balls, integrity check, etc) are already running from there.
Any complaints?
Don't forget to setup the /srv/source-cleanup directory. FTR, I asked Dan a while ago to disable the sourceball cron job when I was working on it. I believe it's working fine now altough I have a hunch that the removal of old sourceballs isn't working as it should. It looks like an old standing bug that might have been fixed since the last release. Anyway, once it's set up, I'll keep an eye on it to see how it goes.
participants (3)
-
Aaron Griffin
-
Eric Bélanger
-
Eric B�langer