[pacman-dev] About makepkg dlagents

About download-agents for makepkg. (This is why I've bothered the developers of pacman this morning with compilation issues... :)) So I wanted to be able to make PKGBUILD and makepkg extract sources for Git repositories. And after asking on the arch-general, I was pointed to the following bug: http://bugs.archlinux.org/task/7816 As a consequence I've started hacking and commed up with the following patches (put at the end). (I've ended up with four pathes from git format-patch... Should I put only the diff between maint branch and my branch?) Also below is the description on the bug-tracker: ~~~~ I've made some hacking staring from pacman Git repository. My work is available at (branch patches/git-dlagent): http://gitorious.org/~ciprian.craciun/pacman/ciprian-craciun-patches In summary: * added a script: /usr/bin/git-dlagent that takes exactly two arguments: URL and output. * updated /etc/makepkg.conf to include the following URL's: git://, git+file://, git+http://, git+https://, git+ssh://, git+rsync; * a Git url looks-like: git+git://projects.archlinux.org/pacman.git#master##pacman.tar where: * what is before the first `#` is the URL used to (maybe) clone the Git repository. * what is after the first `#` and before `##` is the refspec (head / tag / etc.) to export; * what is after `##` (optional this part is) it can be used to make the file-part of the URL look like a tar file; How git-dlagent it works: * in case git+file:// is given `git --git-dir=... archive` is done; * else it tries to do `git archive --remote=...`; (usually this doesn't work unless the repository is git and the barkend is enabled); * but anyway it fallbacks to `git clone ...`, and then a local git archive; ~~~~ (My approach just uses the Git / SVN / etc tools to create a .tar or .zip file, and leave the rest up to makepkg. I've also patched makepkg to ignore those hashes that are equal to `--`.) ~~~~ I would say it integrated prety nice with the whole makepkg, and also does not impact the source code by much. An example of an URL used in PKGBUILD would be the following: git+file:///stores/erebus-1/ciprian/workbench/pacman/.git#patches/master##/$pkgname-$pkgver.tar git://projects.archlinux.org/pacman.git#maint#/pacman-v3.3.2.tar Is anyone interested in adopting this patch? Any comments? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From c08594be518814646f7e9d9224a0068e9e4d6f34 Mon Sep 17 00:00:00 2001 From: Ciprian Dorin, Craciun Date: Mon, 19 Oct 2009 22:53:06 +0300 Subject: [PATCH 1/4] Added option to ignore a source integrity hash. (We just replace the hash with `--`.) Added makepkg download-agent for git. (git-dlagent.bash)

--- scripts/git-dlagent.sh.in | 78 +++++++++++++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 14 +++++--- 2 files changed, 87 insertions(+), 5 deletions(-) create mode 100644 scripts/git-dlagent.sh.in diff --git a/scripts/git-dlagent.sh.in b/scripts/git-dlagent.sh.in new file mode 100644 index 0000000..a2240a7 --- /dev/null +++ b/scripts/git-dlagent.sh.in @@ -0,0 +1,78 @@ +#!/bin/bash + +set -e -u -o pipefail || exit 1 +test "${#}" -eq 2 + +uri="${1}" +output="${2}" + +git_archive_method=unknown + + +if [[ "${uri}" =~ ^git://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +then + uri="${uri/%##*}" + reference="${uri/#*#}" + uri="${uri/%#*}" + + git_archive_method=remote + +elif [[ "${uri}" =~ ^(git\+)?file:///[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +then + uri="${uri/%##*}" + reference="${uri/#*#}" + uri="${uri/%#*}" + uri="${uri/#git+file:\/\/}" + + git_archive_method=local + +elif [[ "${uri}" =~ ^(git\+)?(ssh|http|https|rsync)://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +then + uri="${uri/%##*}" + reference="${uri/#*#}" + uri="${uri/%#*}" + uri="${uri/#git+}" + + git_archive_method=remote + +else + echo "[ee] \`${uri}\` is not a Git URI." >&2 + exit 1 +fi + + +case "${git_archive_method}" in + + ( local ) + + git --git-dir="${uri}" archive --format=tar --output="${output}" "${reference}" + + exit 0 + ;; + + ( remote ) + + git archive --format=tar --output="${output}" --remote="${uri}" "${reference}" \ + && exit 0 + + echo "[ww] \`${uri}\` does not support git-archive; going after clone." >&2 + + clone="$( mktemp -d )" + + git clone --bare "${uri}" "${clone}" + + git --git-dir="${clone}" archive --format=tar --output="${output}" "${reference}" + + find "${clone}" -delete + + exit 0 + ;; + + ( * ) + echo "[ee] unknown git-archive-method." >&2 + exit 1 + ;; + +esac + +exit 1 diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 3662228..3cdf2aa 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -591,12 +591,16 @@ check_checksums() { if [ $found -gt 0 ] ; then local expectedsum="$(echo ${integrity_sums[$idx]} | tr '[A-F]' '[a-f]')" - local realsum="$(openssl dgst -${integ} "$file" | awk '{print $NF}')" - if [ "$expectedsum" = "$realsum" ]; then - echo "$(gettext "Passed")" >&2 + if [ "$expectedsum" = '--' ]; then + echo "$(gettext "Skipped")" >&2 else - echo "$(gettext "FAILED")" >&2 - errors=1 + local realsum="$(openssl dgst -${integ} "$file" | awk '{print $NF}')" + if [ "$expectedsum" = "$realsum" ]; then + echo "$(gettext "Passed")" >&2 + else + echo "$(gettext "FAILED")" >&2 + errors=1 + fi fi fi -- 1.6.3.1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From 47c0d7fef2eefa69081c809dd285fa2fcf88eb36 Mon Sep 17 00:00:00 2001 From: Ciprian Dorin, Craciun Date: Mon, 19 Oct 2009 22:57:53 +0300 Subject: [PATCH 2/4] Integrated git-dlagent into the build system.

--- etc/makepkg.conf.in | 8 +++++++- scripts/Makefile.am | 7 +++++-- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index 1368ff1..29f6e4e 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -12,7 +12,13 @@ DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3 -O %o %u' 'http::/usr/bin/wget -c -t 3 --waitretry=3 -O %o %u' 'https::/usr/bin/wget -c -t 3 --waitretry=3 --no-check-certificate -O %o %u' 'rsync::/usr/bin/rsync -z %u %o' - 'scp::/usr/bin/scp -C %u %o') + 'scp::/usr/bin/scp -C %u %o' + 'git::/usr/bin/git-dlagent %u %o' + 'git+file::/usr/bin/git-dlagent %u %o' + 'git+http::/usr/bin/git-dlagent %u %o' + 'git+https::/usr/bin/git-dlagent %u %o' + 'git+ssh::/usr/bin/git-dlagent %u %o' + 'git+rsync::/usr/bin/git-dlagent %u %o') # Other common tools: # /usr/bin/snarf diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 5d65653..524710c 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -10,14 +10,16 @@ OURSCRIPTS = \ pacman-optimize \ pkgdelta \ rankmirrors \ - repo-add + repo-add \ + git-dlagent EXTRA_DIST = \ makepkg.sh.in \ pacman-optimize.sh.in \ pkgdelta.sh.in \ rankmirrors.py.in \ - repo-add.sh.in + repo-add.sh.in \ + git-dlagent.sh.in # Files that should be removed, but which Automake does not know. MOSTLYCLEANFILES = $(bin_SCRIPTS) *.tmp @@ -67,5 +69,6 @@ repo-add: $(srcdir)/repo-add.sh.in repo-remove: $(srcdir)/repo-add.sh.in rm -f repo-remove $(LN_S) repo-add repo-remove +git-dlagent: $(srcdir)/git-dlagent.sh.in # vim:set ts=2 sw=2 noet: -- 1.6.3.1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From 8fc53a663e7b7521133a05b8e28d9fd3f15efaea Mon Sep 17 00:00:00 2001 From: Ciprian Dorin, Craciun Date: Mon, 19 Oct 2009 23:48:51 +0300 Subject: [PATCH 3/4] Updated git-dlagent. git-archive is usable only for file:// or git:// or git+ssh:// repositories.

--- scripts/git-dlagent.sh.in | 23 +++++++++++++++++------ 1 files changed, 17 insertions(+), 6 deletions(-) diff --git a/scripts/git-dlagent.sh.in b/scripts/git-dlagent.sh.in index a2240a7..993c7b5 100644 --- a/scripts/git-dlagent.sh.in +++ b/scripts/git-dlagent.sh.in @@ -26,7 +26,7 @@ then git_archive_method=local -elif [[ "${uri}" =~ ^(git\+)?(ssh|http|https|rsync)://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +elif [[ "${uri}" =~ ^(git\+)?ssh://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] then uri="${uri/%##*}" reference="${uri/#*#}" @@ -35,6 +35,15 @@ then git_archive_method=remote +elif [[ "${uri}" =~ ^(git\+)?(http|https|rsync)://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +then + uri="${uri/%##*}" + reference="${uri/#*#}" + uri="${uri/%#*}" + uri="${uri/#git+}" + + git_archive_method=remote-force + else echo "[ee] \`${uri}\` is not a Git URI." >&2 exit 1 @@ -50,12 +59,14 @@ case "${git_archive_method}" in exit 0 ;; - ( remote ) - - git archive --format=tar --output="${output}" --remote="${uri}" "${reference}" \ - && exit 0 + ( remote | remote-force ) - echo "[ww] \`${uri}\` does not support git-archive; going after clone." >&2 + if test "${git_archive_method}" == remote + then + git archive --format=tar --output="${output}" --remote="${uri}" "${reference}" \ + && exit 0 + echo "[ww] \`${uri}\` does not support git-archive; going after clone." >&2 + fi clone="$( mktemp -d )" -- 1.6.3.1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From 31a684f8fc561d3b4ef6dc20ec05b6089e771bbd Mon Sep 17 00:00:00 2001 From: Ciprian Dorin, Craciun Date: Tue, 20 Oct 2009 09:03:23 +0300 Subject: [PATCH 4/4] Fixed git-dlagent (failure to work without the `git+` prefix).

Modified the way the URL is interpretted: git+<url-part>#refspec##/<name>.tar And now the archive should contain a folder with the name <name>. (The `##/<name>.tar` is optional.) --- scripts/git-dlagent.sh.in | 29 +++++++++++++++++++++-------- 1 files changed, 21 insertions(+), 8 deletions(-) diff --git a/scripts/git-dlagent.sh.in b/scripts/git-dlagent.sh.in index 993c7b5..7b7ec24 100644 --- a/scripts/git-dlagent.sh.in +++ b/scripts/git-dlagent.sh.in @@ -9,25 +9,35 @@ output="${2}" git_archive_method=unknown -if [[ "${uri}" =~ ^git://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +if [[ "${uri}" =~ ^git://[^/#]+[^#]+#[^#]+(##/[a-zA-Z0-9_.-]+\.tar)?$ ]] then + + prefix="${uri/#*##\/}" + prefix="${prefix/%.tar}" uri="${uri/%##*}" reference="${uri/#*#}" uri="${uri/%#*}" git_archive_method=remote -elif [[ "${uri}" =~ ^(git\+)?file:///[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +elif [[ "${uri}" =~ ^(git\+)?file:///[^/#]+[^#]+#[^#]+(##/[a-zA-Z0-9_.-]+\.tar)?$ ]] then + + prefix="${uri/#*##\/}" + prefix="${prefix/%.tar}" uri="${uri/%##*}" reference="${uri/#*#}" uri="${uri/%#*}" - uri="${uri/#git+file:\/\/}" + uri="${uri/#git+}" + uri="${uri/#file:\/\/}" git_archive_method=local -elif [[ "${uri}" =~ ^(git\+)?ssh://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +elif [[ "${uri}" =~ ^(git\+)?ssh://[^/#]+[^#]+#[^#]+(##/[a-zA-Z0-9_.-]+\.tar)?$ ]] then + + prefix="${uri/#*##\/}" + prefix="${prefix/%.tar}" uri="${uri/%##*}" reference="${uri/#*#}" uri="${uri/%#*}" @@ -35,8 +45,11 @@ then git_archive_method=remote -elif [[ "${uri}" =~ ^(git\+)?(http|https|rsync)://[^/#]+[^#]+#[^#]+(##[a-zA-Z0-9_.-]+\.tar)?$ ]] +elif [[ "${uri}" =~ ^(git\+)?(http|https|rsync)://[^/#]+[^#]+#[^#]+(##/[a-zA-Z0-9_.-]+\.tar)?$ ]] then + + prefix="${uri/#*##\/}" + prefix="${prefix/%.tar}" uri="${uri/%##*}" reference="${uri/#*#}" uri="${uri/%#*}" @@ -54,7 +67,7 @@ case "${git_archive_method}" in ( local ) - git --git-dir="${uri}" archive --format=tar --output="${output}" "${reference}" + git --git-dir="${uri}" archive --format=tar --output="${output}" --prefix="${prefix}/" "${reference}" exit 0 ;; @@ -63,7 +76,7 @@ case "${git_archive_method}" in if test "${git_archive_method}" == remote then - git archive --format=tar --output="${output}" --remote="${uri}" "${reference}" \ + git archive --format=tar --output="${output}" --remote="${uri}" --prefix="${prefix}/" "${reference}" \ && exit 0 echo "[ww] \`${uri}\` does not support git-archive; going after clone." >&2 fi @@ -72,7 +85,7 @@ case "${git_archive_method}" in git clone --bare "${uri}" "${clone}" - git --git-dir="${clone}" archive --format=tar --output="${output}" "${reference}" + git --git-dir="${clone}" archive --format=tar --output="${output}" --prefix="${prefix}/" "${reference}" find "${clone}" -delete -- 1.6.3.1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~