[pacman-dev] [patch] makepkg: update checksums (was: add checksums and resume)
This is in reference to http://bugs.archlinux.org/task/15051 and a somewhat-old http://mailman.archlinux.org/pipermail/pacman-dev/2008-March/005939.html :) The initial idea was actually not substantial enough and it also went on to continue the build, so here is another one updated with Allan's suggestion. * Do not resume (maybe a separate switch for that..and for another person another day) * Consider all supported checksums from buildscript (regardless of user's config) * Do not support buildscripts with checksums for more than one architecture The last one is something I had hoped I wouldn't end up with. It's technically possible, but visually ugly. Firstly, to override $CARCH one needs to do something before generate_checksums, and this doesn't appear to be as trivial as I'd thought. Then, there is simply no way to redirect a properly-formatted multi-arch checksum output. Echo'ing along with the function produces a huge number of white spaces (or tabs), though nothing harmful. Any deeper trick is simply not going to be worth it. Obviously may not be the best way to do this, so adapt as you like if it's worth implementing. Against current git. diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 2941a5d..d3d0d61 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -504,7 +504,6 @@ download_sources() { generate_checksums() { msg "$(gettext "Generating checksums for source files...")" - plain "" if [ ! $(type -p openssl) ]; then error "$(gettext "Cannot find openssl.")" @@ -523,6 +522,7 @@ generate_checksums() { local ct=0 local numsrc=${#source[@]} + echo echo -n "${integ}sums=(" local i=0; @@ -557,6 +557,45 @@ generate_checksums() { done } +update_checksums() { + local file="$startdir/$BUILDSCRIPT" + local integ + + # necessary override for --geninteg + unset INTEGRITY_CHECK + + for integ in md5 sha1 sha256 sha384 sha512; do + grep "${integ}sums=(" $file &> /dev/null && \ + INTEGRITY_CHECK=(${INTEGRITY_CHECK[@]} $integ) + done + + integ=${INTEGRITY_CHECK[0]} + if grep "&& ${integ}sums=(" "$file" | grep '$CARCH' &> /dev/null; then + error "$(gettext "Multi-arch checksum update not supported. Exiting...")" + exit 1 + else + for integ in ${INTEGRITY_CHECK[@]}; do + sed -i "/${integ}sums=(/,/)$/d" "$file" + done + + if grep "# vim: set" "$file" &> /dev/null; then + sed -i '/# vim: set/d' "$file" + sed -i -e :a -e '/^\n*$/{$d;N;ba' -e '}' "$file" + local vimset=1 + fi + + generate_checksums >> "$file" + + if [ $vimset -eq 1 ]; then + echo >> "$file" + echo "# vim: set ts=2 sw=2 et:" >> "$file" + fi + fi + + msg "$(gettext "Updated checksums in %s")" "$BUILDSCRIPT" + return 0 +} + check_checksums() { [ ${#source[@]} -eq 0 ] && return 0 @@ -1431,6 +1470,7 @@ usage() { echo "$(gettext " -e, --noextract Do not extract source files (use existing src/ dir)")" echo "$(gettext " -f, --force Overwrite existing package")" echo "$(gettext " -g, --geninteg Generate integrity checks for source files")" + printf "$(gettext " -G, --getinteg Update integrity checks in %s")\n" "$BUILDSCRIPT" echo "$(gettext " -h, --help This help")" echo "$(gettext " -i, --install Install package after successful build")" echo "$(gettext " -L, --log Log package build process")" @@ -1477,9 +1517,9 @@ fi ARGLIST=$@ # Parse Command Line Options. -OPT_SHORT="AcCdefFghiLmop:rRsV" +OPT_SHORT="AcCdefFgGhiLmop:rRsV" OPT_LONG="allsource,asroot,ignorearch,clean,cleancache,nodeps" -OPT_LONG="$OPT_LONG,noextract,force,forcever:,geninteg,help,holdver" +OPT_LONG="$OPT_LONG,noextract,force,forcever:,geninteg,getinteg,help,holdver" OPT_LONG="$OPT_LONG,install,log,nocolor,nobuild,rmdeps,repackage,skipinteg" OPT_LONG="$OPT_LONG,source,syncdeps,version,config:" # Pacman Options @@ -1512,6 +1552,7 @@ while true; do --forcever) shift; FORCE_VER=$1;; -F) INFAKEROOT=1 ;; -g|--geninteg) GENINTEG=1 ;; + -G| --getinteg) GETINTEG=1 ;; --holdver) HOLDVER=1 ;; -i|--install) INSTALL=1 ;; -L|--log) LOGGING=1 ;; @@ -1680,6 +1721,15 @@ if [ "$GENINTEG" -eq 1 ]; then exit 0 # $E_OK fi +if [ "$GETINTEG" -eq 1 ]; then + mkdir -p "$srcdir" + cd "$srcdir" + download_sources + update_checksums + cd "$startdir" + exit 0 +fi + if [ "$(type -t package)" = "function" ]; then PKGFUNC=1 fi
Oops, here it is attached in case.
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
On Mon, Oct 12, 2009 at 12:01 AM, Ray Rashif <schivmeister@gmail.com> wrote:
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
Allan, I know you are busy these days, so I can take a hack at reviewing this but I'd also like to hear what other people think of the idea in general. I know we just closed a feature request asking for this: http://bugs.archlinux.org/task/15809 But before I do that- Ray, can you please send the patch inline instead? Read doc/submitting-patches.txt for more details or see it here (http://www.archlinux.org/pacman/submitting-patches.html). And we only need the patch against git master; anything else isn't too relevant for actually getting pushed into the tree. -Dan
Dan McGee wrote:
On Mon, Oct 12, 2009 at 12:01 AM, Ray Rashif <schivmeister@gmail.com> wrote:
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
Allan, I know you are busy these days, so I can take a hack at reviewing this but I'd also like to hear what other people think of the idea in general. I know we just closed a feature request asking for this: http://bugs.archlinux.org/task/15809
There is a discussion going on in http://bugs.archlinux.org/task/15051 . Xavier has had a good look at this and posted an updated patch there. The remaining issue is PKGBUILDs with architecture specific checksums.
On Wed, Oct 14, 2009 at 10:51 PM, Allan McRae <allan@archlinux.org> wrote:
Dan McGee wrote:
On Mon, Oct 12, 2009 at 12:01 AM, Ray Rashif <schivmeister@gmail.com> wrote:
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
Allan, I know you are busy these days, so I can take a hack at reviewing this but I'd also like to hear what other people think of the idea in general. I know we just closed a feature request asking for this: http://bugs.archlinux.org/task/15809
There is a discussion going on in http://bugs.archlinux.org/task/15051 . Xavier has had a good look at this and posted an updated patch there. The remaining issue is PKGBUILDs with architecture specific checksums.
/me lets the discussion continue without him :P -Dan
On Thu, Oct 15, 2009 at 7:12 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Wed, Oct 14, 2009 at 10:51 PM, Allan McRae <allan@archlinux.org> wrote:
Dan McGee wrote:
On Mon, Oct 12, 2009 at 12:01 AM, Ray Rashif <schivmeister@gmail.com> wrote:
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
Allan, I know you are busy these days, so I can take a hack at reviewing this but I'd also like to hear what other people think of the idea in general. I know we just closed a feature request asking for this: http://bugs.archlinux.org/task/15809
There is a discussion going on in http://bugs.archlinux.org/task/15051 . Xavier has had a good look at this and posted an updated patch there. The remaining issue is PKGBUILDs with architecture specific checksums.
/me lets the discussion continue without him :P
Well the discussion was already almost over if you look at the comments :) The problem is that this requires some sed/awk skills and I have none at all. The sed commands proposed until now either did not work or failed in bad ways on some pkgbuilds, deleting parts of it. The awk way seemed more reliable than the sed ones : { rm PKGBUILD; awk '$0 ~ /^md5sums/ {i = 1; system("makepkg -g 2>/dev/null")}; !i {print}; $0 ~ /\)/ {i = 0}' > PKGBUILD; } < PKGBUILD But does it makes sense to incorporate this in makepkg ? To have makepkg call himself ?
Xavier wrote:
On Thu, Oct 15, 2009 at 7:12 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Wed, Oct 14, 2009 at 10:51 PM, Allan McRae <allan@archlinux.org> wrote:
Dan McGee wrote:
On Mon, Oct 12, 2009 at 12:01 AM, Ray Rashif <schivmeister@gmail.com> wrote:
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
Allan, I know you are busy these days, so I can take a hack at reviewing this but I'd also like to hear what other people think of the idea in general. I know we just closed a feature request asking for this: http://bugs.archlinux.org/task/15809
There is a discussion going on in http://bugs.archlinux.org/task/15051 . Xavier has had a good look at this and posted an updated patch there. The remaining issue is PKGBUILDs with architecture specific checksums.
/me lets the discussion continue without him :P
Well the discussion was already almost over if you look at the comments :)
The problem is that this requires some sed/awk skills and I have none at all. The sed commands proposed until now either did not work or failed in bad ways on some pkgbuilds, deleting parts of it.
The awk way seemed more reliable than the sed ones : { rm PKGBUILD; awk '$0 ~ /^md5sums/ {i = 1; system("makepkg -g 2>/dev/null")}; !i {print}; $0 ~ /\)/ {i = 0}' > PKGBUILD; } < PKGBUILD
But does it makes sense to incorporate this in makepkg ? To have makepkg call himself ?
makepkg already does call itself :P I obviously need to read the bug report.... but does the awk way get around the acrhitecture specific checksum issue?
2009/10/15 Allan McRae <allan@archlinux.org>
Xavier wrote:
On Thu, Oct 15, 2009 at 7:12 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Wed, Oct 14, 2009 at 10:51 PM, Allan McRae <allan@archlinux.org> wrote:
Dan McGee wrote:
On Mon, Oct 12, 2009 at 12:01 AM, Ray Rashif <schivmeister@gmail.com> wrote:
Fixed no-vimset bail, reverted plain "", fixed bail on no sums by default; source makepkg.
Allan, I know you are busy these days, so I can take a hack at reviewing this but I'd also like to hear what other people think of the idea in general. I know we just closed a feature request asking for this: http://bugs.archlinux.org/task/15809
There is a discussion going on in http://bugs.archlinux.org/task/15051. Xavier has had a good look at this and posted an updated patch there. The remaining issue is PKGBUILDs with architecture specific checksums.
/me lets the discussion continue without him :P
Well the discussion was already almost over if you look at the comments :)
The problem is that this requires some sed/awk skills and I have none at all. The sed commands proposed until now either did not work or failed in bad ways on some pkgbuilds, deleting parts of it.
The awk way seemed more reliable than the sed ones : { rm PKGBUILD; awk '$0 ~ /^md5sums/ {i = 1; system("makepkg -g 2>/dev/null")}; !i {print}; $0 ~ /\)/ {i = 0}' > PKGBUILD; } < PKGBUILD
But does it makes sense to incorporate this in makepkg ? To have makepkg call himself ?
makepkg already does call itself :P
I obviously need to read the bug report.... but does the awk way get around the acrhitecture specific checksum issue?
Indeed, the awk one doesn't do anything funny. But still, no solution to arch-specific sums. Unless of course you find a nice way to redirect and present it nicely in the PKGBUILD. And is why I thought it shouldn't be supported, albeit in a not-so-foolproof way. Mainly because it appeared to me that there weren't that many packages which would fail given that kind of treatment (we can of course back up PKGBUILD to /tmp or something): grep -r "&& md5sums=(" /var/abs | grep "/PKGBUILD:" | sed 's/:.*//' gives: /var/abs/community/jdk/PKGBUILD /var/abs/community/jdk/PKGBUILD /var/abs/community/jre/PKGBUILD /var/abs/community/jre/PKGBUILD /var/abs/community/libvdpau/PKGBUILD /var/abs/community/tuxguitar/PKGBUILD /var/abs/extra/nvidia-96xx-utils/PKGBUILD /var/abs/extra/nvidia-96xx/PKGBUILD /var/abs/extra/nvidia-173xx/PKGBUILD /var/abs/extra/nvidia/PKGBUILD /var/abs/extra/nxclient/PKGBUILD /var/abs/extra/nvidia-utils/PKGBUILD /var/abs/extra/nvidia-173xx-utils/PKGBUILD Total pkgs affected: 13 out of 4000+ (provided all PKGBUILDs are "standard", i.e "&&" is on the same line as ${integ}sums in this case integ=md5) Interestingly, it appears there are no multi-arch sums for the rest of the hash types, as far as poor grep'ing is concerned =/
On Thu, Oct 15, 2009 at 9:34 AM, Ray Rashif <schivmeister@gmail.com> wrote:
grep -r "&& md5sums=(" /var/abs | grep "/PKGBUILD:" | sed 's/:.*//' gives:
/var/abs/community/jdk/PKGBUILD /var/abs/community/jdk/PKGBUILD /var/abs/community/jre/PKGBUILD /var/abs/community/jre/PKGBUILD /var/abs/community/libvdpau/PKGBUILD /var/abs/community/tuxguitar/PKGBUILD /var/abs/extra/nvidia-96xx-utils/PKGBUILD /var/abs/extra/nvidia-96xx/PKGBUILD /var/abs/extra/nvidia-173xx/PKGBUILD /var/abs/extra/nvidia/PKGBUILD /var/abs/extra/nxclient/PKGBUILD /var/abs/extra/nvidia-utils/PKGBUILD /var/abs/extra/nvidia-173xx-utils/PKGBUILD
Total pkgs affected: 13 out of 4000+ (provided all PKGBUILDs are "standard", i.e "&&" is on the same line as ${integ}sums in this case integ=md5)
Making the assumption that pkgbuilds are standard is really looking for troubles and/or wrong and false results :) I don't want to support multi-arch sums, there is an infinite number of ways to write them, so it's impossible to deal with that. What I would find acceptable is just to delete the standard and easy checksums, without affecting anything else. And appending the new checksums at the end. Even better would be to replace the standard and easy checksums in place , and just append the others at the end of the pkgbuild.
On Thu, Oct 15, 2009 at 3:57 AM, Xavier <shiningxc@gmail.com> wrote:
On Thu, Oct 15, 2009 at 9:34 AM, Ray Rashif <schivmeister@gmail.com> wrote:
grep -r "&& md5sums=(" /var/abs | grep "/PKGBUILD:" | sed 's/:.*//' gives:
/var/abs/community/jdk/PKGBUILD /var/abs/community/jdk/PKGBUILD /var/abs/community/jre/PKGBUILD /var/abs/community/jre/PKGBUILD /var/abs/community/libvdpau/PKGBUILD /var/abs/community/tuxguitar/PKGBUILD /var/abs/extra/nvidia-96xx-utils/PKGBUILD /var/abs/extra/nvidia-96xx/PKGBUILD /var/abs/extra/nvidia-173xx/PKGBUILD /var/abs/extra/nvidia/PKGBUILD /var/abs/extra/nxclient/PKGBUILD /var/abs/extra/nvidia-utils/PKGBUILD /var/abs/extra/nvidia-173xx-utils/PKGBUILD
Total pkgs affected: 13 out of 4000+ (provided all PKGBUILDs are "standard", i.e "&&" is on the same line as ${integ}sums in this case integ=md5)
Making the assumption that pkgbuilds are standard is really looking for troubles and/or wrong and false results :)
I don't want to support multi-arch sums, there is an infinite number of ways to write them, so it's impossible to deal with that.
What I would find acceptable is just to delete the standard and easy checksums, without affecting anything else. And appending the new checksums at the end.
Even better would be to replace the standard and easy checksums in place , and just append the others at the end of the pkgbuild.
Just remember this- if we release a feature that is half-baked, we will pay for it. Look at optdepends and how many people think it is their right to have those parsed and we need options to install them and blah blah blah. People will try to use this for all varieties of md5sums. People will file bug reports when it doesn't work on crazy PKGBUILDs. And most of all, people will be pissed if we mess up their PKGBUILD in any way, even if it was their fault. -Dan
Xavier:
Even better would be to replace the standard and easy checksums in
place , and just append the others at the end of the pkgbuild.
Ahh I see. In that case what we need is an edit which won't touch the unsupported areas. But that would mean being able to somehow check for their existence. Maybe Allan can take a look at that.
Dan:
Just remember this- if we release a feature that is half-baked, we
will pay for it. Look at optdepends and how many people think it is their right to have those parsed and we need options to install them and blah blah blah.
People will try to use this for all varieties of md5sums. People will file bug reports when it doesn't work on crazy PKGBUILDs. And most of all, people will be pissed if we mess up their PKGBUILD in any way, even if it was their fault.
Yes, precisely so. Especially when there are actually requests to make it an "unsupported" feature in makepkg =p
On Thu, Oct 15, 2009 at 2:50 PM, Ray Rashif <schivmeister@gmail.com> wrote:
Even better would be to replace the standard and easy checksums in
place , and just append the others at the end of the pkgbuild.
Ahh I see. In that case what we need is an edit which won't touch the unsupported areas. But that would mean being able to somehow check for their existence. Maybe Allan can take a look at that.
That's not what I meant. I don't care much what happens with complex md5sums line, as long as ONLY the md5sums line are lost or broken and not the whole pkgbuild. It is amazing how many messages we exchanged (like 20 between here and the bug comments) for a completely null progress. I presented 3 technical problems : 1) sed -ne '1h;1!H;${;g;s/md5sums=([^)]*)/'$(makepkg -g 2>/dev/null)'/g;p;}' PKGBUILD sed: -e expression #1, char 73: unterminated `s' command 2) sed "/md5sums=([^)]*)/d" -> broken in some case 3) { rm PKGBUILD; awk '$0 ~ /^md5sums/ {i = 1; system("makepkg -g 2>/dev/null")}; !i {print}; $0 ~ /\)/ {i = 0}' > PKGBUILD; } < PKGBUILD -> cannot call an internal makepkg bash function (e.g: generate_checksums md5) If you have nothing to contribute and no alternative to propose, no need to write books about it :) We just won't implement the feature request, that's all.
2009/10/15 Xavier <shiningxc@gmail.com>
That's not what I meant. I don't care much what happens with complex md5sums line, as long as ONLY the md5sums line are lost or broken and not the whole pkgbuild.
It is amazing how many messages we exchanged (like 20 between here and the bug comments) for a completely null progress.
I presented 3 technical problems : 1) sed -ne '1h;1!H;${;g;s/md5sums=([^)]*)/'$(makepkg -g 2>/dev/null)'/g;p;}' PKGBUILD sed: -e expression #1, char 73: unterminated `s' command 2) sed "/md5sums=([^)]*)/d" -> broken in some case 3) { rm PKGBUILD; awk '$0 ~ /^md5sums/ {i = 1; system("makepkg -g 2>/dev/null")}; !i {print}; $0 ~ /\)/ {i = 0}' > PKGBUILD; } < PKGBUILD -> cannot call an internal makepkg bash function (e.g: generate_checksums md5)
If you have nothing to contribute and no alternative to propose, no need to write books about it :) We just won't implement the feature request, that's all.
My bad :) Mystery solved. Against your patched makepkg: @@ -587,7 +587,11 @@ local integlist=$(get_integlist) for integ in ${integlist[@]}; do - sed -i "/${integ}sums=(/,/)$/d" "$file" + if grep ${integ}sums "$file" | grep ")" &> /dev/null; then + sed -i "/${integ}sums=(.*)/d" "$file" + else + sed -i "/${integ}sums=(/,/)$/d" "$file" + fi done generate_checksums >> "$file" That sed only worked if the patterns were separated by newlines, so now we do two kinds. Not perfect, but at least meets your criterion.
On Thu, Oct 15, 2009 at 8:15 AM, Allan McRae <allan@archlinux.org> wrote:
makepkg already does call itself :P
I obviously need to read the bug report.... but does the awk way get around the acrhitecture specific checksum issue?
ahah, I fail ! of course makepkg does that already for the fakeroot stuff.. And yes awk works better. But another limitation of the awk way is that I cannot update each checksums one by one (because I need to call an internal makepkg function, generate_checksums, with a parameter, and I couldn't do that by calling makepkg). By the way, the issue with sed is not exactly the architecture specific stuff, I mean it is just a side-effect of it. It is just that the proposed sed command fails when writing several md5sums line in a non-standard way (e.g. using carch=... && md5sums=...) I still did not figure out why sed removed more than what it should in these cases, and which pattern exactly triggers the bug
participants (4)
-
Allan McRae
-
Dan McGee
-
Ray Rashif
-
Xavier