[arch-dev-public] [Patch] makechrootpkg: fixed support for splitted package
Hi, Here (inline and attached) are a couple of patches for makechrootpkg. ==> Remove error message when namcap is not installed in chroot (not sure why this line was added as installing namcap in chroot will dirty it). Also added a check on the PKGBUILD. --- makechrootpkg.orig 2009-08-06 20:07:21.000000000 -0400 +++ makechrootpkg 2009-08-06 23:43:24.000000000 -0400 @@ -220,7 +220,7 @@ export HOME=/build sudo -u nobody makepkg $MAKEPKG_ARGS || touch BUILD_FAILED [ -f BUILD_FAILED ] && exit 1 -which namcap 2>&1 >/dev/null && namcap /pkgdest/*${PKGEXT} > /pkgdest/namcap.log +which namcap 2>&1 &>/dev/null && namcap /build/PKGBUILD /pkgdest/*${PKGEXT} > /pkgdest/namcap.log exit 0 EOF ) > "$uniondir/chrootbuild" ==> Move all built packages in the PKGDEST or WORKDIR (pkgname is now an array). I also did some fixes because using the * between quotes doesn't work. @@ -229,25 +229,28 @@ if mkarchroot -r "/chrootbuild" "$uniondir"; then source ${WORKDIR}/PKGBUILD - pkgfile=${chrootdir}/union/pkgdest/${pkgname}-${pkgver}-${pkgrel}-*${PKGEXT} + for _pkgname in ${pkgname[@]}; do + pkgfile="${chrootdir}"/union/pkgdest/${_pkgname}-${pkgver}-${pkgrel}-*${PKGEXT} + + if [ -n "$add_to_db" -a -e "$pkgfile" ]; then + [ -d "${chrootdir}/union/repo" ] || mkdir -p "${chrootdir}/union/repo" + pushd "${chrootdir}/union/repo" >/dev/null + cp "$pkgfile" . + repo-add repo.db.tar.${DB_COMPRESSION} ${_pkgname}-${pkgver}-${pkgrel}-*${PKGEXT} + popd >/dev/null + fi + + if [ -e $pkgfile ]; then + if [ -n "$PKGDEST" ]; then + echo "Moving completed ${_pkgname} package file to ${PKGDEST}" + mv $pkgfile "${PKGDEST}" + else + echo "Moving completed ${_pkgname} package file to ${WORKDIR}" + mv $pkgfile "${WORKDIR}" + fi + fi + done - if [ -n "$add_to_db" -a -e "$pkgfile" ]; then - [ -d "${chrootdir}/union/repo" ] || mkdir -p "${chrootdir}/union/repo" - pushd "${chrootdir}/union/repo" >/dev/null - cp "$pkgfile" . - repo-add repo.db.tar.${DB_COMPRESSION} "${pkgname}-${pkgver}-${pkgrel}-*${PKGEXT}" - popd >/dev/null - fi - - if [ -e "$pkgfile" ]; then - if [ -n "$PKGDEST" ]; then - echo "Moving completed package file to ${PKGDEST}" - mv "$pkgfile" "${PKGDEST}" - else - echo "Moving completed package file to ${WORKDIR}" - mv "$pkgfile" "${WORKDIR}" - fi - fi for f in ${chrootdir}/union/srcdest/*; do [ -e "$f" ] || continue if [ -n "$SRCDEST" ]; then
Eric Bélanger wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
==> Remove error message when namcap is not installed in chroot (not sure why this line was added as installing namcap in chroot will dirty it). Also added a check on the PKGBUILD.
This was apparently added as namcap has issues checking a package for which the dependencies are not installed on your system. The deps will have obvioulsy been installed into the chroot. Saying that, I agree that namcap (and thus python) should not be in a clean chroot and I just deleted that line in the version I am using. Allan
On Fri, Aug 7, 2009 at 1:31 AM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
==> Remove error message when namcap is not installed in chroot (not sure why this line was added as installing namcap in chroot will dirty it). Also added a check on the PKGBUILD.
This was apparently added as namcap has issues checking a package for which the dependencies are not installed on your system. The deps will have obvioulsy been installed into the chroot. Saying that, I agree that namcap (and thus python) should not be in a clean chroot and I just deleted that line in the version I am using.
That's correct. I can remove it, but I find it a little... ugly any other way. The line should actually not do anything if namcap is missing (which namcap should return an error if it is not installed)
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) : --- /usr/sbin/makechrootpkg 2009-08-06 23:44:28.000000000 -0400 +++ makechrootpkg 2009-08-09 21:57:39.000000000 -0400 @@ -230,7 +230,7 @@ source ${WORKDIR}/PKGBUILD for _pkgname in ${pkgname[@]}; do - pkgfile="${chrootdir}"/union/pkgdest/${_pkgname}-${pkgver}-${pkgrel}-*${PKGEXT} + pkgfile="${chrootdir}"/union/pkgdest/${_pkgname}-*${PKGEXT} if [ -n "$add_to_db" -a -e "$pkgfile" ]; then [ -d "${chrootdir}/union/repo" ] || mkdir -p "${chrootdir}/union/repo"
Eric Bélanger wrote:
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) :
Be careful there.... man PKGBUILD: All options and directives for the split packages default to the global values given within the PKGBUILD. However, some of these can be overridden within each split package’s packaging function. The following variables can be overridden: pkgdesc, license, groups, depends, optdepends, provides, conflicts, replaces, backup, options and install. So overriding pkgver and pkgrel may not work as expected. Allan
On Sun, Aug 9, 2009 at 10:08 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) :
Be careful there.... man PKGBUILD:
All options and directives for the split packages default to the global values given within the PKGBUILD. However, some of these can be overridden within each split package’s packaging function. The following variables can be overridden: pkgdesc, license, groups, depends, optdepends, provides, conflicts, replaces, backup, options and install.
So overriding pkgver and pkgrel may not work as expected.
Allan
Thanks for the warning, I wasn't aware about that. From the package's name and the .PKGINFO it seems to work, at least for different pkgver. However, I haven't tested the packages yet. Eric
Eric Bélanger wrote:
On Sun, Aug 9, 2009 at 10:08 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) :
Be careful there.... man PKGBUILD:
All options and directives for the split packages default to the global values given within the PKGBUILD. However, some of these can be overridden within each split package’s packaging function. The following variables can be overridden: pkgdesc, license, groups, depends, optdepends, provides, conflicts, replaces, backup, options and install.
So overriding pkgver and pkgrel may not work as expected.
Allan
Thanks for the warning, I wasn't aware about that. From the package's name and the .PKGINFO it seems to work, at least for different pkgver. However, I haven't tested the packages yet.
My suspicion is that it will work when the package with the different pkgver/pkgrel is last. makepkg just does not revert those variables after exiting the package function. Are there other use cases like this? That functionality could be added... Allan
On Sun, Aug 9, 2009 at 10:33 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Sun, Aug 9, 2009 at 10:08 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) :
Be careful there.... man PKGBUILD:
All options and directives for the split packages default to the global values given within the PKGBUILD. However, some of these can be overridden within each split package’s packaging function. The following variables can be overridden: pkgdesc, license, groups, depends, optdepends, provides, conflicts, replaces, backup, options and install.
So overriding pkgver and pkgrel may not work as expected.
Allan
Thanks for the warning, I wasn't aware about that. From the package's name and the .PKGINFO it seems to work, at least for different pkgver. However, I haven't tested the packages yet.
My suspicion is that it will work when the package with the different pkgver/pkgrel is last. makepkg just does not revert those variables after exiting the package function. Are there other use cases like this? That functionality could be added...
Allan
Actually I used it the other way around: pkgname=('lvm2' 'device-mapper') pkgver=2.02.51 _pkgverdm=1.02.36 _pkgverlvm=${pkgver} pkgrel=1 . . . package_device-mapper() { pkgver=${_pkgverdm} ... } package_lvm2() { ... } In my knowledge, only lvm2 and device-mapper use different pkgver. But it might be usefull to be able to change these variables too, especially pkgrel. This will let us fix and upload only the affected packages. Otherwise, if we change the pkgrel of a PKGBUILD, we need to uploads and force the users to update all the packages that are built with that PKGBUILD even if they aren't affected by the fix.
Eric Bélanger wrote:
On Sun, Aug 9, 2009 at 10:33 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Sun, Aug 9, 2009 at 10:08 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Hi,
Here (inline and attached) are a couple of patches for makechrootpkg.
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) :
Be careful there.... man PKGBUILD:
All options and directives for the split packages default to the global values given within the PKGBUILD. However, some of these can be overridden within each split package’s packaging function. The following variables can be overridden: pkgdesc, license, groups, depends, optdepends, provides, conflicts, replaces, backup, options and install.
So overriding pkgver and pkgrel may not work as expected.
Allan
Thanks for the warning, I wasn't aware about that. From the package's name and the .PKGINFO it seems to work, at least for different pkgver. However, I haven't tested the packages yet.
My suspicion is that it will work when the package with the different pkgver/pkgrel is last. makepkg just does not revert those variables after exiting the package function. Are there other use cases like this? That functionality could be added...
Allan
Actually I used it the other way around:
pkgname=('lvm2' 'device-mapper')
makepkg uses the order specified in pkgname not the order of the package_*() functions. So in this case you are fine.
In my knowledge, only lvm2 and device-mapper use different pkgver. But it might be usefull to be able to change these variables too, especially pkgrel. This will let us fix and upload only the affected packages. Otherwise, if we change the pkgrel of a PKGBUILD, we need to uploads and force the users to update all the packages that are built with that PKGBUILD even if they aren't affected by the fix.
Adding this would be quite easy. I had never found a use case for different pkgvers until now so ad never considered it. No idea why I had not considered pkgrels before. Allan
On Mon, Aug 10, 2009 at 2:23 AM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Sun, Aug 9, 2009 at 10:33 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Sun, Aug 9, 2009 at 10:08 PM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Fri, Aug 7, 2009 at 2:19 AM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
> > Hi, > > Here (inline and attached) are a couple of patches for makechrootpkg. > > > >
Here's a second patch. It fixes the copying of the package from the chroot when the pkgver or pkgrel are different (e.g. device-mapper & lvm2) :
Be careful there.... man PKGBUILD:
All options and directives for the split packages default to the global values given within the PKGBUILD. However, some of these can be overridden within each split package’s packaging function. The following variables can be overridden: pkgdesc, license, groups, depends, optdepends, provides, conflicts, replaces, backup, options and install.
So overriding pkgver and pkgrel may not work as expected.
Allan
Thanks for the warning, I wasn't aware about that. From the package's name and the .PKGINFO it seems to work, at least for different pkgver. However, I haven't tested the packages yet.
My suspicion is that it will work when the package with the different pkgver/pkgrel is last. makepkg just does not revert those variables after exiting the package function. Are there other use cases like this? That functionality could be added...
Allan
Actually I used it the other way around:
pkgname=('lvm2' 'device-mapper')
makepkg uses the order specified in pkgname not the order of the package_*() functions. So in this case you are fine.
Specifiyng the wanted pkgver and pkgrel in the package_*() also works BTW.
In my knowledge, only lvm2 and device-mapper use different pkgver. But it might be usefull to be able to change these variables too, especially pkgrel. This will let us fix and upload only the affected packages. Otherwise, if we change the pkgrel of a PKGBUILD, we need to uploads and force the users to update all the packages that are built with that PKGBUILD even if they aren't affected by the fix.
Adding this would be quite easy. I had never found a use case for different pkgvers until now so ad never considered it. No idea why I had not considered pkgrels before.
Allan
I merged and pushed the split package support for makechrootpkg. Regarding the namcap issue, I did it a little differently. For some reason I always forget the order: >/dev/null 2>&1 #works fine 2>&1 >/dev/null #does not However, I just looked up your change in man bash... &>/dev/null is equal to >/dev/null 2>&1 So I fixed that in the next patch
On Wed, Aug 12, 2009 at 4:53 PM, Aaron Griffin<aaronmgriffin@gmail.com> wrote:
I merged and pushed the split package support for makechrootpkg.
Regarding the namcap issue, I did it a little differently. For some reason I always forget the order: >/dev/null 2>&1 #works fine 2>&1 >/dev/null #does not
However, I just looked up your change in man bash... &>/dev/null is equal to >/dev/null 2>&1 So I fixed that in the next patch
Thanks. Another small patch to remove a superfluous parenthesis: --- makechrootpkg 2009-08-13 03:28:12.000000000 -0400 +++ /usr/sbin/makechrootpkg 2009-08-13 03:29:03.000000000 -0400 @@ -254,10 +254,10 @@ for f in ${chrootdir}/union/srcdest/*; do [ -e "$f" ] || continue if [ -n "$SRCDEST" ]; then - echo "Moving downloaded source file ($(basename $f) to ${SRCDEST}" + echo "Moving downloaded source file $(basename $f) to ${SRCDEST}" mv "$f" "${SRCDEST}" else - echo "Moving downloaded source file ($(basename $f) to ${WORKDIR}" + echo "Moving downloaded source file $(basename $f) to ${WORKDIR}" mv "$f" "${WORKDIR}" fi done
Am Thu, 13 Aug 2009 03:34:29 -0400 schrieb Eric Bélanger <snowmaniscool@gmail.com>:
On Wed, Aug 12, 2009 at 4:53 PM, Aaron Griffin<aaronmgriffin@gmail.com> wrote:
I merged and pushed the split package support for makechrootpkg.
Regarding the namcap issue, I did it a little differently. For some reason I always forget the order: >/dev/null 2>&1 #works fine 2>&1 >/dev/null #does not
However, I just looked up your change in man bash... &>/dev/null is equal to >/dev/null 2>&1 So I fixed that in the next patch
Thanks. Another small patch to remove a superfluous parenthesis:
--- makechrootpkg 2009-08-13 03:28:12.000000000 -0400 +++ /usr/sbin/makechrootpkg 2009-08-13 03:29:03.000000000 -0400 @@ -254,10 +254,10 @@ for f in ${chrootdir}/union/srcdest/*; do [ -e "$f" ] || continue if [ -n "$SRCDEST" ]; then - echo "Moving downloaded source file ($(basename $f) to ${SRCDEST}" + echo "Moving downloaded source file $(basename $f) to ${SRCDEST}" mv "$f" "${SRCDEST}" else - echo "Moving downloaded source file ($(basename $f) to ${WORKDIR}" + echo "Moving downloaded source file $(basename $f) to ${WORKDIR}" mv "$f" "${WORKDIR}" fi done
Moving stuff from SRCDEST made on my system always copying several GBs to builddir and afair back. No idea why it pulls all stuff from SRCDEST into the builddir and later back. At least this happened on my system. Anybody else using a SRCDEST? And support for scp'ing all splitted packages via testingpkg/extrapkg in one step would be nice. -Andy
On Thu, Aug 13, 2009 at 3:10 PM, Andreas Radke<a.radke@arcor.de> wrote:
Am Thu, 13 Aug 2009 03:34:29 -0400 schrieb Eric Bélanger <snowmaniscool@gmail.com>:
On Wed, Aug 12, 2009 at 4:53 PM, Aaron Griffin<aaronmgriffin@gmail.com> wrote:
I merged and pushed the split package support for makechrootpkg.
Regarding the namcap issue, I did it a little differently. For some reason I always forget the order: >/dev/null 2>&1 #works fine 2>&1 >/dev/null #does not
However, I just looked up your change in man bash... &>/dev/null is equal to >/dev/null 2>&1 So I fixed that in the next patch
Thanks. Another small patch to remove a superfluous parenthesis:
--- makechrootpkg 2009-08-13 03:28:12.000000000 -0400 +++ /usr/sbin/makechrootpkg 2009-08-13 03:29:03.000000000 -0400 @@ -254,10 +254,10 @@ for f in ${chrootdir}/union/srcdest/*; do [ -e "$f" ] || continue if [ -n "$SRCDEST" ]; then - echo "Moving downloaded source file ($(basename $f) to ${SRCDEST}" + echo "Moving downloaded source file $(basename $f) to ${SRCDEST}" mv "$f" "${SRCDEST}" else - echo "Moving downloaded source file ($(basename $f) to ${WORKDIR}" + echo "Moving downloaded source file $(basename $f) to ${WORKDIR}" mv "$f" "${WORKDIR}" fi done
Moving stuff from SRCDEST made on my system always copying several GBs to builddir and afair back. No idea why it pulls all stuff from SRCDEST into the builddir and later back. At least this happened on my system.
Anybody else using a SRCDEST?
Use the makechrootpkg from git. I only copies the files in the PKGBUILD source array.
And support for scp'ing all splitted packages via testingpkg/extrapkg in one step would be nice.
Yes. If no one does it, I might check what still needs to be done to the devtools for the splitted/any support as doing these things manually is starting be tedious. I have a splitted package update coming up so it might be a good test. Eric
-Andy
On Thu, Aug 13, 2009 at 4:10 PM, Eric Bélanger<snowmaniscool@gmail.com> wrote:
Moving stuff from SRCDEST made on my system always copying several GBs to builddir and afair back. No idea why it pulls all stuff from SRCDEST into the builddir and later back. At least this happened on my system.
Anybody else using a SRCDEST?
Use the makechrootpkg from git. I only copies the files in the PKGBUILD source array.
I meant: It only copies the files in the PKGBUILD source array.
On Mon, Aug 10, 2009 at 2:23 AM, Allan McRae<allan@archlinux.org> wrote:
Adding this would be quite easy. I had never found a use case for different pkgvers until now so ad never considered it. No idea why I had not considered pkgrels before.
Could you also do this for the arch variable? Pierre told me that, currently, using 'any' arch in split packages doesn't work. I haven't had the time to see what prevents this (makepkg? devtools? dbscripts?) but I have a hunch that not having the arch variable overridden might play a part in that. A such fonctionality will be useful to provide big API docs in a seperate 'any' package. Another thing (I can submit a FR in flyspray if you prefer) that would be nice: being able to build specific packages from a splitted PKGBUILD. I think that makepkg should build all packages by default and have an option that allows us to specify the desired packages to be built.
Eric Bélanger wrote:
On Mon, Aug 10, 2009 at 2:23 AM, Allan McRae<allan@archlinux.org> wrote:
Adding this would be quite easy. I had never found a use case for different pkgvers until now so ad never considered it. No idea why I had not considered pkgrels before.
Could you also do this for the arch variable? Pierre told me that, currently, using 'any' arch in split packages doesn't work. I haven't had the time to see what prevents this (makepkg? devtools? dbscripts?) but I have a hunch that not having the arch variable overridden might play a part in that. A such fonctionality will be useful to provide big API docs in a seperate 'any' package.
I'd say all of the above probably need fixed for this. I have had a brief look at makepkg support. It is a bit weird as the arch variable is handled differently to all others but I think I can fix it. It is on my list of things to do. Can you please file bug reports and assign to me for allowing different pkgver/pkgrel and arch lines so I do not loose track of them.
Another thing (I can submit a FR in flyspray if you prefer) that would be nice: being able to build specific packages from a splitted PKGBUILD. I think that makepkg should build all packages by default and have an option that allows us to specify the desired packages to be built.
That would be fairly easy to implement although the actual package stages should be really fast compared to the build one. Please file a bug report. Allan
On Sun, Aug 16, 2009 at 7:35 AM, Allan McRae<allan@archlinux.org> wrote:
Eric Bélanger wrote:
On Mon, Aug 10, 2009 at 2:23 AM, Allan McRae<allan@archlinux.org> wrote:
Adding this would be quite easy. I had never found a use case for different pkgvers until now so ad never considered it. No idea why I had not considered pkgrels before.
Could you also do this for the arch variable? Pierre told me that, currently, using 'any' arch in split packages doesn't work. I haven't had the time to see what prevents this (makepkg? devtools? dbscripts?) but I have a hunch that not having the arch variable overridden might play a part in that. A such fonctionality will be useful to provide big API docs in a seperate 'any' package.
I'd say all of the above probably need fixed for this. I have had a brief look at makepkg support. It is a bit weird as the arch variable is handled differently to all others but I think I can fix it. It is on my list of things to do.
Can you please file bug reports and assign to me for allowing different pkgver/pkgrel and arch lines so I do not loose track of them.
Another thing (I can submit a FR in flyspray if you prefer) that would be nice: being able to build specific packages from a splitted PKGBUILD. I think that makepkg should build all packages by default and have an option that allows us to specify the desired packages to be built.
That would be fairly easy to implement although the actual package stages should be really fast compared to the build one. Please file a bug report.
I filed bug report for both of these. About the specific packages build: You need to keep in mind that some of these splitted packages provides a lot of packages (e.g. kdegames) and that the packages functions can run time consuming commands. Also, I am supposed to submit a patch for split package support for commitpkg (I might do so before having a split pkg ready to test it). I'll make it so it only scp the packages present in trunk so we won't need to upload the packages that we don't update. If makepkg doesn't build the packages that we don't need, that'll save us from having to remove it. Eric
Allan
participants (4)
-
Aaron Griffin
-
Allan McRae
-
Andreas Radke
-
Eric Bélanger