[pacman-dev] [PATCH] makepkg: add --allsource option
This supplements the --source option and does nearly the same thing, except downloaded source files are included in the archive as well. This is mainly an RFC patch and interested parties are encouraged to comment. I was also planning on making a patch for adding the package/ prefix to the archived files, but apparently no archive program makes this easy to do with is just plain stupid. If someone knows how without creating a temporary dir and moving files there, let me know. Signed-off-by: Dan McGee <dan@archlinux.org> --- scripts/makepkg.sh.in | 32 ++++++++++++++++++++++++-------- 1 files changed, 24 insertions(+), 8 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 27da281..2c479fd 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -856,6 +856,15 @@ create_package() { create_srcpackage() { cd "$startdir" + if [ "$SOURCEONLY" = "2" ]; then + # get back to our src directory so we can begin with sources + mkdir -p "$srcdir" + cd "$srcdir" + download_sources + # we can only check checksums if we have all files + check_checksums + cd "$startdir" + fi msg "$(gettext "Creating source package...")" local comp_files="$BUILDSCRIPT" msg2 "$(gettext "Adding %s...")" "$BUILDSCRIPT" @@ -874,11 +883,15 @@ create_srcpackage() { comp_files="$comp_files ChangeLog" fi - local i - for i in ${source[@]}; do - if [ -f $i ]; then - msg2 "$(gettext "Adding %s...")" "$i" - comp_files="$comp_files $i" + local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$srcdir/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $srcdir $file" fi done @@ -1061,8 +1074,10 @@ SRCDEST=${SRCDEST:-$startdir} #default to $startdir if undefined # Parse Command Line Options. OPT_SHORT="AbcCdefFghiLmop:rRsSV" -OPT_LONG="ignorearch,asroot,builddeps,clean,cleancache,nodeps,noextract,force,forcever:,geninteg,help,holdver" -OPT_LONG="$OPT_LONG,install,log,nocolor,nobuild,rmdeps,repackage,source,syncdeps,usesudo,version" +OPT_LONG="allsource,asroot,ignorearch,builddeps,clean,cleancache,nodeps" +OPT_LONG="$OPT_LONG,noextract,force,forcever:,geninteg,help,holdver" +OPT_LONG="$OPT_LONG,install,log,nocolor,nobuild,rmdeps,repackage,source" +OPT_LONG="$OPT_LONG,syncdeps,usesudo,version" # Pacman Options OPT_LONG="$OPT_LONG,noconfirm,noprogressbar" OPT_TEMP="$(getopt -o "$OPT_SHORT" -l "$OPT_LONG" -n "$(basename "$0")" -- "$@" || echo 'GETOPT GO BANG!')" @@ -1080,6 +1095,7 @@ while true; do --noprogressbar) PACMAN_OPTS="$PACMAN_OPTS --noprogressbar" ;; # Makepkg Options + --allsource) SOURCEONLY=2 ;; --asroot) ASROOT=1 ;; -A|--ignorearch) IGNOREARCH=1 ;; -c|--clean) CLEANUP=1 ;; @@ -1316,7 +1332,7 @@ if [ $EUID -eq 0 ]; then fi # if we are creating a source-only package, go no further -if [ "$SOURCEONLY" = "1" ]; then +if [ "$SOURCEONLY" != "0" ]; then if [ -f "$PKGDEST/${pkgname}-${pkgver}-${pkgrel}${SRCEXT}" \ -a "$FORCE" = "0" ]; then error "$(gettext "A package has already been built. (use -f to overwrite)")" -- 1.5.5.3
On Sat, May 31, 2008 at 12:07:45PM -0500, Dan McGee <dan@archlinux.org> wrote:
This supplements the --source option and does nearly the same thing, except downloaded source files are included in the archive as well.
This is mainly an RFC patch and interested parties are encouraged to comment. I was also planning on making a patch for adding the package/ prefix to the archived files, but apparently no archive program makes this easy to do with is just plain stupid. If someone knows how without creating a temporary dir and moving files there, let me know.
i don't want to flame or anything, but what's the purpose of this? is there any program that would make use of such an archive? except that if you put it out to http then broken out files are browsable, while if you tar it up (like the rpm guys do with srpm) then the user has to download the whole source package, even if he/she is only interested in the buildscript or patches.
On Sat, May 31, 2008 at 3:47 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
On Sat, May 31, 2008 at 12:07:45PM -0500, Dan McGee <dan@archlinux.org> wrote:
This supplements the --source option and does nearly the same thing, except downloaded source files are included in the archive as well.
This is mainly an RFC patch and interested parties are encouraged to comment. I was also planning on making a patch for adding the package/ prefix to the archived files, but apparently no archive program makes this easy to do with is just plain stupid. If someone knows how without creating a temporary dir and moving files there, let me know.
i don't want to flame or anything, but what's the purpose of this? is there any program that would make use of such an archive? except that if you put it out to http then broken out files are browsable, while if you tar it up (like the rpm guys do with srpm) then the user has to download the whole source package, even if he/she is only interested in the buildscript or patches.
Yes there is use for this- to help people using makepkg comply with the GPL and providing source and the necessary tools to build it when they provide binary packages. I didn't remove the --source option on purpose; you can still generate a src.tar.gz package with the minimal files needed and none of the downloaded source. -Dan
On Sat, May 31, 2008 at 04:32:15PM -0500, Dan McGee <dpmcgee@gmail.com> wrote:
Yes there is use for this- to help people using makepkg comply with the GPL and providing source and the necessary tools to build it when they provide binary packages. I didn't remove the --source option on purpose; you can still generate a src.tar.gz package with the minimal files needed and none of the downloaded source.
ah, i see. it's your decision, but maybe it would worth considering what Gentoo and Frugalware does: just run a script from cron that extracts urls from buildscripts, checks for missing downloaded source tarballs, obtains them and you're done. http://git.frugalware.org/gitweb/gitweb.cgi?p=frugalware-current.git;a=blob;... of course this src.tar.gz approach may have other benefits i don't see..
Miklos Vajna wrote:
On Sat, May 31, 2008 at 04:32:15PM -0500, Dan McGee <dpmcgee@gmail.com> wrote:
Yes there is use for this- to help people using makepkg comply with the GPL and providing source and the necessary tools to build it when they provide binary packages. I didn't remove the --source option on purpose; you can still generate a src.tar.gz package with the minimal files needed and none of the downloaded source.
ah, i see. it's your decision, but maybe it would worth considering what Gentoo and Frugalware does: just run a script from cron that extracts urls from buildscripts, checks for missing downloaded source tarballs, obtains them and you're done.
http://git.frugalware.org/gitweb/gitweb.cgi?p=frugalware-current.git;a=blob;...
of course this src.tar.gz approach may have other benefits i don't see..
Don't only think of the for use by the main distro. There are plenty of people hosting a small custom repo that does not warrant such a cron job. Allan
On Sat, May 31, 2008 at 11:32 PM, Dan McGee <dpmcgee@gmail.com> wrote:
Yes there is use for this- to help people using makepkg comply with the GPL and providing source and the necessary tools to build it when they provide binary packages. I didn't remove the --source option on purpose; you can still generate a src.tar.gz package with the minimal files needed and none of the downloaded source.
So you are referring to this, right? : http://archlinux.org/pipermail/arch-dev-public/2008-May/006206.html I am still wondering if it would not be nicer and more efficient to just host the remote sources there. All pkgbuilds and patches are already available from abs or from the web interface. Only the remote sources are missing. Besides the same source is used for all revisions of a package. Please note that I am just thinking out loud here, I just want to be convinced about the usefulness of the --allsource option and of putting everything together instead of keeping things separated.
On Sat, May 31, 2008 at 10:47 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
i don't want to flame or anything, but what's the purpose of this? is there any program that would make use of such an archive? except that if you put it out to http then broken out files are browsable, while if you tar it up (like the rpm guys do with srpm) then the user has to download the whole source package, even if he/she is only interested in the buildscript or patches.
I agree, I also think that keeping the buildscript/patches (archlinux specific) and the real source files (upstream) separate is nicer and more flexible.
On Mon, Jun 2, 2008 at 4:26 AM, Xavier <shiningxc@gmail.com> wrote:
On Sat, May 31, 2008 at 10:47 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
i don't want to flame or anything, but what's the purpose of this? is there any program that would make use of such an archive? except that if you put it out to http then broken out files are browsable, while if you tar it up (like the rpm guys do with srpm) then the user has to download the whole source package, even if he/she is only interested in the buildscript or patches.
I agree, I also think that keeping the buildscript/patches (archlinux specific) and the real source files (upstream) separate is nicer and more flexible.
So what are you saying, did anyone actually look at the patch? I didn't change the existing --source option one bit, that *still* only includes local source files. All I did was add a completely new --allsource option that allows you to also build an src.tar.gz with all source, local and remote. How much more flexible can we get than having two options here? -Dan
On Mon, Jun 02, 2008 at 06:47:36AM -0500, Dan McGee <dpmcgee@gmail.com> wrote:
So what are you saying, did anyone actually look at the patch? I didn't change the existing --source option one bit, that *still* only includes local source files.
yes, i read that patch :)
All I did was add a completely new --allsource option that allows you to also build an src.tar.gz with all source, local and remote. How much more flexible can we get than having two options here?
i just said that creating such a huge srpm-like tarball is a bad idea, and encouraging people to do so by adding an option for that is not something i would recommend. just my two cents, though.
Miklos Vajna wrote:
On Mon, Jun 02, 2008 at 06:47:36AM -0500, Dan McGee <dpmcgee@gmail.com> wrote:
So what are you saying, did anyone actually look at the patch? I didn't change the existing --source option one bit, that *still* only includes local source files.
yes, i read that patch :)
All I did was add a completely new --allsource option that allows you to also build an src.tar.gz with all source, local and remote. How much more flexible can we get than having two options here?
i just said that creating such a huge srpm-like tarball is a bad idea, and encouraging people to do so by adding an option for that is not something i would recommend. just my two cents, though.
I see no problem with this. It is an optional extra. Nobody is being forced to use it or download the packages if someone else uses it. And it is actually useful for those hosting custom repos as they can quickly create a complete source package to host for GPL(2?) compliance. The only cost here is one extra if statement if you are using the --source option and another long option name to come up with an abbreviation for (if that idea goes ahead). I.e. essentially zero. Allan
On Sat, 31 May 2008 12:07:45 -0500 Dan McGee <dan@archlinux.org> wrote:
This is mainly an RFC patch and interested parties are encouraged to comment. I was also planning on making a patch for adding the package/ prefix to the archived files, but apparently no archive program makes this easy to do with is just plain stupid. If someone knows how without creating a temporary dir and moving files there, let me know.
I think you can use a symlink to make the one directory seem like another. At least it works on a little script I wrote for doing aurbuild releases. I don't need to cp or mv anything and I get a tarball with top-level directory based on the version number.
Dan McGee wrote:
<snip> + local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$srcdir/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $srcdir $file"
I'm a little confused here. The file $srcdir/$file is just a link to $SRCDEST/$file which may be $startdir/$file. So does this not just add the symbolic link to the tarball. Also, the -C option flags all following files to be from the $srcdir directory. So, anything added after this will also get its symbolic link added. If I'm not going down the wrong track here, this should fix it... + local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files -C $startdir $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$SRCDEST/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $SRCDEST $file"
On Mon, Jun 2, 2008 at 9:56 AM, Allan McRae <mcrae_allan@hotmail.com> wrote:
Dan McGee wrote:
<snip> + local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$srcdir/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $srcdir $file"
I'm a little confused here. The file $srcdir/$file is just a link to $SRCDEST/$file which may be $startdir/$file. So does this not just add the symbolic link to the tarball. Also, the -C option flags all following files to be from the $srcdir directory. So, anything added after this will also get its symbolic link added.
If I'm not going down the wrong track here, this should fix it...
Yeah, this wasn't tested super hard or anything so I think you are on to something here. The -C option is definitely a bit odd but it kind of does what we want. Since all files are available in src/ (although some might be symlinks), can't we use 1 -C flag in combination with -H to follow symlinks to our source files? While we are making changes, can you see an easy way to get everything under a pkgname/ dir (maybe with pkgver-pkgrel too) inside the tarball? Unfortunately bsdtar has no --prefix option to easily do this, and our friend Loui says it is doable but has not offered a line of code.
+ local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files -C $startdir $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$SRCDEST/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $SRCDEST $file"
_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://archlinux.org/mailman/listinfo/pacman-dev
Dan McGee wrote:
On Mon, Jun 2, 2008 at 9:56 AM, Allan McRae <mcrae_allan@hotmail.com> wrote:
Dan McGee wrote:
<snip> + local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$srcdir/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $srcdir $file"
I'm a little confused here. The file $srcdir/$file is just a link to $SRCDEST/$file which may be $startdir/$file. So does this not just add the symbolic link to the tarball. Also, the -C option flags all following files to be from the $srcdir directory. So, anything added after this will also get its symbolic link added.
If I'm not going down the wrong track here, this should fix it...
Yeah, this wasn't tested super hard or anything so I think you are on to something here. The -C option is definitely a bit odd but it kind of does what we want. Since all files are available in src/ (although some might be symlinks), can't we use 1 -C flag in combination with -H to follow symlinks to our source files?
The files are not in src/ if the --source was used so we have to be careful there.
While we are making changes, can you see an easy way to get everything under a pkgname/ dir (maybe with pkgver-pkgrel too) inside the tarball? Unfortunately bsdtar has no --prefix option to easily do this, and our friend Loui says it is doable but has not offered a line of code.
How about: - Make directory of form pkgname-pkgver-pkgrel in a temporary directory - Instead of collecting files in "comp_files", just place links into that directory. - Use bsdtar with -L flag on that directory. - Clean up I think that is along the lines of what Loui was suggesting. It also removes the whole -C issue without adding any further SOURCEONLY=2 checks. Allan
On Mon, Jun 2, 2008 at 10:56 AM, Allan McRae <mcrae_allan@hotmail.com> wrote:
Dan McGee wrote:
On Mon, Jun 2, 2008 at 9:56 AM, Allan McRae <mcrae_allan@hotmail.com> wrote:
Dan McGee wrote:
<snip> + local netfile + for netfile in ${source[@]}; do + local file=$(strip_url "$netfile") + if [ -f $netfile ]; then + msg2 "$(gettext "Adding %s...")" "$netfile" + comp_files="$comp_files $netfile" + elif [ "$SOURCEONLY" = "2" -a -f "$srcdir/$file" ]; then + msg2 "$(gettext "Adding %s...")" "$file" + comp_files="$comp_files -C $srcdir $file"
I'm a little confused here. The file $srcdir/$file is just a link to $SRCDEST/$file which may be $startdir/$file. So does this not just add the symbolic link to the tarball. Also, the -C option flags all following files to be from the $srcdir directory. So, anything added after this will also get its symbolic link added.
If I'm not going down the wrong track here, this should fix it...
Yeah, this wasn't tested super hard or anything so I think you are on to something here. The -C option is definitely a bit odd but it kind of does what we want. Since all files are available in src/ (although some might be symlinks), can't we use 1 -C flag in combination with -H to follow symlinks to our source files?
The files are not in src/ if the --source was used so we have to be careful there.
While we are making changes, can you see an easy way to get everything under a pkgname/ dir (maybe with pkgver-pkgrel too) inside the tarball? Unfortunately bsdtar has no --prefix option to easily do this, and our friend Loui says it is doable but has not offered a line of code.
How about: - Make directory of form pkgname-pkgver-pkgrel in a temporary directory - Instead of collecting files in "comp_files", just place links into that directory. - Use bsdtar with -L flag on that directory. - Clean up
I think that is along the lines of what Loui was suggesting. It also removes the whole -C issue without adding any further SOURCEONLY=2 checks.
Yeah, I think I like this idea. Feel free to rip my patch apart. :) We might want to take a closer look at how this is all done as the original --source patch was written quickly, and the --allsource option above was kind of jury-rigged in. I did like the fact that if we download all sources we can verify the checksums- perhaps we can still find a way to check md5sums for the local-only source as well before we package it up in an src.tar.gz? We should be able to do some kind of similar action: for file in source: if netfile && allsource then verify md5sum if localfile then verify md5sum Maybe add a flag or something to the checksum function to determine whether to check netfiles or not. Wow, this patch is going to turn into a sequence of 5. -Dan
On Tue, 03 Jun 2008 01:56:49 +1000 Allan McRae <mcrae_allan@hotmail.com> wrote:
Dan McGee wrote:
While we are making changes, can you see an easy way to get everything under a pkgname/ dir (maybe with pkgver-pkgrel too) inside the tarball? Unfortunately bsdtar has no --prefix option to easily do this, and our friend Loui says it is doable but has not offered a line of code.
I think that is along the lines of what Loui was suggesting. It also removes the whole -C issue without adding any further SOURCEONLY=2 checks.
I can't quite grok the makepkg source code so I'll just demonstrate what I do via a lil script. Enjoy! :D #!/bin/bash # Script for making source tarballs # $1 = directory name # $2 = release version if [ "$1" == "" ]; then echo "Usage: $0 <directory_name> <release_version>" exit fi echo "Creating temporary symlink $1-$2" ln -s $1 $1-$2 tar -cvvzf $1-$2.tar.gz $(find $1-$2/ -type f) echo "Removing symlink $1-$2" rm $1-$2 exit
participants (6)
-
Allan McRae
-
Dan McGee
-
Dan McGee
-
Loui
-
Miklos Vajna
-
Xavier