[pacman-dev] [PATCH] makepkg : allow to specify a download filename
A source entry can now have the following form, to specify a different filename : "filename::http://path/to/file" Of course, the old syntax is still supported : "http://path/to/file" And as before, in the second case, the filename used is simply "file". This fixes FS#11292, because handling multiple source files with the same name is now possible (just choose a different filename). But it will also allow to deal much more nicely with funny url like this by using a sane filename (and unfortunately, there are quite a few) : http://www.vim.org/scripts/download_script.php?src_id=6992 Signed-off-by: Xavier Chantry <shiningxc@gmail.com> --- doc/PKGBUILD.5.txt | 4 ++++ scripts/makepkg.sh.in | 35 ++++++++++++++++++++++++----------- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/doc/PKGBUILD.5.txt b/doc/PKGBUILD.5.txt index cf6eb23..743eba1 100644 --- a/doc/PKGBUILD.5.txt +++ b/doc/PKGBUILD.5.txt @@ -87,6 +87,10 @@ similar to `$_basekernver`. variables if possible when specifying the download location. Any files that are compressed will automatically be extracted, unless found in the noextract array listed below. + + + It is possible to specify an optional filename, which is practical with + some weird url, and also for handling multiple source files with the same + name : `source=('filename::url')` *noextract (array)*:: An array of filenames corresponding to those from the source array. Files diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index ddf6cd8..d30713f 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -163,11 +163,23 @@ trap 'trap_exit "$(gettext "TERM signal caught. Exiting...")"' TERM HUP QUIT trap 'trap_exit "$(gettext "Aborted by user! Exiting...")"' INT trap 'trap_exit "$(gettext "An unknown error has occured. Exiting...")"' ERR - -strip_url() { - echo "$1" | sed 's|^.*://.*/||g' +# a source entry can have two forms : +# 1) "filename::http://path/to/file" +# 2) "http://path/to/file" + +# extract the filename from a source entry +get_filename() { + # if a filename is specified, use it + local filename=$(echo $1 | sed 's|::.*||') + # if it is just an url, we only keep the last component + echo "$filename" | sed 's|^.*://.*/||g' } +# extract the url from a source entry +get_url() { + # strip an eventual filename + echo $1 | sed 's|.*::||' +} ## # Checks to see if options are present in makepkg.conf or PKGBUILD; @@ -418,7 +430,8 @@ download_sources() { local netfile for netfile in "${source[@]}"; do - local file=$(strip_url "$netfile") + local file=$(get_filename "$netfile") + local url=$(get_url "$netfile") if [ -f "$startdir/$file" ]; then msg2 "$(gettext "Found %s in build dir")" "$file" rm -f "$srcdir/$file" @@ -432,18 +445,18 @@ download_sources() { fi # if we get here, check to make sure it was a URL, else fail - if [ "$file" = "$netfile" ]; then + if [ "$file" = "$url" ]; then error "$(gettext "%s was not found in the build directory and is not a URL.")" "$netfile" exit 1 # $E_MISSING_FILE fi # find the client we should use for this URL - local dlclient=$(get_downloadclient "$netfile") || exit $? + local dlclient=$(get_downloadclient "$url") || exit $? msg2 "$(gettext "Downloading %s...")" "$file" # fix flyspray bug #3289 local ret=0 - download_file "$dlclient" "$netfile" "$file" || ret=$? + download_file "$dlclient" "$url" "$file" || ret=$? if [ $ret -gt 0 ]; then error "$(gettext "Failure while downloading %s")" "$file" plain "$(gettext "Aborting...")" @@ -491,7 +504,7 @@ generate_checksums() { local netfile for netfile in "${source[@]}"; do - local file="$(strip_url "$netfile")" + local file="$(get_filename "$netfile")" if [ ! -f "$file" ] ; then if [ ! -f "$SRCDEST/$file" ] ; then @@ -529,7 +542,7 @@ check_checksums() { local idx=0 local file for file in "${source[@]}"; do - file="$(strip_url "$file")" + file="$(get_filename "$file")" echo -n " $file ... " >&2 if [ ! -f "$file" ] ; then @@ -574,7 +587,7 @@ extract_sources() { msg "$(gettext "Extracting Sources...")" local netfile for netfile in "${source[@]}"; do - file=$(strip_url "$netfile") + file=$(get_filename "$netfile") if in_array "$file" ${noextract[@]}; then #skip source files in the noextract=() array # these are marked explicitly to NOT be extracted @@ -957,7 +970,7 @@ create_srcpackage() { local netfile for netfile in "${source[@]}"; do - local file=$(strip_url "$netfile") + local file=$(get_filename "$netfile") if [ -f "$netfile" ]; then msg2 "$(gettext "Adding %s...")" "$netfile" ln -s "${startdir}/$netfile" "${srclinks}/${pkgname}" -- 1.6.0
On Sat, Aug 23, 2008 at 7:11 PM, Xavier Chantry <shiningxc@gmail.com> wrote:
A source entry can now have the following form, to specify a different filename : "filename::http://path/to/file"
Of course, the old syntax is still supported : "http://path/to/file"
And as before, in the second case, the filename used is simply "file".
This fixes FS#11292, because handling multiple source files with the same name is now possible (just choose a different filename).
But it will also allow to deal much more nicely with funny url like this by using a sane filename (and unfortunately, there are quite a few) : http://www.vim.org/scripts/download_script.php?src_id=6992
Signed-off-by: Xavier Chantry <shiningxc@gmail.com> --- doc/PKGBUILD.5.txt | 4 ++++ scripts/makepkg.sh.in | 35 ++++++++++++++++++++++++----------- 2 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/doc/PKGBUILD.5.txt b/doc/PKGBUILD.5.txt index cf6eb23..743eba1 100644 --- a/doc/PKGBUILD.5.txt +++ b/doc/PKGBUILD.5.txt @@ -87,6 +87,10 @@ similar to `$_basekernver`. variables if possible when specifying the download location. Any files that are compressed will automatically be extracted, unless found in the noextract array listed below. + + + It is possible to specify an optional filename, which is practical with + some weird url, and also for handling multiple source files with the same + name : `source=('filename::url')` It is also possible to specify an optional filename, which is helpful with weird URLs and for handling multiple source files with the same name. The syntax is: <example>
'also' helps it flow; 'practical' just isn't that common of a word to use in this situation; URL is capitalized elsewhere in this manpage and plural makes a bit more sense. Finally, I thought splitting the example into its own sentence helps keep it from being a run-on.
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index ddf6cd8..d30713f 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -163,11 +163,23 @@ trap 'trap_exit "$(gettext "TERM signal caught. Exiting...")"' TERM HUP QUIT trap 'trap_exit "$(gettext "Aborted by user! Exiting...")"' INT trap 'trap_exit "$(gettext "An unknown error has occured. Exiting...")"' ERR
- -strip_url() { - echo "$1" | sed 's|^.*://.*/||g' +# a source entry can have two forms : +# 1) "filename::http://path/to/file" +# 2) "http://path/to/file" + +# extract the filename from a source entry +get_filename() { + # if a filename is specified, use it + local filename=$(echo $1 | sed 's|::.*||') + # if it is just an url, we only keep the last component + echo "$filename" | sed 's|^.*://.*/||g' }
+# extract the url from a source entry +get_url() { + # strip an eventual filename + echo $1 | sed 's|.*::||' +}
## # Checks to see if options are present in makepkg.conf or PKGBUILD; @@ -418,7 +430,8 @@ download_sources() {
local netfile for netfile in "${source[@]}"; do - local file=$(strip_url "$netfile") + local file=$(get_filename "$netfile") + local url=$(get_url "$netfile") if [ -f "$startdir/$file" ]; then msg2 "$(gettext "Found %s in build dir")" "$file" rm -f "$srcdir/$file" @@ -432,18 +445,18 @@ download_sources() { fi
# if we get here, check to make sure it was a URL, else fail - if [ "$file" = "$netfile" ]; then + if [ "$file" = "$url" ]; then error "$(gettext "%s was not found in the build directory and is not a URL.")" "$netfile" exit 1 # $E_MISSING_FILE fi
# find the client we should use for this URL - local dlclient=$(get_downloadclient "$netfile") || exit $? + local dlclient=$(get_downloadclient "$url") || exit $?
msg2 "$(gettext "Downloading %s...")" "$file" # fix flyspray bug #3289 local ret=0 - download_file "$dlclient" "$netfile" "$file" || ret=$? + download_file "$dlclient" "$url" "$file" || ret=$? if [ $ret -gt 0 ]; then error "$(gettext "Failure while downloading %s")" "$file" plain "$(gettext "Aborting...")" @@ -491,7 +504,7 @@ generate_checksums() {
local netfile for netfile in "${source[@]}"; do - local file="$(strip_url "$netfile")" + local file="$(get_filename "$netfile")"
if [ ! -f "$file" ] ; then if [ ! -f "$SRCDEST/$file" ] ; then @@ -529,7 +542,7 @@ check_checksums() { local idx=0 local file for file in "${source[@]}"; do - file="$(strip_url "$file")" + file="$(get_filename "$file")" echo -n " $file ... " >&2
if [ ! -f "$file" ] ; then @@ -574,7 +587,7 @@ extract_sources() { msg "$(gettext "Extracting Sources...")" local netfile for netfile in "${source[@]}"; do - file=$(strip_url "$netfile") + file=$(get_filename "$netfile") if in_array "$file" ${noextract[@]}; then #skip source files in the noextract=() array # these are marked explicitly to NOT be extracted @@ -957,7 +970,7 @@ create_srcpackage() {
local netfile for netfile in "${source[@]}"; do - local file=$(strip_url "$netfile") + local file=$(get_filename "$netfile") if [ -f "$netfile" ]; then msg2 "$(gettext "Adding %s...")" "$netfile" ln -s "${startdir}/$netfile" "${srclinks}/${pkgname}" --
Looks great, although if we want to get this in maint, it would be best to get further testing from those who use makepkg daily. -Dan
On Sun, Aug 24, 2008 at 2:25 AM, Dan McGee <dpmcgee@gmail.com> wrote:
It is also possible to specify an optional filename, which is helpful with weird URLs and for handling multiple source files with the same name. The syntax is: <example>
'also' helps it flow; 'practical' just isn't that common of a word to use in this situation; URL is capitalized elsewhere in this manpage and plural makes a bit more sense. Finally, I thought splitting the example into its own sentence helps keep it from being a run-on.
Thanks, it is all fixed.
Looks great, although if we want to get this in maint, it would be best to get further testing from those who use makepkg daily.
Hm, what about posting a makepkg version from maint (so with all the bugfixes) + that patch to aur-general ML?
On Sat, Aug 23, 2008 at 7:40 PM, Xavier <shiningxc@gmail.com> wrote:
On Sun, Aug 24, 2008 at 2:25 AM, Dan McGee <dpmcgee@gmail.com> wrote:
Looks great, although if we want to get this in maint, it would be best to get further testing from those who use makepkg daily.
Hm, what about posting a makepkg version from maint (so with all the bugfixes) + that patch to aur-general ML?
Sounds good to me- I'll let you handle this as I'm not on that list. Hopefully you know how to build it. :P -Dan
A source entry can now have the following form, to specify a different filename : "filename::http://path/to/file" Of course, the old syntax is still supported : "http://path/to/file" And as before, in the second case, the filename used is simply "file". This fixes FS#11292, because handling multiple source files with the same name is now possible (just choose a different filename). But it will also allow to deal much more nicely with funny url like this by using a sane filename (and unfortunately, there are quite a few) : http://www.vim.org/scripts/download_script.php?src_id=6992 Signed-off-by: Xavier Chantry <shiningxc@gmail.com> --- doc/PKGBUILD.5.txt | 4 +++ scripts/makepkg.sh.in | 65 ++++++++++++++++++++++++++++++++++--------------- 2 files changed, 49 insertions(+), 20 deletions(-) diff --git a/doc/PKGBUILD.5.txt b/doc/PKGBUILD.5.txt index 0b1ce64..c63b3ec 100644 --- a/doc/PKGBUILD.5.txt +++ b/doc/PKGBUILD.5.txt @@ -87,6 +87,10 @@ similar to `$_basekernver`. variables if possible when specifying the download location. Any files that are compressed will automatically be extracted, unless found in the noextract array listed below. + + + It is also possible to specify an optional filename, which is helpful + with weird URLs and for handling multiple source files with the same + name. The syntax is: `source=('filename::url')` *noextract (array)*:: An array of filenames corresponding to those from the source array. Files diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index d127d16..1d70f39 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -163,11 +163,23 @@ trap 'trap_exit "$(gettext "TERM signal caught. Exiting...")"' TERM HUP QUIT trap 'trap_exit "$(gettext "Aborted by user! Exiting...")"' INT trap 'trap_exit "$(gettext "An unknown error has occured. Exiting...")"' ERR - -strip_url() { - echo "$1" | sed 's|^.*://.*/||g' +# a source entry can have two forms : +# 1) "filename::http://path/to/file" +# 2) "http://path/to/file" + +# extract the filename from a source entry +get_filename() { + # if a filename is specified, use it + local filename=$(echo $1 | sed 's|::.*||') + # if it is just an url, we only keep the last component + echo "$filename" | sed 's|^.*://.*/||g' } +# extract the url from a source entry +get_url() { + # strip an eventual filename + echo $1 | sed 's|.*::||' +} ## # Checks to see if options are present in makepkg.conf or PKGBUILD; @@ -284,18 +296,33 @@ get_downloadclient() { } download_file() { + # download command local dlcmd=$1 - local netfile=$2 + # url of the file + local url=$2 + # destination file local file=$3 + # temporary download file, default to last component of the url + local dlfile=$(echo "$url" | sed 's|^.*://.*/||g') - if echo "$dlcmd" | grep -q "%u" ; then + # replace %o by the temporary dlfile if it exists + if echo "$dlcmd" | grep -q "%o" ; then dlcmd=${dlcmd//%o/$file.part} - dlcmd=${dlcmd//%u/$netfile} + dlfile="$file.part" + fi + # add the url, either in place of %u or at the end + if echo "$dlcmd" | grep -q "%u" ; then + dlcmd=${dlcmd//%u/$url} else - dlcmd="$dlcmd $netfile" + dlcmd="$dlcmd $url" fi - $dlcmd + $dlcmd || return $? + + # rename the temporary download file to the final destination + if [ "$dlfile" != "$file" ]; then + mv -f "$SRCDEST/$dlfile" "$SRCDEST/$file" + fi } check_deps() { @@ -418,7 +445,8 @@ download_sources() { local netfile for netfile in "${source[@]}"; do - local file=$(strip_url "$netfile") + local file=$(get_filename "$netfile") + local url=$(get_url "$netfile") if [ -f "$startdir/$file" ]; then msg2 "$(gettext "Found %s in build dir")" "$file" rm -f "$srcdir/$file" @@ -432,26 +460,23 @@ download_sources() { fi # if we get here, check to make sure it was a URL, else fail - if [ "$file" = "$netfile" ]; then - error "$(gettext "%s was not found in the build directory and is not a URL.")" "$netfile" + if [ "$file" = "$url" ]; then + error "$(gettext "%s was not found in the build directory and is not a URL.")" "$file" exit 1 # $E_MISSING_FILE fi # find the client we should use for this URL - local dlclient=$(get_downloadclient "$netfile") || exit $? + local dlclient=$(get_downloadclient "$url") || exit $? msg2 "$(gettext "Downloading %s...")" "$file" # fix flyspray bug #3289 local ret=0 - download_file "$dlclient" "$netfile" "$file" || ret=$? + download_file "$dlclient" "$url" "$file" || ret=$? if [ $ret -gt 0 ]; then error "$(gettext "Failure while downloading %s")" "$file" plain "$(gettext "Aborting...")" exit 1 fi - if echo "$dlclient" | grep -q "%o" ; then - mv -f "$SRCDEST/$file.part" "$SRCDEST/$file" - fi rm -f "$srcdir/$file" ln -s "$SRCDEST/$file" "$srcdir/" done @@ -491,7 +516,7 @@ generate_checksums() { local netfile for netfile in "${source[@]}"; do - local file="$(strip_url "$netfile")" + local file="$(get_filename "$netfile")" if [ ! -f "$file" ] ; then if [ ! -f "$SRCDEST/$file" ] ; then @@ -537,7 +562,7 @@ check_checksums() { local idx=0 local file for file in "${source[@]}"; do - file="$(strip_url "$file")" + file="$(get_filename "$file")" echo -n " $file ... " >&2 if [ ! -f "$file" ] ; then @@ -576,7 +601,7 @@ extract_sources() { msg "$(gettext "Extracting Sources...")" local netfile for netfile in "${source[@]}"; do - file=$(strip_url "$netfile") + file=$(get_filename "$netfile") if in_array "$file" ${noextract[@]}; then #skip source files in the noextract=() array # these are marked explicitly to NOT be extracted @@ -959,7 +984,7 @@ create_srcpackage() { local netfile for netfile in "${source[@]}"; do - local file=$(strip_url "$netfile") + local file=$(get_filename "$netfile") if [ -f "$netfile" ]; then msg2 "$(gettext "Adding %s...")" "$netfile" ln -s "${startdir}/$netfile" "${srclinks}/${pkgname}" -- 1.6.0
Xavier Chantry wrote:
A source entry can now have the following form, to specify a different filename : "filename::http://path/to/file"
Of course, the old syntax is still supported : "http://path/to/file"
And as before, in the second case, the filename used is simply "file".
This fixes FS#11292, because handling multiple source files with the same name is now possible (just choose a different filename).
But it will also allow to deal much more nicely with funny url like this by using a sane filename (and unfortunately, there are quite a few) : http://www.vim.org/scripts/download_script.php?src_id=6992
Signed-off-by: Xavier Chantry <shiningxc@gmail.com> --- doc/PKGBUILD.5.txt | 4 +++ scripts/makepkg.sh.in | 65 ++++++++++++++++++++++++++++++++++--------------- 2 files changed, 49 insertions(+), 20 deletions(-)
Looks good. A missing %o in DLAGENTS did need to be supported as the makepkg.conf man pages says that it is optional. As a side note, the description of %u being replaced by the url if present or the url being placed at the end needs tidied up in the man page as it currently sounds like putting the url at the end will alway happen. Allan
On Sun, Aug 24, 2008 at 3:42 PM, Allan McRae <allan@archlinux.org> wrote:
Looks good. A missing %o in DLAGENTS did need to be supported as the makepkg.conf man pages says that it is optional. As a side note, the description of %u being replaced by the url if present or the url being placed at the end needs tidied up in the man page as it currently sounds like putting the url at the end will alway happen.
What about this : DLAGENTS=('protocol::/path/to/command [options]' …) Sets the download agents used to fetch source files specified with a URL in the PKGBUILD(5) file. Options can be specified for each command as well, and any protocol can have a download agent. Several examples are provided in the default makepkg.conf. If present, instances of %u will be replaced with the download URL. Otherwise, the download URL will be placed on the end of the command. If present, instances of %o will be replaced with the local filename, plus a ".part" extension, which allows to do file resumes properly.
Xavier wrote:
On Sun, Aug 24, 2008 at 3:42 PM, Allan McRae <allan@archlinux.org> wrote:
Looks good. A missing %o in DLAGENTS did need to be supported as the makepkg.conf man pages says that it is optional. As a side note, the description of %u being replaced by the url if present or the url being placed at the end needs tidied up in the man page as it currently sounds like putting the url at the end will alway happen.
What about this :
DLAGENTS=('protocol::/path/to/command [options]' …)
Sets the download agents used to fetch source files specified with a URL in the PKGBUILD(5) file. Options can be specified for each command as well, and any protocol can have a download agent. Several examples are provided in the default makepkg.conf. If present, instances of %u will be replaced with the download URL. Otherwise, the download URL will be placed on the end of the command. If present, instances of %o will be replaced with the local filename, plus a ".part" extension, which allows to do file resumes properly.
Slight changes here. I don't like "instances of" as that implies there can be multiple occurrences of each. Also a tidy up of the grammar at the end. I would prefer not to use "If present" twice as it does not read well, but can not think of a better way at this time of night... If present, %u will be replaced with the download URL. Otherwise, the download URL will be placed on the end of the command. If present, %o will be replaced with the local filename, plus a ".part" extension, which allows makepkg to handle resuming file downloads.
On Sun, Aug 24, 2008 at 4:45 PM, Allan McRae <allan@archlinux.org> wrote:
Slight changes here. I don't like "instances of" as that implies there can be multiple occurrences of each. Also a tidy up of the grammar at the end. I would prefer not to use "If present" twice as it does not read well, but can not think of a better way at this time of night...
If present, %u will be replaced with the download URL. Otherwise, the download URL will be placed on the end of the command. If present, %o will be replaced with the local filename, plus a ".part" extension, which allows makepkg to handle resuming file downloads.
Thanks, I applied your corrections. I agree that "if present" twice is not so smooth but well... Not a big deal.
On Sun, Aug 24, 2008 at 2:45 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Sat, Aug 23, 2008 at 7:40 PM, Xavier <shiningxc@gmail.com> wrote:
On Sun, Aug 24, 2008 at 2:25 AM, Dan McGee <dpmcgee@gmail.com> wrote:
Looks great, although if we want to get this in maint, it would be best to get further testing from those who use makepkg daily.
Hm, what about posting a makepkg version from maint (so with all the bugfixes) + that patch to aur-general ML?
Sounds good to me- I'll let you handle this as I'm not on that list. Hopefully you know how to build it. :P
This testing turned out to be useful. This new filename feature did not work with the old DLAGENTS array without %o / %u. It turned out it was pretty straightforward to have this feature working both with the old and new DLAGENTS, mostly by reworking the download_file function. I did quite a few testing with all combinations for that new patch, and it went well. I will submit an updated version on aur-general just in case.
participants (4)
-
Allan McRae
-
Dan McGee
-
Xavier
-
Xavier Chantry