Hi everyone, What's the current recommended way to support multiple mirrors for the same source file in a PKGBUILD? For example, I have xyz-123.tar.gz that can be downloaded from any of "http://a.example.com/xyz-123.tar.gz" " http://b.example.com/xyz-123.tar.gz" "http://c.example.com/xyz-123.tar.gz%22...but one of these may be down at any given time.
If there's no support for this in the sources array itself, we should strongly consider adding it. For example, an ad-hoc way:
_pick_source_mirror() { # ... do something smart in here } source=( "xyz-123::$(_pick_source_mirror( http://a.example.com/xyz-123.tar.gz http://b.example.com/xyz-123.tar.gz http://c.example.com/xyz-123.tar.gz ))" )
A less ad-hoc way: source() { #...some code that returns a list of URLs/files }
Another way: source=(...files without mirrors as usual...) sourcemirrored=( xyz-123::($mirroraURL $mirrorbURL $mirrorcURL) )
Mirrored sources seem like a common enough phenomenon that it's worth special treatment to make the source PKGBUILD variable more versatile if they're not currently supported. Thoughts?
Ido
On Thu, Oct 31, 2013 at 5:46 PM, Ido Rosen ido@kernel.org wrote:
Hi everyone, What's the current recommended way to support multiple mirrors for the same source file in a PKGBUILD? For example, I have xyz-123.tar.gz that can be downloaded from any of "http://a.example.com/xyz-123.tar.gz" " http://b.example.com/xyz-123.tar.gz" "http://c.example.com/xyz-123.tar.gz%22...but one of these may be down at any given time.
If there's no support for this in the sources array itself, we should strongly consider adding it. For example, an ad-hoc way:
_pick_source_mirror() { # ... do something smart in here } source=( "xyz-123::$(_pick_source_mirror( http://a.example.com/xyz-123.tar.gz http://b.example.com/xyz-123.tar.gz http://c.example.com/xyz-123.tar.gz ))" )
A less ad-hoc way: source() { #...some code that returns a list of URLs/files }
Another way: source=(...files without mirrors as usual...) sourcemirrored=( xyz-123::($mirroraURL $mirrorbURL $mirrorcURL) )
Mirrored sources seem like a common enough phenomenon that it's worth special treatment to make the source PKGBUILD variable more versatile if they're not currently supported. Thoughts?
Ido
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
That would be good too, don't know why I didn't think about that simple solution. :-)
On Thu, Oct 31, 2013 at 4:32 PM, Jerome Leclanche adys.wh@gmail.com wrote:
On Thu, Oct 31, 2013 at 5:46 PM, Ido Rosen ido@kernel.org wrote:
Hi everyone, What's the current recommended way to support multiple mirrors for the same source file in a PKGBUILD? For example, I have xyz-123.tar.gz that can be downloaded from any of "http://a.example.com/xyz-123.tar.gz" " http://b.example.com/xyz-123.tar.gz" "http://c.example.com/xyz-123.tar.gz%22...but one of these may be down at any given time.
If there's no support for this in the sources array itself, we should strongly consider adding it. For example, an ad-hoc way:
_pick_source_mirror() { # ... do something smart in here } source=( "xyz-123::$(_pick_source_mirror( http://a.example.com/xyz-123.tar.gz http://b.example.com/xyz-123.tar.gz http://c.example.com/xyz-123.tar.gz ))" )
A less ad-hoc way: source() { #...some code that returns a list of URLs/files }
Another way: source=(...files without mirrors as usual...) sourcemirrored=( xyz-123::($mirroraURL $mirrorbURL $mirrorcURL) )
Mirrored sources seem like a common enough phenomenon that it's worth special treatment to make the source PKGBUILD variable more versatile if they're not currently supported. Thoughts?
Ido
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
This can't work. If the first source fails to download, makepkg will abort.
On Thu, Oct 31, 2013 at 4:45 PM, Dave Reisner d@falconindy.com wrote:
On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
This can't work. If the first source fails to download, makepkg will abort.
What about just detecting if source is an array/list (keep current behavior), or a dictionary/lookup table (new fancy mirror behavior)? If it's a dictionary/lookup table, then try all mirrors before failing a source download.
For examples of how to create lookup tables in bash, see http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash...
On Thu, Oct 31, 2013 at 4:51 PM, Ido Rosen ido@kernel.org wrote:
On Thu, Oct 31, 2013 at 4:45 PM, Dave Reisner d@falconindy.com wrote:
On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
This can't work. If the first source fails to download, makepkg will abort.
What about just detecting if source is an array/list (keep current behavior), or a dictionary/lookup table (new fancy mirror behavior)? If it's a dictionary/lookup table, then try all mirrors before failing a source download.
For examples of how to create lookup tables in bash, see http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash...
Just to be clear, I meant a dictionary/lookup table of strings or list values, e.g.:
source=( ["file.tar.gz"]=("http://a.com/file.tar.gz" " http://b.com/file.tar.gz") ) # would indicate file.tar.gz has 2 mirror URLs.
On Thu, Oct 31, 2013 at 04:51:38PM -0400, Ido Rosen wrote:
On Thu, Oct 31, 2013 at 4:45 PM, Dave Reisner d@falconindy.com wrote:
On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
This can't work. If the first source fails to download, makepkg will abort.
What about just detecting if source is an array/list (keep current behavior), or a dictionary/lookup table (new fancy mirror behavior)? If it's a dictionary/lookup table, then try all mirrors before failing a source download.
For examples of how to create lookup tables in bash, see http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash...
bash4's associative arrays define a 1:1 mapping of string to string. You cannot have multiple elements in the "value." Not to mention that we won't be changing the format of the 'source' variable any time soon.
On Thu, Oct 31, 2013 at 5:04 PM, Dave Reisner d@falconindy.com wrote:
On Thu, Oct 31, 2013 at 04:51:38PM -0400, Ido Rosen wrote:
On Thu, Oct 31, 2013 at 4:45 PM, Dave Reisner d@falconindy.com wrote:
On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:
What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"
J. Leclanche
This can't work. If the first source fails to download, makepkg will abort.
What about just detecting if source is an array/list (keep current behavior), or a dictionary/lookup table (new fancy mirror behavior)? If it's a dictionary/lookup table, then try all mirrors before failing a source download.
For examples of how to create lookup tables in bash, see
http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash. ..
bash4's associative arrays define a 1:1 mapping of string to string. You cannot have multiple elements in the "value." Not to mention that we won't be changing the format of the 'source' variable any time soon.
Okay, that leaves either Jerome's suggestion (if duplicate filenames are detected, treat them as mirrors and don't fail/abort as long as one of them works); supporting source() as a function instead of just as an array; or creating a new variable that can handle mirrors...or doing nothing.
On 31.10.2013 22:12, Ido Rosen wrote:
Okay, that leaves either Jerome's suggestion (if duplicate filenames are detected, treat them as mirrors and don't fail/abort as long as one of them works);
I'd go with that. Doesn't add any new syntax and can be created easily enough like this:
source=("$pkgname-$pkgver.tar.gz"::{"http://foo.com/blub/%22,%22http://somewhere.com/%22%7D%22$pkgname-$pkgver.ta...")
Should be as simple as looping of the sources array after a fail and checking if there is another matching "filename::" element.
Care to submit a patch to pacman-dev?
On Thu, Oct 31, 2013 at 6:18 PM, Florian Pritz bluewind@xinu.at wrote:
On 31.10.2013 22:12, Ido Rosen wrote:
Okay, that leaves either Jerome's suggestion (if duplicate filenames are detected, treat them as mirrors and don't fail/abort as long as one of
them
works);
I'd go with that. Doesn't add any new syntax and can be created easily enough like this:
source=("$pkgname-$pkgver.tar.gz"::{"http://foo.com/blub/"," http://somewhere.com/%22%7D%22$pkgname-$pkgver.tar.gz")
Should be as simple as looping of the sources array after a fail and checking if there is another matching "filename::" element.
Care to submit a patch to pacman-dev?
It looks like a major structural change, since the exit 1 is deep inside each download_{file,git,bzr,...} function definition in makepkg.sh.in in the pacman git repo: For example the one for download_file() in makepkg.sh.in line 379 would probably move up to the download_sources() function, and return 1 instead of exit 1, then the return code would be checked, etc...
If my first pacman patch is going to be this big of a change to makepkg.sh.in I'd feel a lot more comfortable if an existing pacman/makepkg dev were available off-list via email or IM for some brain-picking. Any volunteers?
Ido
On 31.10.2013 23:30, Ido Rosen wrote:
If my first pacman patch is going to be this big of a change to makepkg.sh.in I'd feel a lot more comfortable if an existing pacman/makepkg dev were available off-list via email or IM for some brain-picking. Any volunteers?
Got a poc for a rather simple way of implementing it, not sure if it's going to be pretty enough.
I'll take a closer look tomorrow.
Feel free to ping me via irc, mail or jabber [1].
Here is my POC. It should work for remote files (URLs), local files, VCS repos, and across different types (e.g. if you want to offer git and hg mirrors of a repo).
Totally untested. Feel free to change it if it's broken and take all the credit and submit to pacman-dev. :)
Ido
On Thu, Oct 31, 2013 at 7:20 PM, Florian Pritz bluewind@xinu.at wrote:
On 31.10.2013 23:30, Ido Rosen wrote:
If my first pacman patch is going to be this big of a change to makepkg.sh.in I'd feel a lot more comfortable if an existing
pacman/makepkg
dev were available off-list via email or IM for some brain-picking. Any volunteers?
Got a poc for a rather simple way of implementing it, not sure if it's going to be pretty enough.
I'll take a closer look tomorrow.
Feel free to ping me via irc, mail or jabber [1].
Oops, looks like the attachment didn't make it. Here's the patch pasted:
From bf4b5a6b531e2e9ec629907426ddf854735c649e Mon Sep 17 00:00:00 2001
From: Ido Rosen code@idorosen.com Date: Thu, 31 Oct 2013 19:50:46 -0400 Subject: [PATCH] Added mirror support to makepkg's source array.
To specify multiple mirrors, simply add multiple files to the source array that have the same downloaded filename, e.g.:
source=("file.tar.gz::http://mirror1.example.com/file.tar.gz" "file.tar.gz::http://mirror2.example.com/file.tar.gz")
...makepkg will try them all, and if all fail, will abort. This also applies to VCS repositories and local files references in the source array, so for example:
source=("git+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git" "git+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git")
...will allow cloning the git repository from GitHub if kernel.org is down or fails. --- scripts/makepkg.sh.in | 73 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 28 deletions(-)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 4cb8173..2fe43cf 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -326,7 +326,7 @@ download_local() { else local filename=$(get_filename "$netfile") error "$(gettext "%s was not found in the build directory and is not a URL.")" "$filename" - exit 1 # $E_MISSING_FILE + return 1 # $E_MISSING_FILE fi }
@@ -375,8 +375,7 @@ download_file() { if (( ret )); then [[ ! -s $dlfile ]] && rm -f -- "$dlfile" error "$(gettext "Failure while downloading %s")" "$filename" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi
# rename the temporary download file to the final destination @@ -460,8 +459,7 @@ download_bzr() { msg2 "$(gettext "Branching %s ...")" "${displaylocation}" if ! bzr branch "$url" "$dir" --no-tree --use-existing-dir; then error "$(gettext "Failure while branching %s")" "${displaylocation}" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then # Make sure we are fetching the right repo @@ -470,15 +468,13 @@ download_bzr() { if [[ -n $distant_url ]]; then if [[ $distant_url != "$local_url" ]]; then error "$(gettext "%s is not a branch of %s")" "$dir" "$url" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi else if [[ $url != "$local_url" ]] ; then error "$(gettext "%s is not a branch of %s")" "$dir" "$url" error "$(gettext "The local URL is %s")" "$local_url" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi fi msg2 "$(gettext "Pulling %s ...")" "${displaylocation}" @@ -545,16 +541,14 @@ download_git() { msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git" if ! git clone --mirror "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then cd_safe "$dir" # Make sure we are fetching the right repo if [[ "$url" != "$(git config --get remote.origin.url)" ]] ; then error "$(gettext "%s is not a clone of %s")" "$dir" "$url" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git" if ! git fetch --all -p; then @@ -634,8 +628,7 @@ download_hg() { msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "hg" if ! hg clone -U "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "hg" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "hg" @@ -673,15 +666,13 @@ extract_hg() { ;; *) error "$(gettext "Unrecognized reference: %s")" "${fragment}" - plain "$(gettext "Aborting...")" - exit 1 + return 1 esac fi
if ! hg clone "${ref[@]}" "$dir" "${dir##*/}"; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "hg" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi
popd &>/dev/null @@ -711,8 +702,7 @@ download_svn() { mkdir -p "$dir/.makepkg" if ! svn checkout --config-dir "$dir/.makepkg" "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "svn" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "svn" @@ -776,34 +766,61 @@ download_sources() { GET_VCS=0 fi
+ declare -A downloaded # local since no -g supplied to declare. + declare -A expected # local since no -g supplied to declare. local netfile for netfile in "${source[@]}"; do pushd "$SRCDEST" &>/dev/null + local filepath=$(get_filepath "${netfile}") + expected["${filepath}"]=1 + if [[ ${downloaded["${filepath}"]} ]]; then + msg "$(gettext "Skipping mirror: %s")" "$netfile" + continue # file already downloaded, current mirror is redundant. + fi
local proto=$(get_protocol "$netfile") case "$proto" in local) - download_local "$netfile" + download_local "$netfile" && \ + downloaded["${filepath}"]=1 ;; bzr*) - (( GET_VCS )) && download_bzr "$netfile" + (( GET_VCS )) && download_bzr "$netfile" && \ + downloaded["${filepath}"]=1 ;; git*) - (( GET_VCS )) && download_git "$netfile" + (( GET_VCS )) && download_git "$netfile" && \ + downloaded["${filepath}"]=1 ;; hg*) - (( GET_VCS )) && download_hg "$netfile" + (( GET_VCS )) && download_hg "$netfile" && \ + downloaded["${filepath}"]=1 ;; svn*) - (( GET_VCS )) && download_svn "$netfile" + (( GET_VCS )) && download_svn "$netfile" && \ + downloaded["${filepath}"]=1 ;; *) - download_file "$netfile" + download_file "$netfile" && \ + downloaded["${filepath}"]=1 ;; esac - popd &>/dev/null done + + local failed=() + for expectedfilepath in "${expected[@]}"; do + if [[ ${downloaded["${expectedfilepath}"]} ]]; then + continue + else + failed=(${failed[@]} "${expectedfilepath}") + error "$(gettext "Fatal failure while downloading file %s")" "${expectedfilepath}" + fi + done + if [[ ${#failed[@]} -gt 0 ]]; then + plain "$(gettext "Aborting...")" + exit 1 + fi }
# Automatically update pkgver variable if a pkgver() function is provided
On Thu, Oct 31, 2013 at 01:46:07PM -0400, Ido Rosen wrote:
Hi everyone, What's the current recommended way to support multiple mirrors for the same source file in a PKGBUILD? For example, I have xyz-123.tar.gz that can be downloaded from any of "http://a.example.com/xyz-123.tar.gz" " http://b.example.com/xyz-123.tar.gz" "http://c.example.com/xyz-123.tar.gz%22...but one of these may be down at any given time.
If there's no support for this in the sources array itself, we should strongly consider adding it.
---end quoted text---
Are you trying tio re-implement Metalinks? They have become standartized way to do what you want ages ago. They are even supported by curl! Add metalink file to package (or reference one from upstream), redefine DLAGENTS to something like 'metalink::/usr/bin/curl --metalink %u' and, if possible, convince makepkg developers to add that DLAGENT to default makepkg.conf. Voilà, no one's KISS principles violated.
On 1 November 2013 18:58, alexander.r@gmx.com wrote:
Are you trying tio re-implement Metalinks? They have become standartized way to do what you want ages ago. They are even supported by curl! Add metalink file to package (or reference one from upstream), redefine DLAGENTS to something like 'metalink::/usr/bin/curl --metalink %u' and, if possible, convince makepkg developers to add that DLAGENT to default makepkg.conf. Voilà, no one's KISS principles violated.
IMHO, this sounds better than a buildscript full of hyperlinks. Alternative mirrors should be an optional feature. I'd even go as far as to suggest just having a .mirrors file (lowercase to differentiate it from pacman files) with a list of optional locations for each source, while adding a bit of code to makepkg to go through the list if the primary source locations fail for whatever reason. But that's my uninformed opinion.
-- GPG/PGP ID: C0711BF1
On Fri, Nov 1, 2013 at 7:21 AM, Rashif Ray Rahman schiv@archlinux.orgwrote:
On 1 November 2013 18:58, alexander.r@gmx.com wrote:
Are you trying tio re-implement Metalinks? They have become standartized
way to do what you want ages
ago. They are even supported by curl! Add metalink file to package (or
reference one from upstream),
redefine DLAGENTS to something like 'metalink::/usr/bin/curl --metalink
%u' and, if possible, convince
makepkg developers to add that DLAGENT to default makepkg.conf. Voilà,
no one's KISS principles
violated.
IMHO, this sounds better than a buildscript full of hyperlinks. Alternative mirrors should be an optional feature. I'd even go as far as to suggest just having a .mirrors file (lowercase to differentiate it from pacman files) with a list of optional locations for each source, while adding a bit of code to makepkg to go through the list if the primary source locations fail for whatever reason. But that's my uninformed opinion.
-- GPG/PGP ID: C0711BF1
If metalinks/some external file are the only way to do this, you would need a local source entry for the metalink / mirrors file AND a reference to each file to pull from that metalink/mirrors file in the source array: (1) This breaks something conceptually to me, since the PKGBUILD is no longer the self-contained descriptor of where to get everything and how to put it all together into the package, now you need this other file or that other file at download-time (rather than just at prepare()/build()/package() time). (2) It also is less KISS than just having multiple source array entries for a mirrored file, since the packager and the person reading the package now need to understand another file format to parse out where the file is coming from. (3) Since there are extra files you're carrying along and it's not clear that those files are only needed at prepare() or build() time, now they're needed at initial download time, and the ordering of the source array becomes important, since you need the metalink/mirrors local file before the metalink link itself...
I'm not opposed to adding metalink, torrent, carrier pigeon, and sneakernet support to DLAGENTS by default, but the simple mirror case that solves this 99% of the time ("file A can come from locations X or Y") is trivially and more KISS-ably solved by having two entries for file A in the source array...
On 2 November 2013 00:14, Ido Rosen ido@kernel.org wrote:
If metalinks/some external file are the only way to do this, you would need a local source entry for the metalink / mirrors file AND a reference to each file to pull from that metalink/mirrors file in the source array: (1) This breaks something conceptually to me, since the PKGBUILD is no longer the self-contained descriptor of where to get everything and how to put it all together into the package, now you need this other file or that other file at download-time (rather than just at prepare()/build()/package() time). (2) It also is less KISS than just having multiple source array entries for a mirrored file, since the packager and the person reading the package now need to understand another file format to parse out where the file is coming from. (3) Since there are extra files you're carrying along and it's not clear that those files are only needed at prepare() or build() time, now they're needed at initial download time, and the ordering of the source array becomes important, since you need the metalink/mirrors local file before the metalink link itself...
For metalinks and such, yeah. But in my proposed scheme, the .mirrors file would contain only alternative links. The primary links would still be part of the source array. This is how I envision the structure:
<<EOF http://somewhere.com/foo.tar.gz http://someother.com/foo.tar.gz
http://nextsource.com/bar.tar.gz http://nextsource2.com/bar.tar.gz EOF
So empty lines separate sources. The order shall follow the order of the sources listed in the buildscript. The PKGBUILD would still contain the sources in the format we know. If this .mirrors file does not exist, the PKGBUILD is still valid and self-contained.
Of course, this is very rudimentary. All it proposes is to fall back to alternative download links sequentially. If you want things like parallel downloads from multiple mirrors, that'd require more work.
And I'm not sure the pacman/makepkg devs like hidden files. I personally use a .contrib file to move over contributor tags when there are too many. It doesn't become a distraction and it's there for those who want historical information.
-- GPG/PGP ID: C0711BF1
On Fri, Nov 01, 2013 at 12:14:25PM -0400, Ido Rosen wrote:
If metalinks/some external file are the only way to do this, you would need a local source entry for the metalink / mirrors file AND a reference to each file to pull from that metalink/mirrors file in the source array
This is obviously incorrect. Please re-read your message, so that no one has to waste more breath.
It also is less KISS than just having multiple source array entries for a mirrored file, since the packager and the person reading the package now need to understand another file format to parse out where the file is coming from.
I'm not opposed to adding metalink, torrent, carrier pigeon, and sneakernet support to DLAGENTS by default
You are the one, who wants metalink, torrent, carrier pigeon, and sneakernet support to be implemented in pure Bash as part of makepkg. With numerous mirrors and sofisticated source arrays, cluttering PKGBUILD source, nobody would indeed be able to understand, what's going on in it's prepare() or build() functions. Obviously, fell free to do something like that yourself; all you need is to re-define prepare() to do whatever you want. Just don't advertise that as something, great enough to become the standard.
On Fri, Nov 1, 2013 at 3:27 PM, alexander.r@gmx.com wrote:
On Fri, Nov 01, 2013 at 12:14:25PM -0400, Ido Rosen wrote:
If metalinks/some external file are the only way to do this, you would
need
a local source entry for the metalink / mirrors file AND a reference to each file to pull from that metalink/mirrors file in the source array
This is obviously incorrect. Please re-read your message, so that no one has to waste more breath.
I've submitted a patch for the mirror idea that follows Jerome's way of doing it, which Florian modified and sent over to pacman-dev. If you want it done differently, submit a patch to help make pacman better rather than making toxic comments, you'll alienate less peers that way. :-)
The point that perhaps I wasn't making clearly (or verbosely) enough was that in the current implementation of makepkg, adding the metalink/.mirrors file to the source array then downloading it in prepare() as you yourself suggested does involve having those files as local files (download_local() in makepkg.sh.in) in the PKGBUILD and then referring to them again in the PKGBUILD, on top of having to understand the metalink format or a new .mirrors format (as opposed to a simple additional array entry as was Jerome's suggestion). The result is that the reader must understand those extra file formats and must go somewhere outside of the PKGBUILD to understand where downloaded files are coming from. This would be the case if metalinks/a separate file were the only or primary mechanism for supporting mirroring. For the common case I brought up, where there are 2 or 3 mirrors for the primary source file(s), metalinks/a new .mirrors file format seems overly complex to me.
It also is less KISS than just having multiple source array entries for a mirrored file, since the packager and the person reading
the
package now need to understand another file format to parse out where the file is coming from.
I'm not opposed to adding metalink, torrent, carrier pigeon, and
sneakernet
support to DLAGENTS by default
You are the one, who wants metalink, torrent, carrier pigeon, and sneakernet support to be implemented in pure Bash as part of makepkg. With numerous mirrors and
Nice straw man. You were the one who suggested metalinks; I said I don't care either way if they're implemented or not (i.e. it is not a vote for or against their implementation... What I said in the paragraphs you misquoted is that I'm neutral to whether the extra DLAGENTs like metalinks, torrents, etc. are implemented or not, and that this is a separate issue. Jerome's proposal is a simpler and more KISS way of solving the common use case / problem that I was trying to solve: mirror A is down for a file may be downloaded from mirrors A or B.
sofisticated source arrays, cluttering PKGBUILD source, nobody would indeed be able to understand, what's going on in it's prepare() or build() functions. Obviously, fell free to do something like that yourself; all you need is to re-define prepare() to do whatever you want. Just don't advertise that as something, great enough to become the standard.
I did like Jerome's suggestion, which was different and simpler than my original suggestion of supporting sources as an associative array. That's why I implemented it and sent it to Florian. The only change that patch makes is that now makepkg aborts at the end of trying to download all the files rather than after the first failed download, and treats source entries with duplicate filenames as equivalents/interchangeable/mirrors. To put it more simply:
source=("http://a.com/file.tar.gz" "http://b.com/file.tar.gz") ^ now makepkg recognizes these as mirrors of file.tar.gz, and will only abort if *both* (rather than either) fail to download file.tar.gz.
Cheers, Ido
-- My AUR packages - https://aur.archlinux.org/packages.php?SeB=m&K=AlexanderR
aur-general@lists.archlinux.org