[pacman-dev] [PATCH] Added mirror support to makepkg's source array.
From: Ido Rosen <code@idorosen.com> To specify multiple mirrors, simply add multiple files to the source array that have the same downloaded filename, e.g.: source=("file.tar.gz::http://mirror1.example.com/file.tar.gz" "file.tar.gz::http://mirror2.example.com/file.tar.gz") ...makepkg will try them all, and if all fail, will abort. This also applies to VCS repositories and local files references in the source array, so for example: source=("git+https://github.com/torvalds/linux.git "git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git") ...will allow cloning the git repository from GitHub if kernel.org is down or fails. Signed-off-by: Florian Pritz <bluewind@xinu.at> --- I've swichted from get_filepath for the array index to get_filename as get_filepath only returns something useful if the local file already exists. I guess this was just an oversight, but noted for reference. I've also tested this a little for http urls and it seems to work as expected. One thing that we might want to fix is that the downloads get validated multiple times right now (once for each entry). scripts/makepkg.sh.in | 73 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 28 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 4cb8173..452df09 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -326,7 +326,7 @@ download_local() { else local filename=$(get_filename "$netfile") error "$(gettext "%s was not found in the build directory and is not a URL.")" "$filename" - exit 1 # $E_MISSING_FILE + return 1 # $E_MISSING_FILE fi } @@ -375,8 +375,7 @@ download_file() { if (( ret )); then [[ ! -s $dlfile ]] && rm -f -- "$dlfile" error "$(gettext "Failure while downloading %s")" "$filename" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi # rename the temporary download file to the final destination @@ -460,8 +459,7 @@ download_bzr() { msg2 "$(gettext "Branching %s ...")" "${displaylocation}" if ! bzr branch "$url" "$dir" --no-tree --use-existing-dir; then error "$(gettext "Failure while branching %s")" "${displaylocation}" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then # Make sure we are fetching the right repo @@ -470,15 +468,13 @@ download_bzr() { if [[ -n $distant_url ]]; then if [[ $distant_url != "$local_url" ]]; then error "$(gettext "%s is not a branch of %s")" "$dir" "$url" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi else if [[ $url != "$local_url" ]] ; then error "$(gettext "%s is not a branch of %s")" "$dir" "$url" error "$(gettext "The local URL is %s")" "$local_url" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi fi msg2 "$(gettext "Pulling %s ...")" "${displaylocation}" @@ -545,16 +541,14 @@ download_git() { msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git" if ! git clone --mirror "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then cd_safe "$dir" # Make sure we are fetching the right repo if [[ "$url" != "$(git config --get remote.origin.url)" ]] ; then error "$(gettext "%s is not a clone of %s")" "$dir" "$url" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git" if ! git fetch --all -p; then @@ -634,8 +628,7 @@ download_hg() { msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "hg" if ! hg clone -U "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "hg" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "hg" @@ -673,15 +666,13 @@ extract_hg() { ;; *) error "$(gettext "Unrecognized reference: %s")" "${fragment}" - plain "$(gettext "Aborting...")" - exit 1 + return 1 esac fi if ! hg clone "${ref[@]}" "$dir" "${dir##*/}"; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "hg" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi popd &>/dev/null @@ -711,8 +702,7 @@ download_svn() { mkdir -p "$dir/.makepkg" if ! svn checkout --config-dir "$dir/.makepkg" "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "svn" - plain "$(gettext "Aborting...")" - exit 1 + return 1 fi elif (( ! HOLDVER )); then msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "svn" @@ -776,34 +766,61 @@ download_sources() { GET_VCS=0 fi + declare -A downloaded # local since no -g supplied to declare. + declare -A expected # local since no -g supplied to declare. local netfile for netfile in "${source[@]}"; do pushd "$SRCDEST" &>/dev/null + local filename=$(get_filename "${netfile}") + expected["${filename}"]=1 + if [[ ${downloaded["${filename}"]} ]]; then + msg "$(gettext "Skipping mirror: %s")" "$netfile" + continue # file already downloaded, current mirror is redundant. + fi local proto=$(get_protocol "$netfile") case "$proto" in local) - download_local "$netfile" + download_local "$netfile" && \ + downloaded["${filename}"]=1 ;; bzr*) - (( GET_VCS )) && download_bzr "$netfile" + (( GET_VCS )) && download_bzr "$netfile" && \ + downloaded["${filename}"]=1 ;; git*) - (( GET_VCS )) && download_git "$netfile" + (( GET_VCS )) && download_git "$netfile" && \ + downloaded["${filename}"]=1 ;; hg*) - (( GET_VCS )) && download_hg "$netfile" + (( GET_VCS )) && download_hg "$netfile" && \ + downloaded["${filename}"]=1 ;; svn*) - (( GET_VCS )) && download_svn "$netfile" + (( GET_VCS )) && download_svn "$netfile" && \ + downloaded["${filename}"]=1 ;; *) - download_file "$netfile" + download_file "$netfile" && \ + downloaded["${filename}"]=1 ;; esac - popd &>/dev/null done + + local failed=() + for expectedfilename in "${!expected[@]}"; do + if [[ ${downloaded["${expectedfilename}"]} ]]; then + continue + else + failed=(${failed[@]} "${expectedfilename}") + error "$(gettext "Fatal failure while downloading file %s")" "${expectedfilename}" + fi + done + if [[ ${#failed[@]} -gt 0 ]]; then + plain "$(gettext "Aborting...")" + exit 1 + fi } # Automatically update pkgver variable if a pkgver() function is provided -- 1.8.4.2
On 01/11/13 20:00, Florian Pritz wrote:
From: Ido Rosen <code@idorosen.com>
To specify multiple mirrors, simply add multiple files to the source array that have the same downloaded filename, e.g.:
source=("file.tar.gz::http://mirror1.example.com/file.tar.gz" "file.tar.gz::http://mirror2.example.com/file.tar.gz")
...makepkg will try them all, and if all fail, will abort. This also applies to VCS repositories and local files references in the source array, so for example:
source=("git+https://github.com/torvalds/linux.git "git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git")
...will allow cloning the git repository from GitHub if kernel.org is down or fails.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
We discussed this on IRC. I really dislike the multiple source lines "downloading" the same file approach. An idea is doing: source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/") makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure. To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array. Comments or other ideas? Allan
Allan, On Fri, Nov 15, 2013 at 12:06 PM, Allan McRae <allan@archlinux.org> wrote:
On 01/11/13 20:00, Florian Pritz wrote:
From: Ido Rosen <code@idorosen.com>
To specify multiple mirrors, simply add multiple files to the source array that have the same downloaded filename, e.g.:
source=("file.tar.gz::http://mirror1.example.com/file.tar.gz" "file.tar.gz::http://mirror2.example.com/file.tar.gz")
...makepkg will try them all, and if all fail, will abort. This also applies to VCS repositories and local files references in the source array, so for example:
source=("git+https://github.com/torvalds/linux.git "git+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git")
...will allow cloning the git repository from GitHub if kernel.org is down or fails.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
We discussed this on IRC. I really dislike the multiple source lines "downloading" the same file approach.
An idea is doing:
source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/")
makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure.
To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array.
Comments or other ideas?
Would this then implement the mirror:// or sourceforge:// protocol as a new DLAGENT? It seems to me that DLAGENTs should serve a different (but also important) purpose and should be conceptually separate - I would avoid overloading the scheme/protocol field for this case. Or am I misunderstanding your suggestion? Also, would this mean adding a new variable for each mirrored site? That seems less flexible than Jerome's formatting that I implemented. I really like the simplicity of the original patch, and it doesn't break existing PKGBUILDs AFAICT... If you're vetoing that, here is a compromise: we could create a new DLAGENT called mirrorlist that allows specifying the variable name in which the mirror list is stored for that file, and an optional suffix: source=("destinationfile.tar.gz::mirrorlist://_variablename/suffix/file.tar.gz") _variablename=("http://foo.com/" "http://bar.com/") # ... would download from http://foo.com/suffix/file.tar.gz, then try the same for bar.com upon failure, and put it in destinationfile.tar.gz. source=("destinationfile.tar.gz::mirrorlist://_variablename") _variablename=("http://foo.com/suffix/file.tar.gz" " http://bar.com/suffix/file.tar.gz") # ... same as above. # (destinationfile.tar.gz:: is optional as usual...) Whatever your final consensus/decision is, I'll be happy to submit a new patch, just let me know.
Allan
On 16/11/13 07:53, Ido Rosen wrote:
Allan,
On Fri, Nov 15, 2013 at 12:06 PM, Allan McRae <allan@archlinux.org> wrote:
On 01/11/13 20:00, Florian Pritz wrote:
From: Ido Rosen <code@idorosen.com>
To specify multiple mirrors, simply add multiple files to the source array that have the same downloaded filename, e.g.:
source=("file.tar.gz::http://mirror1.example.com/file.tar.gz" "file.tar.gz::http://mirror2.example.com/file.tar.gz")
...makepkg will try them all, and if all fail, will abort. This also applies to VCS repositories and local files references in the source array, so for example:
source=("git+https://github.com/torvalds/linux.git "git+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git")
...will allow cloning the git repository from GitHub if kernel.org is down or fails.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
We discussed this on IRC. I really dislike the multiple source lines "downloading" the same file approach.
An idea is doing:
source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/")
makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure.
To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array.
Comments or other ideas?
Would this then implement the mirror:// or sourceforge:// protocol as a new DLAGENT?
No. My proposal had nothing to do with DLAGENT. We call get_protocol() and see an unrecognised protocol. We see that the specified "protocol" corresponds to an array in the PKGBUILD. So we know we are dealing with a mirror source list.
It seems to me that DLAGENTs should serve a different (but also important) purpose and should be conceptually separate - I would avoid overloading the scheme/protocol field for this case. Or am I misunderstanding your suggestion? Also, would this mean adding a new variable for each mirrored site? That seems less flexible than Jerome's formatting that I implemented.
I really like the simplicity of the original patch, and it doesn't break existing PKGBUILDs AFAICT... If you're vetoing that, here is a compromise:
I'm definitely vetoing the original proposal - having multiple source lines for one file is really ugly and I did not consider it simple at all.
we could create a new DLAGENT called mirrorlist that allows specifying the variable name in which the mirror list is stored for that file, and an optional suffix:
source=("destinationfile.tar.gz::mirrorlist://_variablename/suffix/file.tar.gz") _variablename=("http://foo.com/" "http://bar.com/") # ... would download from http://foo.com/suffix/file.tar.gz, then try the same for bar.com upon failure, and put it in destinationfile.tar.gz.
That is essentially what I proposed. Having a constant protocol string makes sense. I'd use "mirror://" because it is shorter. Implement this one...
source=("destinationfile.tar.gz::mirrorlist://_variablename") _variablename=("http://foo.com/suffix/file.tar.gz" " http://bar.com/suffix/file.tar.gz") # ... same as above. # (destinationfile.tar.gz:: is optional as usual...)
... and not that one. We should be able to extract the filename from the source URL (if the file name is not given in the prefix). Allan
On 2013-11-15 21:06 +1000 Allan McRae wrote:
An idea is doing:
source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/")
makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure.
To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array.
Comments or other ideas?
Detecting "unknown" protocols is not future proof and will lead to an easy but never-ending game of whack-a-mole. It would also lead to unnecessary repetition in the "mirror" array. I think it would be better to use a variable delimiter (possibly an illegal URL character, or a PKGBUILD-variable for maximum flexibility), e.g. source_delim='*' source=("http://*mirror*file.tar.gz") mirror=("foo.com/" "bar.com/") You could even use "source_" as a prefix for the source variables to avoid namespace polution (source_mirror, source_foo, etc). Allowing variables at arbitrary points provides greater flexibility and reduces verbosity (e.g. the repetition of the scheme in this case). If you really want to limit this to URL prefixes, then I suggest enforcing a policy of using lower-case schemes for regular URLs and upper-case pseudo-schemes for mirrors, e.g. source=("MIRROR://file.tar.gz") MIRROR=("http://foo.com/" "http://bar.com/") While I think the previous approach is better, this would also avoid the eventual grief of a hard-coded list of recognized protocols. Incidentally, if you generalize this enough you could probably re-use the code to implement dependency alternatives, if you are interested in that. The logic overlaps. Regards, Xyne
On 18/11/13 03:46, Xyne wrote:
On 2013-11-15 21:06 +1000 Allan McRae wrote:
An idea is doing:
source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/")
makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure.
To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array.
Comments or other ideas?
Detecting "unknown" protocols is not future proof and will lead to an easy but never-ending game of whack-a-mole. It would also lead to unnecessary repetition in the "mirror" array.
I think it would be better to use a variable delimiter (possibly an illegal URL character, or a PKGBUILD-variable for maximum flexibility), e.g.
source_delim='*' source=("http://*mirror*file.tar.gz") mirror=("foo.com/" "bar.com/")
You could even use "source_" as a prefix for the source variables to avoid namespace polution (source_mirror, source_foo, etc).
Allowing variables at arbitrary points provides greater flexibility and reduces verbosity (e.g. the repetition of the scheme in this case). If you really want to limit this to URL prefixes, then I suggest enforcing a policy of using lower-case schemes for regular URLs and upper-case pseudo-schemes for mirrors, e.g.
source=("MIRROR://file.tar.gz") MIRROR=("http://foo.com/" "http://bar.com/")
While I think the previous approach is better, this would also avoid the eventual grief of a hard-coded list of recognized protocols.
What did you think about the proposal in an earlier reply to this thread: source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/") I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine. Allan
On Mon, Nov 18, 2013 at 3:09 AM, Allan McRae <allan@archlinux.org> wrote:
What did you think about the proposal in an earlier reply to this thread:
source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")
I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.
Allan
Personally, I wouldn't notice that _foo part there and go WTF if the URL isn't valid. This is as bad as the previous suggestion, where I thought of this: source=("http://looks.ok/good.tar.gz") [...fast-forward to bottom of PKGBUILD] http=("http://malware.com/evil_file.tar.gz") whereas with the latest suggestion, one would just source=("http://_it/looks.ok/good.tar.gz") [...fast-forward to bottom of PKGBUILD] _it=("http://malware.com/evil_file.tar.gz") PKGBUILDs are supposed to be easily readable, and I don't want to have to be sober and well-rested to read them. Although I don't really have a better suggestion than what has been proposed, I'm pretty sure the current suggestions would make reading PKGBIULDs harder. Maybe a more limited and obvious approach would be this: source=("mirrors:good.tar.gz") mirrors=("http://looks.ok/good.tar.gz"...) Drop (or exclusively use) the double-slash here, this is not about URLs. Enforce or limit the token to "mirrors" - or /mirrors[0-9]*/ to support multiple files. This would not need to break existing PKGBUILDs as I don't think "mirrors:*" would match any source array item in the world of PKGBUILDs, and in the case of "mirrors//" I can even be sure. cheers! mar77i
On 20/11/13 22:51, Martti Kühne wrote:
On Mon, Nov 18, 2013 at 3:09 AM, Allan McRae <allan@archlinux.org> wrote:
What did you think about the proposal in an earlier reply to this thread:
source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")
I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.
Allan
Personally, I wouldn't notice that _foo part there and go WTF if the URL isn't valid. This is as bad as the previous suggestion, where I thought of this:
source=("http://looks.ok/good.tar.gz") [...fast-forward to bottom of PKGBUILD] http=("http://malware.com/evil_file.tar.gz")
whereas with the latest suggestion, one would just
source=("http://_it/looks.ok/good.tar.gz") [...fast-forward to bottom of PKGBUILD] _it=("http://malware.com/evil_file.tar.gz")
Ummm.... both those would be perfectly safe with the suggestion in my email above because the sources are prefixed with http:// and so the not arrays with malware.com would do nothing. Only source lines starting with mirror:// would require looking at the array of mirror sources. A
Hey folks, I just wanted to throw a few things out in this discussion, if that's alright. First, I like the idea of supporting mirrors, but I think we need to take certain factors into account that not all the proposals offered have. For example, not all the proposals take into account the possibility of supporting different protocols as the same mirror. This is one aspect where Allan's proposal is quite nice because the mirrors array would have you input full URLs including their protocol. However, not all mirrors follow the same directory hierarchy, so perhaps mirrored entries in the source array should be required to look something like this: source=(mirrors://filename.tar.gz) mirrors=('https://foo.br/whar/did/you/put/' 'ftp://bar.fu/put/it/here/') Excluding the rest of the path so that the mirrors array allows for multiple longer URLs which might differ from one another on where the filename is located. The second bit that needs to be figured out is how best to offer the ability to mirror multiple files in the source array. Should it be done by having multiple mirrors arrays with slightly differing names? I honestly don't know, but personally, I find that to look ugly, so I'd recommend something else. Perhaps it would not be too difficult to add some kind of note to each entry in the mirrors array which denotes which source array entry it points to? E.g.,: source=('mirrors://filename.tar.gz' 'mirrors://namedfile.tar.gz') mirrors=('0|https://foo.br/whar/did/you/put/' '0|ftp://bar.fu/put/it/here/' '1|http://fu.bar/it/be/here/' '1|sftp://bu.far/here/it/be/') I know this is not terribly pretty, but it satisfies the need for supporting multiple mirrored sources and it uses a convention already in place (source array indexes), though it does refer to them in a new style. Anyway, that's just my two cents. I hope they were helpful. All the best, --- Sam Stuewe (a.k.a. HalosGhost)
On Wed, Nov 20, 2013 at 11:22:36AM -0600, Sam Stuewe wrote:
Hey folks, I just wanted to throw a few things out in this discussion, if that's alright.
First, I like the idea of supporting mirrors, but I think we need to take certain factors into account that not all the proposals offered have. For example, ...
How about we take a step back and figure out how much actual need there is for this sort of feature? 10 packages? 100 packages? 1000 packages? If you ignore the AUR, are there *any* cases where this is useful? d
On Wed, 20 Nov 2013 13:06:30 -0500 Dave Reisner <d@falconindy.com> wrote:
How about we take a step back and figure out how much actual need there is for this sort of feature? 10 packages? 100 packages? 1000 packages? If you ignore the AUR, are there *any* cases where this is useful?
Maybe a simple || operator for the source array is enough. -- Joakim
On 2013-11-18 12:09 +1000 Allan McRae wrote:
What did you think about the proposal in an earlier reply to this thread:
source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")
I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.
Allan
That works, but I would prefer differential syntax to prevent confusion with regular URLs. "MIRROR://" should be enough. It stands out, and we already use upper-case for "SKIP" in the checksums array, so I think it is stylistically coherent. Regards, Xyne
On Wed, Nov 20, 2013 at 7:55 PM, Xyne <xyne@archlinux.ca> wrote:
On 2013-11-18 12:09 +1000 Allan McRae wrote:
What did you think about the proposal in an earlier reply to this thread:
source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")
I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.
Allan
That works, but I would prefer differential syntax to prevent confusion with regular URLs. "MIRROR://" should be enough. It stands out, and we already use upper-case for "SKIP" in the checksums array, so I think it is stylistically coherent.
Regards, Xyne
In the interest of getting mirror support into PKGBUILD/makepkg/pacman, regardless of style or aesthetic considerations. Seems like these are the two current options being considered: (a) my original patch (multiple same-filenamed entries in $source), or, (b) the proposal of having [dest_filename::]mirror://<_identifier>/[path_prefix/]<filename> ...scheme that references a shell variable or function name to get the list from.
From my perspective: (a) is already written and easy to grok, involving relatively minor changes to makepkg.sh.in; (b) is more flexible but involves duplication of code in makepkg, multiple entrypoints into download_* functions...
So, do we vote or does Allan decide or do we roll dice? :-) I'm happy to resubmit the patch either way. Ido
On 25/11/13 07:36, Ido Rosen wrote:
On Wed, Nov 20, 2013 at 7:55 PM, Xyne <xyne@archlinux.ca> wrote:
On 2013-11-18 12:09 +1000 Allan McRae wrote:
What did you think about the proposal in an earlier reply to this thread:
source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")
I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.
Allan
That works, but I would prefer differential syntax to prevent confusion with regular URLs. "MIRROR://" should be enough. It stands out, and we already use upper-case for "SKIP" in the checksums array, so I think it is stylistically coherent.
Regards, Xyne
In the interest of getting mirror support into PKGBUILD/makepkg/pacman, regardless of style or aesthetic considerations. Seems like these are the two current options being considered:
(a) my original patch (multiple same-filenamed entries in $source), or, (b) the proposal of having [dest_filename::]mirror://<_identifier>/[path_prefix/]<filename> ...scheme that references a shell variable or function name to get the list from.
From my perspective: (a) is already written and easy to grok, involving relatively minor changes to makepkg.sh.in; (b) is more flexible but involves duplication of code in makepkg, multiple entrypoints into download_* functions...
I think b) will involve a single function to be added (download_mirror) which loops and passes to the download_... functions as necessary. So it is very self contained. And the duplication of farming out to the right function can be refactored.
So, do we vote or does Allan decide or do we roll dice? :-) I'm happy to resubmit the patch either way.
I decide! :) But I tend to listen to votes containing explanations... One question needs answered first. Should there be able to be two different mirror arrays. I.e. If a PKGBUILD has multiple source files available across mirrors, will those be the same mirrors? If the answer is no, we can simplify this some more to: source=('mirror://foo/foo.tar.xz') mirror=('http://bar.com/download' 'http://baz.com/mirror/download') My gut feeling is in two parts: 1) if your PKGBUILD requires two different mirror arrays, you are probably doing it wrong 2) I doubt the number of packages using mirrors will be large, yet along needing two mirror arrays. Allan
One question needs answered first. Should there be able to be two different mirror arrays. I.e. If a PKGBUILD has multiple source files available across mirrors, will those be the same mirrors? Though I think the flexibility of having an option for multiple source files to have mirrored sources, I agree that the number of use cases is
If the answer is no, we can simplify this some more to:
source=('mirror://foo/foo.tar.xz') mirror=('http://bar.com/download' 'http://baz.com/mirror/download') I do love how simple that would be. If the end result of this proposal is this simple solution, or one slightly more complex one that supports multiple sources with mirrors, I will be satisfied either way. The only extra argument I would make in favor of supporting multiple mirrored
On 2013-11-24 23:46, Allan McRae wrote: probably very small. sources is that it seems odd to allow a special case only for one source file. Wouldn't it be more less hacky and more standardized if PKGBUILDs treated source files as equal? All the best, --- Sam Stuewe (a.k.a. HalosGhost)
On 25/11/13 17:15, Sam Stuewe wrote:
One question needs answered first. Should there be able to be two different mirror arrays. I.e. If a PKGBUILD has multiple source files available across mirrors, will those be the same mirrors? Though I think the flexibility of having an option for multiple source files to have mirrored sources, I agree that the number of use cases is
If the answer is no, we can simplify this some more to:
source=('mirror://foo/foo.tar.xz') mirror=('http://bar.com/download' 'http://baz.com/mirror/download') I do love how simple that would be. If the end result of this proposal is this simple solution, or one slightly more complex one that supports multiple sources with mirrors, I will be satisfied either way. The only extra argument I would make in favor of supporting multiple mirrored
On 2013-11-24 23:46, Allan McRae wrote: probably very small. sources is that it seems odd to allow a special case only for one source file. Wouldn't it be more less hacky and more standardized if PKGBUILDs treated source files as equal?
I'm fairly certain the number of PKGBUILDs that have source files that are mirrored in different locations can be well approximated by zero. I'm not even sure the mirror feature will get a lot of use, given many (most?) places that provide mirrors of their files set up a round robin or geolocation to balance the load across mirrors anyway. I know there are obvious exceptions to that, so I am happy adding mirror support. Someone would have to convince me that adding more than one source mirror array would be a useful prospect. Allan
participants (8)
-
Allan McRae
-
Dave Reisner
-
Florian Pritz
-
Ido Rosen
-
Joakim Hernberg
-
Martti Kühne
-
Sam Stuewe
-
Xyne