On 16/11/13 07:53, Ido Rosen wrote:

Allan,

On Fri, Nov 15, 2013 at 12:06 PM, Allan McRae wrote:

...

On 01/11/13 20:00, Florian Pritz wrote:

...

From: Ido Rosen

To specify multiple mirrors, simply add multiple files to the source array that have the same downloaded filename, e.g.:

source=("file.tar.gz::http://mirror1.example.com/file.tar.gz" "file.tar.gz::http://mirror2.example.com/file.tar.gz")

...makepkg will try them all, and if all fail, will abort. This also applies to VCS repositories and local files references in the source array, so for example:

source=("git+https://github.com/torvalds/linux.git "git+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git")

...will allow cloning the git repository from GitHub if kernel.org is down or fails.

Signed-off-by: Florian Pritz

We discussed this on IRC. I really dislike the multiple source lines "downloading" the same file approach.

An idea is doing:

source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/")

makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure.

To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array.

Comments or other ideas?

Would this then implement the mirror:// or sourceforge:// protocol as a new DLAGENT?

No. My proposal had nothing to do with DLAGENT. We call get_protocol() and see an unrecognised protocol. We see that the specified "protocol" corresponds to an array in the PKGBUILD. So we know we are dealing with a mirror source list.

It seems to me that DLAGENTs should serve a different (but also important) purpose and should be conceptually separate - I would avoid overloading the scheme/protocol field for this case. Or am I misunderstanding your suggestion? Also, would this mean adding a new variable for each mirrored site? That seems less flexible than Jerome's formatting that I implemented.

I really like the simplicity of the original patch, and it doesn't break existing PKGBUILDs AFAICT... If you're vetoing that, here is a compromise:

I'm definitely vetoing the original proposal - having multiple source lines for one file is really ugly and I did not consider it simple at all.

we could create a new DLAGENT called mirrorlist that allows specifying the variable name in which the mirror list is stored for that file, and an optional suffix:

source=("destinationfile.tar.gz::mirrorlist://_variablename/suffix/file.tar.gz") _variablename=("http://foo.com/" "http://bar.com/") # ... would download from http://foo.com/suffix/file.tar.gz, then try the same for bar.com upon failure, and put it in destinationfile.tar.gz.

That is essentially what I proposed. Having a constant protocol string makes sense. I'd use "mirror://" because it is shorter. Implement this one...

source=("destinationfile.tar.gz::mirrorlist://_variablename") _variablename=("http://foo.com/suffix/file.tar.gz" " http://bar.com/suffix/file.tar.gz") # ... same as above. # (destinationfile.tar.gz:: is optional as usual...)

... and not that one. We should be able to extract the filename from the source URL (if the file name is not given in the prefix). Allan

On 18/11/13 03:46, Xyne wrote:

On 2013-11-15 21:06 +1000 Allan McRae wrote:

...

An idea is doing:

source=("mirror://file.tar.gz") mirror=("http://foo.com/" "http://bar.com/")

makepkg would replace "mirror://" in the source line with a value from the "mirror" array and try downloading from each one until success or total failure.

To allow multiple sources to have different mirrors, I am proposing we detect an unknown protocol in the url and look for the corresponding array. e.g. we could have source=("sourceforge://file.tar.gz") and look for the "sourceforge" array.

Comments or other ideas?

Detecting "unknown" protocols is not future proof and will lead to an easy but never-ending game of whack-a-mole. It would also lead to unnecessary repetition in the "mirror" array.

I think it would be better to use a variable delimiter (possibly an illegal URL character, or a PKGBUILD-variable for maximum flexibility), e.g.

source_delim='*' source=("http://*mirror*file.tar.gz") mirror=("foo.com/" "bar.com/")

You could even use "source_" as a prefix for the source variables to avoid namespace polution (source_mirror, source_foo, etc).

Allowing variables at arbitrary points provides greater flexibility and reduces verbosity (e.g. the repetition of the scheme in this case). If you really want to limit this to URL prefixes, then I suggest enforcing a policy of using lower-case schemes for regular URLs and upper-case pseudo-schemes for mirrors, e.g.

source=("MIRROR://file.tar.gz") MIRROR=("http://foo.com/" "http://bar.com/")

While I think the previous approach is better, this would also avoid the eventual grief of a hard-coded list of recognized protocols.

What did you think about the proposal in an earlier reply to this thread: source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/") I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine. Allan

On 20/11/13 22:51, Martti Kühne wrote:

On Mon, Nov 18, 2013 at 3:09 AM, Allan McRae wrote:

...

What did you think about the proposal in an earlier reply to this thread:

source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")

I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.

Allan

Personally, I wouldn't notice that _foo part there and go WTF if the URL isn't valid. This is as bad as the previous suggestion, where I thought of this:

source=("http://looks.ok/good.tar.gz") [...fast-forward to bottom of PKGBUILD] http=("http://malware.com/evil_file.tar.gz")

whereas with the latest suggestion, one would just

source=("http://_it/looks.ok/good.tar.gz") [...fast-forward to bottom of PKGBUILD] _it=("http://malware.com/evil_file.tar.gz")

Ummm.... both those would be perfectly safe with the suggestion in my email above because the sources are prefixed with http:// and so the not arrays with malware.com would do nothing. Only source lines starting with mirror:// would require looking at the array of mirror sources. A

On Wed, Nov 20, 2013 at 11:22:36AM -0600, Sam Stuewe wrote:

Hey folks, I just wanted to throw a few things out in this discussion, if that's alright.

First, I like the idea of supporting mirrors, but I think we need to take certain factors into account that not all the proposals offered have. For example, ...

How about we take a step back and figure out how much actual need there is for this sort of feature? 10 packages? 100 packages? 1000 packages? If you ignore the AUR, are there *any* cases where this is useful? d

On 2013-11-18 12:09 +1000 Allan McRae wrote:

What did you think about the proposal in an earlier reply to this thread:

source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")

I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.

Allan

That works, but I would prefer differential syntax to prevent confusion with regular URLs. "MIRROR://" should be enough. It stands out, and we already use upper-case for "SKIP" in the checksums array, so I think it is stylistically coherent. Regards, Xyne

On 25/11/13 07:36, Ido Rosen wrote:

On Wed, Nov 20, 2013 at 7:55 PM, Xyne wrote:

...

On 2013-11-18 12:09 +1000 Allan McRae wrote:

...

What did you think about the proposal in an earlier reply to this thread:

source=("mirror://_foo/blah/blah/foo.tar.gz") _foo=("http://foo.com/" "http://bar.com/")

I think we can bet safely that "mirror://" will not become a valid protocol. And we are replacing the start of the URL so just using the initial "/" as the delimiter is fine.

Allan

That works, but I would prefer differential syntax to prevent confusion with regular URLs. "MIRROR://" should be enough. It stands out, and we already use upper-case for "SKIP" in the checksums array, so I think it is stylistically coherent.

Regards, Xyne

In the interest of getting mirror support into PKGBUILD/makepkg/pacman, regardless of style or aesthetic considerations. Seems like these are the two current options being considered:

(a) my original patch (multiple same-filenamed entries in $source), or, (b) the proposal of having [dest_filename::]mirror://<_identifier>/[path_prefix/]<filename> ...scheme that references a shell variable or function name to get the list from.

...

From my perspective: (a) is already written and easy to grok, involving relatively minor changes to makepkg.sh.in; (b) is more flexible but involves duplication of code in makepkg, multiple entrypoints into download_* functions...

I think b) will involve a single function to be added (download_mirror) which loops and passes to the download_... functions as necessary. So it is very self contained. And the duplication of farming out to the right function can be refactored.

So, do we vote or does Allan decide or do we roll dice? :-) I'm happy to resubmit the patch either way.

I decide! :) But I tend to listen to votes containing explanations... One question needs answered first. Should there be able to be two different mirror arrays. I.e. If a PKGBUILD has multiple source files available across mirrors, will those be the same mirrors? If the answer is no, we can simplify this some more to: source=('mirror://foo/foo.tar.xz') mirror=('http://bar.com/download' 'http://baz.com/mirror/download') My gut feeling is in two parts: 1) if your PKGBUILD requires two different mirror arrays, you are probably doing it wrong 2) I doubt the number of packages using mirrors will be large, yet along needing two mirror arrays. Allan

On 25/11/13 17:15, Sam Stuewe wrote:

...

One question needs answered first. Should there be able to be two different mirror arrays. I.e. If a PKGBUILD has multiple source files available across mirrors, will those be the same mirrors? Though I think the flexibility of having an option for multiple source files to have mirrored sources, I agree that the number of use cases is

...

If the answer is no, we can simplify this some more to:

source=('mirror://foo/foo.tar.xz') mirror=('http://bar.com/download' 'http://baz.com/mirror/download') I do love how simple that would be. If the end result of this proposal is this simple solution, or one slightly more complex one that supports multiple sources with mirrors, I will be satisfied either way. The only extra argument I would make in favor of supporting multiple mirrored

On 2013-11-24 23:46, Allan McRae wrote: probably very small. sources is that it seems odd to allow a special case only for one source file. Wouldn't it be more less hacky and more standardized if PKGBUILDs treated source files as equal?

I'm fairly certain the number of PKGBUILDs that have source files that are mirrored in different locations can be well approximated by zero. I'm not even sure the mirror feature will get a lot of use, given many (most?) places that provide mirrors of their files set up a round robin or geolocation to balance the load across mirrors anyway. I know there are obvious exceptions to that, so I am happy adding mirror support. Someone would have to convince me that adding more than one source mirror array would be a useful prospect. Allan