That would be good too, don't know why I didn't think about that simple solution. :-) On Thu, Oct 31, 2013 at 4:32 PM, Jerome Leclanche <adys.wh@gmail.com> wrote:

On Thu, Oct 31, 2013 at 5:46 PM, Ido Rosen <ido@kernel.org> wrote:

...

Hi everyone, What's the current recommended way to support multiple mirrors for the same source file in a PKGBUILD? For example, I have xyz-123.tar.gz that can be downloaded from any of "http://a.example.com/xyz-123.tar.gz" " http://b.example.com/xyz-123.tar.gz" "http://c.example.com/xyz-123.tar.gz"...but one of these may be down at any given time.

If there's no support for this in the sources array itself, we should strongly consider adding it. For example, an ad-hoc way:

_pick_source_mirror() { # ... do something smart in here } source=( "xyz-123::$(_pick_source_mirror( http://a.example.com/xyz-123.tar.gz http://b.example.com/xyz-123.tar.gz http://c.example.com/xyz-123.tar.gz ))" )

A less ad-hoc way: source() { #...some code that returns a list of URLs/files }

Another way: source=(...files without mirrors as usual...) sourcemirrored=( xyz-123::($mirroraURL $mirrorbURL $mirrorcURL) )

Mirrored sources seem like a common enough phenomenon that it's worth special treatment to make the source PKGBUILD variable more versatile if they're not currently supported. Thoughts?

Ido

What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"

J. Leclanche

On Thu, Oct 31, 2013 at 4:45 PM, Dave Reisner <d@falconindy.com> wrote:

On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:

...

What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"

J. Leclanche

This can't work. If the first source fails to download, makepkg will abort.

What about just detecting if source is an array/list (keep current behavior), or a dictionary/lookup table (new fancy mirror behavior)? If it's a dictionary/lookup table, then try all mirrors before failing a source download. For examples of how to create lookup tables in bash, see http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash...

On Thu, Oct 31, 2013 at 04:51:38PM -0400, Ido Rosen wrote:

On Thu, Oct 31, 2013 at 4:45 PM, Dave Reisner <d@falconindy.com> wrote:

...

On Thu, Oct 31, 2013 at 08:32:49PM +0000, Jerome Leclanche wrote:

...

What about reusing the "filename" feature; if it sees multiple times the same files it treats it as a mirror, eg. "file.tar.gz::http://example.com/file.tar.gz" "file.tar.gz::http://mirror.example.com/file.tar.gz"

J. Leclanche

This can't work. If the first source fails to download, makepkg will abort.

What about just detecting if source is an array/list (keep current behavior), or a dictionary/lookup table (new fancy mirror behavior)? If it's a dictionary/lookup table, then try all mirrors before failing a source download.

For examples of how to create lookup tables in bash, see http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash...

bash4's associative arrays define a 1:1 mapping of string to string. You cannot have multiple elements in the "value." Not to mention that we won't be changing the format of the 'source' variable any time soon.

On 31.10.2013 22:12, Ido Rosen wrote:

Okay, that leaves either Jerome's suggestion (if duplicate filenames are detected, treat them as mirrors and don't fail/abort as long as one of them works);

I'd go with that. Doesn't add any new syntax and can be created easily enough like this: source=("$pkgname-$pkgver.tar.gz"::{"http://foo.com/blub/","http://somewhere.com/"}"$pkgname-$pkgver.tar.gz") Should be as simple as looping of the sources array after a fail and checking if there is another matching "filename::" element. Care to submit a patch to pacman-dev?

On 31.10.2013 23:30, Ido Rosen wrote:

If my first pacman patch is going to be this big of a change to makepkg.sh.in I'd feel a lot more comfortable if an existing pacman/makepkg dev were available off-list via email or IM for some brain-picking. Any volunteers?

Got a poc for a rather simple way of implementing it, not sure if it's going to be pretty enough. I'll take a closer look tomorrow. Feel free to ping me via irc, mail or jabber [1]. [1]: https://www.archlinux.org/developers/#bluewind

On 1 November 2013 18:58, <alexander.r@gmx.com> wrote:

Are you trying tio re-implement Metalinks? They have become standartized way to do what you want ages ago. They are even supported by curl! Add metalink file to package (or reference one from upstream), redefine DLAGENTS to something like 'metalink::/usr/bin/curl --metalink %u' and, if possible, convince makepkg developers to add that DLAGENT to default makepkg.conf. Voilà, no one's KISS principles violated.

IMHO, this sounds better than a buildscript full of hyperlinks. Alternative mirrors should be an optional feature. I'd even go as far as to suggest just having a .mirrors file (lowercase to differentiate it from pacman files) with a list of optional locations for each source, while adding a bit of code to makepkg to go through the list if the primary source locations fail for whatever reason. But that's my uninformed opinion. -- GPG/PGP ID: C0711BF1

On 2 November 2013 00:14, Ido Rosen <ido@kernel.org> wrote:

If metalinks/some external file are the only way to do this, you would need a local source entry for the metalink / mirrors file AND a reference to each file to pull from that metalink/mirrors file in the source array: (1) This breaks something conceptually to me, since the PKGBUILD is no longer the self-contained descriptor of where to get everything and how to put it all together into the package, now you need this other file or that other file at download-time (rather than just at prepare()/build()/package() time). (2) It also is less KISS than just having multiple source array entries for a mirrored file, since the packager and the person reading the package now need to understand another file format to parse out where the file is coming from. (3) Since there are extra files you're carrying along and it's not clear that those files are only needed at prepare() or build() time, now they're needed at initial download time, and the ordering of the source array becomes important, since you need the metalink/mirrors local file before the metalink link itself...

For metalinks and such, yeah. But in my proposed scheme, the .mirrors file would contain only alternative links. The primary links would still be part of the source array. This is how I envision the structure: <<EOF http://somewhere.com/foo.tar.gz http://someother.com/foo.tar.gz http://nextsource.com/bar.tar.gz http://nextsource2.com/bar.tar.gz EOF So empty lines separate sources. The order shall follow the order of the sources listed in the buildscript. The PKGBUILD would still contain the sources in the format we know. If this .mirrors file does not exist, the PKGBUILD is still valid and self-contained. Of course, this is very rudimentary. All it proposes is to fall back to alternative download links sequentially. If you want things like parallel downloads from multiple mirrors, that'd require more work. And I'm not sure the pacman/makepkg devs like hidden files. I personally use a .contrib file to move over contributor tags when there are too many. It doesn't become a distraction and it's there for those who want historical information. -- GPG/PGP ID: C0711BF1

On Fri, Nov 1, 2013 at 3:27 PM, <alexander.r@gmx.com> wrote:

On Fri, Nov 01, 2013 at 12:14:25PM -0400, Ido Rosen wrote:

...

If metalinks/some external file are the only way to do this, you would need a local source entry for the metalink / mirrors file AND a reference to each file to pull from that metalink/mirrors file in the source array

This is obviously incorrect. Please re-read your message, so that no one has to waste more breath.

I've submitted a patch for the mirror idea that follows Jerome's way of doing it, which Florian modified and sent over to pacman-dev. If you want it done differently, submit a patch to help make pacman better rather than making toxic comments, you'll alienate less peers that way. :-) The point that perhaps I wasn't making clearly (or verbosely) enough was that in the current implementation of makepkg, adding the metalink/.mirrors file to the source array then downloading it in prepare() as you yourself suggested does involve having those files as local files (download_local() in makepkg.sh.in) in the PKGBUILD and then referring to them again in the PKGBUILD, on top of having to understand the metalink format or a new .mirrors format (as opposed to a simple additional array entry as was Jerome's suggestion). The result is that the reader must understand those extra file formats and must go somewhere outside of the PKGBUILD to understand where downloaded files are coming from. This would be the case if metalinks/a separate file were the only or primary mechanism for supporting mirroring. For the common case I brought up, where there are 2 or 3 mirrors for the primary source file(s), metalinks/a new .mirrors file format seems overly complex to me.

...

It also is less KISS than just having multiple source array entries for a mirrored file, since the packager and the person reading the package now need to understand another file format to parse out where the file is coming from.

...

I'm not opposed to adding metalink, torrent, carrier pigeon, and sneakernet support to DLAGENTS by default

You are the one, who wants metalink, torrent, carrier pigeon, and sneakernet support to be implemented in pure Bash as part of makepkg. With numerous mirrors and

Nice straw man. You were the one who suggested metalinks; I said I don't care either way if they're implemented or not (i.e. it is not a vote for or against their implementation... What I said in the paragraphs you misquoted is that I'm neutral to whether the extra DLAGENTs like metalinks, torrents, etc. are implemented or not, and that this is a separate issue. Jerome's proposal is a simpler and more KISS way of solving the common use case / problem that I was trying to solve: mirror A is down for a file may be downloaded from mirrors A or B.

sofisticated source arrays, cluttering PKGBUILD source, nobody would indeed be able to understand, what's going on in it's prepare() or build() functions. Obviously, fell free to do something like that yourself; all you need is to re-define prepare() to do whatever you want. Just don't advertise that as something, great enough to become the standard.

I did like Jerome's suggestion, which was different and simpler than my original suggestion of supporting sources as an associative array. That's why I implemented it and sent it to Florian. The only change that patch makes is that now makepkg aborts at the end of trying to download all the files rather than after the first failed download, and treats source entries with duplicate filenames as equivalents/interchangeable/mirrors. To put it more simply: source=("http://a.com/file.tar.gz" "http://b.com/file.tar.gz") ^ now makepkg recognizes these as mirrors of file.tar.gz, and will only abort if *both* (rather than either) fail to download file.tar.gz. Cheers, Ido

-- My AUR packages - https://aur.archlinux.org/packages.php?SeB=m&K=AlexanderR