Hi! To quickly introduce myself, I am the author of MirrorBrain and I noticed you discussion here. On Tue, Mar 10, 2009 at 11:33:36 +0100, Dieter Plaetinck wrote:

On Tue, 10 Mar 2009 15:40:43 +1100 "Tim G" <tim at digitalpacific.com.au> wrote:

...

Hey everyone,

I am not sure if it has been brought up before, but has anyone thought about providing a MirrorBrain http repository? OpenSUSE has one at http://download.opensuse.org/ and it works quite well. If we are able to provide something like this, it would mean that we could provide transparent mirroring. MirrorBrain even looks after creating mirror lists

http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-KDE4-LiveCD -i686.iso?mirrorlist

MirrorBrain also creates Metalinks. Didn't you guys start using those? I saw http://bbs.archlinux.org/viewtopic.php?pid=573250 (regarding ISO images) and I noticed usage for package management being mentioned here: http://en.wikipedia.org/wiki/Metalink#In_use "Metalink is also being tested with Arch Linux's Pacman which is used for package management." openSUSE is switching to metalinks for downloads of packages and metadata by the package management with the next release (using aria2c in the background). If you consider doing something similar, MirrorBrain would be the perfect basis to deploy this. I'd be very happy to help out with getting MirrorBrain up and running for you guys, if you are interested!

...

Anyways, have a look at it, http://mirrorbrain.org/features

Wow this looks cool. It could save us from some pratical issues we have (such as "don't give out links like archlinux.org/.., we don't want to much traffic there"). If I understand this correctly we can use archlinux.org links everywhere and the redirection stuff will be taken care of automatically. (that way we maybe don't need to throttle the main mirror anymore?)

Yes.

Also the tool to check which mirrors are incomplete and redirect accordingly is nice.

Though if archlinux.org goes down, and pretty much everything links to it, we have a bigger problem.

There can be more than one server - you could have a setup with failover server, or actually run different servers on different continents. On the other hand, openSUSE has been running with a single machine for over two years now (due to regrettable lack of resources...), and other than hard disk failures or human error, there were no issues with the setup. That said, the best (and easiest) way to achieve failover would probably be to run two servers (US, Europe) and use GeoDNS to distribute requests geographically to them. If one goes down, just switch the clients over to the other one by a little DNS change (keeping the TTL to 10 minutes or so.) At the same time, such a setup has the advantage of giving clients a faster response due to less roundtrip time. If you want to know more, please let me know. I'm subscribed to the pacman-dev mailing list, and will stay for a little while, but it would be safe to reach me at my email, or you could just post to the mirrorbrain at mirrorbrain.org mailing list (you don't need to subscribe first). Thanks, Peter

Hi Dieter! Am 03.07.2009 um 18:08 schrieb Dieter Plaetinck:

Community effort (not official). I've never seen the use of this, as our torrent files are webseed enabled. (eg you can do pull it with torrent + pull from various mirrors at the same time)

It's an interesting thing to note, and I haven't looked into that so far. For openSUSE, mirrors always turned out to be far more powerful then torrents, since P2P is always throttled by the total uplink capacity. We use it as supplement (through metalinks and separately). I also see torrent capacity be "filled" too slow, and disappearing too soon again, and only useful during the days when a download is extremely popular. But that might also be lack of organization by openSUSE or lack of interest in the community.

...

and I noticed usage for package management being mentioned here: http://en.wikipedia.org/wiki/Metalink#In_use "Metalink is also being tested with Arch Linux's Pacman which is used for package management."

Not sure about that.

...

openSUSE is switching to metalinks for downloads of packages and metadata by the package management with the next release (using aria2c in the background). If you consider doing something similar, MirrorBrain would be the perfect basis to deploy this.

"Simplicity" is on the top of our list of values, metalinks seem to introduce a layer of complexity (but it also brings some niceties). Imho I would look at the option of just using mirrorbrain as mirror manager and redirector, without the metalinks.

Yes, no problem as metalinks are optional.

...

I'd be very happy to help out with getting MirrorBrain up and running for you guys, if you are interested!

That's very nice. I've said it earlier, I think mirrorbrain looks very useful. Though I'm afraid I don't have much spare time in reserve.

I think it always greatly helps if software like this is available in packaged form. I wonder if I could help out with packaging MirrorBrain for Arch Linux; I have experience in packaging RPMs (only).

PS: Will you be at froscon by any chance? Several Arch Linux guys will be there. (I was at fosdem too but missed your talk)

Good to know! I live in Cologne, so that's near. Let's see! Peter