Hi! To quickly introduce myself, I am the author of MirrorBrain and I noticed you discussion here. On Tue, Mar 10, 2009 at 11:33:36 +0100, Dieter Plaetinck wrote:
On Tue, 10 Mar 2009 15:40:43 +1100 "Tim G" <tim at digitalpacific.com.au> wrote:
Hey everyone,
I am not sure if it has been brought up before, but has anyone thought about providing a MirrorBrain http repository? OpenSUSE has one at http://download.opensuse.org/ and it works quite well. If we are able to provide something like this, it would mean that we could provide transparent mirroring. MirrorBrain even looks after creating mirror lists
http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-KDE4-LiveCD -i686.iso?mirrorlist
MirrorBrain also creates Metalinks. Didn't you guys start using those? I saw http://bbs.archlinux.org/viewtopic.php?pid=573250 (regarding ISO images) and I noticed usage for package management being mentioned here: http://en.wikipedia.org/wiki/Metalink#In_use "Metalink is also being tested with Arch Linux's Pacman which is used for package management." openSUSE is switching to metalinks for downloads of packages and metadata by the package management with the next release (using aria2c in the background). If you consider doing something similar, MirrorBrain would be the perfect basis to deploy this. I'd be very happy to help out with getting MirrorBrain up and running for you guys, if you are interested!
Anyways, have a look at it, http://mirrorbrain.org/features
Wow this looks cool. It could save us from some pratical issues we have (such as "don't give out links like archlinux.org/.., we don't want to much traffic there"). If I understand this correctly we can use archlinux.org links everywhere and the redirection stuff will be taken care of automatically. (that way we maybe don't need to throttle the main mirror anymore?)
Yes.
Also the tool to check which mirrors are incomplete and redirect accordingly is nice.
Though if archlinux.org goes down, and pretty much everything links to it, we have a bigger problem.
There can be more than one server - you could have a setup with failover server, or actually run different servers on different continents. On the other hand, openSUSE has been running with a single machine for over two years now (due to regrettable lack of resources...), and other than hard disk failures or human error, there were no issues with the setup. That said, the best (and easiest) way to achieve failover would probably be to run two servers (US, Europe) and use GeoDNS to distribute requests geographically to them. If one goes down, just switch the clients over to the other one by a little DNS change (keeping the TTL to 10 minutes or so.) At the same time, such a setup has the advantage of giving clients a faster response due to less roundtrip time. If you want to know more, please let me know. I'm subscribed to the pacman-dev mailing list, and will stay for a little while, but it would be safe to reach me at my email, or you could just post to the mirrorbrain at mirrorbrain.org mailing list (you don't need to subscribe first). Thanks, Peter