[pacman-dev] MirrorBrain for mirrors?

[Peter Poeml]

Hi!

To quickly introduce myself, I am the author of MirrorBrain and I
noticed you discussion here.

On Tue, Mar 10, 2009 at 11:33:36 +0100, Dieter Plaetinck wrote:
> On Tue, 10 Mar 2009 15:40:43 +1100
> "Tim G" <tim at digitalpacific.com.au> wrote:
> 
> > Hey everyone,
> > 
> >  
> > 
> > I am not sure if it has been brought up before, but has anyone
> > thought about providing a MirrorBrain http repository? OpenSUSE has
> > one at http://download.opensuse.org/ and it works quite well. If we
> > are able to provide something like this, it would mean that we could
> > provide transparent mirroring. MirrorBrain even looks after creating
> > mirror lists
> > 
> >  
> > 
> > http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-KDE4-LiveCD
> > -i686.iso?mirrorlist

MirrorBrain also creates Metalinks. Didn't you guys start using
those? I saw http://bbs.archlinux.org/viewtopic.php?pid=573250
(regarding ISO images) and I noticed usage for package management being
mentioned here:
http://en.wikipedia.org/wiki/Metalink#In_use
"Metalink is also being tested with Arch Linux's Pacman which is used for
package management."

openSUSE is switching to metalinks for downloads of packages and
metadata by the package management with the next release (using aria2c
in the background).  If you consider doing something similar,
MirrorBrain would be the perfect basis to deploy this.

I'd be very happy to help out with getting MirrorBrain up and running
for you guys, if you are interested!

> > Anyways, have a look at it, http://mirrorbrain.org/features
> > 
> 
> Wow this looks cool.  It could save us from some pratical issues we
> have (such as "don't give out links like archlinux.org/.., we don't want
> to much traffic there").  If I understand this correctly we can use
> archlinux.org links everywhere and the redirection stuff will be taken
> care of automatically. (that way we maybe don't need to throttle the
> main mirror anymore?)

Yes.

> Also the tool to check which mirrors are incomplete and redirect
> accordingly is nice.
> 
> Though if archlinux.org goes down, and pretty much everything links to
> it, we have a bigger problem.

There can be more than one server - you could have a setup with failover
server, or actually run different servers on different continents. On
the other hand, openSUSE has been running with a single machine for over
two years now (due to regrettable lack of resources...), and other than
hard disk failures or human error, there were no issues with the setup.

That said, the best (and easiest) way to achieve failover would probably
be to run two servers (US, Europe) and use GeoDNS to distribute requests
geographically to them. If one goes down, just switch the clients over
to the other one by a little DNS change (keeping the TTL to 10 minutes or
so.) At the same time, such a setup has the advantage of giving clients
a faster response due to less roundtrip time.

If you want to know more, please let me know. I'm subscribed to the
pacman-dev mailing list, and will stay for a little while, but it would
be safe to reach me at my email, or you could just post to the
mirrorbrain at mirrorbrain.org mailing list (you don't need to subscribe
first).

Thanks,
Peter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://www.archlinux.org/pipermail/pacman-dev/attachments/20090703/758839ea/attachment.pgp>