[arch-devops] Mirrorbrain

[Phillip Smith]

It's been discussed in the past, but I thought it worth having a
semi-formal and documented discussion about it.

Do we think setting up mirrorbrain[0] would be a worthwhile service? For
those not familiar, the elevator pitch is that it's an open-source download
redirector to coordinate a simple CDN. It is used by a number of
open-source projects such as vlc, Gnome, KDE, LibreOffice and OpenSUSE.

Architecturally speaking, it maintains a list of our mirrors and local copy
of our repos, and monitors those mirrors for availability and staleness.
HTTP(s) clients (ie, pacman) are then redirected to an appropriate mirror
based on the clients geographic location and mirror health.

Benefits include:
1. Ensuring users always receive up-to-date packages (mirrorbrain won't
redirect to a mirror if that mirrors version of that package is outdated
compared to the authoritative repository).
2. Automated monitoring of our mirror network to proactively detect stale
or broken mirrors. Mirrorlist files (ie, /etc/pacman.d/mirrorlist) can be
automatically generated based on MirrorBrain's data.
3. Reduced load on core mirrors, and load-balancing (for example, a 1Gbit
mirror can be weighted to receive 10x the traffic of a 100MBps mirror).
4. Automatic MetaLink and Torrent file generation, with web-seeds
(currently handled by hefur on luna?).

Exceptions to redirection can be applied, for example to ensure
security-sensitive files (checksum files perhaps) are always served
directly from the authoritative repo.

Requirements are pretty basic; apache with mod_asn, postgresql, and some
python+perl modules. It can be run behind a reverse-proxy if we wanted to
hide apache behind nginx.

It's not a particular fast-moving project (last release 2014) but that's a
reflection of it's stability IMHO.

What do others think? I'm happy to take on the implementation project.

Cheers,
~p

[0] http://mirrorbrain.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20180912/de677c48/attachment.html>