[arch-mirrors] [sysadmin] CDN based/caching mirror?

[Kristian Klausen]

On 30.01.2020 17.04, Konstantin Ryabitsev wrote:
> On Sun, 26 Jan 2020 at 11:19, Kristian Klausen via arch-mirrors
> <arch-mirrors at archlinux.org> wrote:
>> I'm considering setting up a Arch Linux mirror and I'm considering a
>> different design.
>>
>> So instead of mirroring the whole thing, the idea is to mirror only the
>> database files (core.db etc) and download the packages on demand from a
>> Tier 1 mirror (and let nginx cache them). By doing it that way, I only
>> download requested packages from the Tier 1 mirrors, instead of
>> downloading the whole thing (saving Tier 1 bandwidth).
>>
>> To provide even better performance a CDN (ex: Cloudflare) could be used
>> to provide more caching. So we end up with a setup like this:
>> Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple upstream)
>>
>> Do I miss something? Is this a bad idea?
> If you are trying to save Tier1 some bandwidth, you'll probably
> actually end up causing them more problems due to increased random
> seek waits. Tier1 mirrors may not necessarily have fast storage -- for
> example, all kernel.org mirror nodes have terabytes of spinning rust
> and about half-a-TB of ssd used via lvm-cache. It works great for
> Tier1 setups because most Tier2 mirrors want the same set of recent
> updates that are served out of ssd cache. If a new mirror comes along
> and wants to slurp and entire distro, that is fine too, because even
> if there's higher iowait latency, the Tier2 mirror isn't working
> against any HTTP timeouts or impatient clients and doesn't care if the
> data arrives at a slower rate due to higher iowait. Tier1 can also
> tell Tier2 mirror "I'm overloaded right now, please try again later"
> and it'll be fine as most Tier2 mirrors can wait an hour or two before
> receiving updates.
>
> Making Tier1 mirrors a "cold cache" for your setup will likely cause
> more disk thrash for them, but will also result in poorer service for
> people using your mirror due to the reasons I listed above.

Tier 1 mirrors is also used directly by end-users (correct me if I'm wrong)
So worst-case (cache miss) my SSD-backed shared cache won't be 
noticeable slower than pulling directly from the Tier 1 mirror. 
Best-case (cache hit) I'm saving the Tier 1 mirror some bandwidth and 
disk usage.
My idea is basically tiered caching (CDN -> Nginx SSD-backed shared 
cache -> Tier 1 mirror(s)), is that worse than status quo? :)

> If someone
> tries to install a package and watches their download bar sit at 0 for
> half a minute due to backend proxies fetching data from Tier1 origin,
> that's going to result in frustrated people.

Nginx streams the data as it is received from the upstream server, so 
worst-case (cache miss) the data can be delivered as fast as received 
from the upstream server.

> TL;DR: If you can afford CDN-fronting your mirror, that should be
> mostly fine, but I would recommend against using Tier1 as your
> cache-miss backend. Storage is cheap and most Tier1 mirrors have
> unlimited bandwidth, so just run a Tier2 mirror (with slow/fast
> storage caching) and keep local copies of everything.
>
> -K
> (mirrors.kernel.org administrator)