[arch-mirrors] CDN based/caching mirror?

[Kristian Klausen]

On 27.01.2020 13.15, Giancarlo Razzolini wrote:
> Em janeiro 26, 2020 13:19 Kristian Klausen via arch-mirrors escreveu:
>>
>> So instead of mirroring the whole thing, the idea is to mirror only 
>> the database files (core.db etc) and download the packages on demand 
>> from a Tier 1 mirror (and let nginx cache them). By doing it that 
>> way, I only download requested packages from the Tier 1 mirrors, 
>> instead of downloading the whole thing (saving Tier 1 bandwidth).
>>
>
> How you actually serve the files is irrelevant to us, but it might not be
> irrelevant to the users, even if they don't actually *know* how you're 
> serving
> files. I see some latency problems that can happen with such scenario.

I'm not sure latency is a big issue, the mirror won't be faster than the 
"weakest link" (the upstream Tier 1 server) if the package isn't cached 
and it cost a few roundtrips + TCP ramp up, but that's about it.
(nginx doesn't need to receive the whole response first, it can "stream" 
the data to the client as it receive it from the upstream server)

> Also, as pointed already, Tier 1 mirrors don't need their bandwidth 
> saved really.

I got that impression from the wiki, but private mirror != 
official/public mirror.
https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors

>
>> To provide even better performance a CDN (ex: Cloudflare) could be 
>> used to provide more caching. So we end up with a setup like this:
>> Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple 
>> upstream)
>>
>
> You'll have even more latency issues for packages you have not cached 
> yet.

See my previous comment.

> I wonder how are you going to do invalidatios.

I'm under the impression that package files never changes? That database 
files do change though, but I think that can be easily handled by either 
using a low "max-age" (1-5 min) or simply using "no-cache" ("Caches must 
check with the origin server for validation before using the cached copy.")

>
>> Do I miss something? Is this a bad idea?
>> If I do setup a mirror like that, is there any chance it could be 
>> added as a official mirror?
>>
>
> As I've said, we won't care how you serve the files. As long as you 
> serve them and
> serve the right file. But users will.

In that case, I will see if I can come up with a solution and open a 
"feature request" to be a official mirror if it works out.

>
> Regards,
> Giancarlo Razzolini
>