On 27.01.2020 13.15, Giancarlo Razzolini wrote:
Em janeiro 26, 2020 13:19 Kristian Klausen via arch-mirrors escreveu:
So instead of mirroring the whole thing, the idea is to mirror only the database files (core.db etc) and download the packages on demand from a Tier 1 mirror (and let nginx cache them). By doing it that way, I only download requested packages from the Tier 1 mirrors, instead of downloading the whole thing (saving Tier 1 bandwidth).
How you actually serve the files is irrelevant to us, but it might not be irrelevant to the users, even if they don't actually *know* how you're serving files. I see some latency problems that can happen with such scenario.
I'm not sure latency is a big issue, the mirror won't be faster than the "weakest link" (the upstream Tier 1 server) if the package isn't cached and it cost a few roundtrips + TCP ramp up, but that's about it. (nginx doesn't need to receive the whole response first, it can "stream" the data to the client as it receive it from the upstream server)
Also, as pointed already, Tier 1 mirrors don't need their bandwidth saved really.
I got that impression from the wiki, but private mirror != official/public mirror. https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors
To provide even better performance a CDN (ex: Cloudflare) could be used to provide more caching. So we end up with a setup like this: Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple upstream)
You'll have even more latency issues for packages you have not cached yet.
See my previous comment.
I wonder how are you going to do invalidatios.
I'm under the impression that package files never changes? That database files do change though, but I think that can be easily handled by either using a low "max-age" (1-5 min) or simply using "no-cache" ("Caches must check with the origin server for validation before using the cached copy.")
Do I miss something? Is this a bad idea? If I do setup a mirror like that, is there any chance it could be added as a official mirror?
As I've said, we won't care how you serve the files. As long as you serve them and serve the right file. But users will.
In that case, I will see if I can come up with a solution and open a "feature request" to be a official mirror if it works out.
Regards, Giancarlo Razzolini