[arch-mirrors] CDN based/caching mirror?

[Sevki Hasirci]

I can't, I'm no longer at CF, but I'll try to get in touch with the person
I handed this over to and see what's it's state.

On Wed, Mar 4, 2020 at 3:28 AM Ip, Ivan <m at lifeho.me> wrote:

> Hello there,
>
> May I confirm is this official from Cloudflare?
>
> Yours faithfully,
> Ivan Ip<m at lifeho.me>
>
> On Tue, Mar 3, 2020 at 1:55 AM Sevki Hasirci <s at sevki.org> wrote:
> >
> > I'll just leave the one I did when I was at CF here then :)
> https://cloudflaremirrors.com/archlinux
> >
> > On Thu, Feb 27, 2020 at 8:46 PM Kristian Klausen via arch-mirrors <
> arch-mirrors at archlinux.org> wrote:
> >>
> >> On 26.01.2020 17.19, Kristian Klausen via arch-mirrors wrote:
> >> > Hi
> >> >
> >> > I'm considering setting up a Arch Linux mirror and I'm considering a
> >> > different design.
> >> >
> >>
> >> Hi
> >>
> >> I just got time to implement this and the setup looks like this:
> >> Cloudflare -> Cloudflare Workers -> Backblaze B2 bucket <- Tier1 mirror
> >>
> >> The files is synced from mirror.ams1.nl.leaseweb.net every hour to the
> >> Backblaze B2 bucket and they are fetched from the bucket with the help
> >> of a Cloudflare Workers script.
> >> Cloudflare is configured to cache everything (size <=2GB*), database
> >> files is cached for 5 minute everything else is cached for 24 hours.
> >> * CF is sponsoring a plan with a higher limit than the 512MB default
> >>
> >> I have done some quick testing, and time to first byte isn't impressive
> >> (at least not when downloading from Europe), but the speed is acceptable
> >> (80-100MB/s is achievable if the file is cached, and 8-12MB/s if not
> >> (tested from Europe)).
> >>
> >> To make it easier to implement, I took some shortcuts:
> >> * Directory listing isn't implemented
> >> * "latest" files isn't synced
> >> * Only packages in "pool/" is synced, the package files in the different
> >> repo isn't synced, but if you request a package
> >> (\.pkg\.tar\.(xz|zst)(|.sig)$) it is automatic retrieved from the pool/
> >> directory. This means that you can download ex Firefox from both:
> >>
> https://archlinux.amirror.xyz/extra/os/x86_64/firefox-73.0.1-1-x86_64.pkg.tar.zst
> >>
> https://archlinux.amirror.xyz/community/os/x86_64/firefox-73.0.1-1-x86_64.pkg.tar.zst
> >>
> >> I'm not sure if the shortcuts is acceptable, but it can be fixed if it
> >> is a issue.
> >>
> >> Also please note that: archive, other and sources isn't synced.
> >>
> >> Feel free to try it out: https://archlinux.amirror.xyz/
> >>
> >> Best regards
> >> Kristian Klausen
> >>
> >> > So instead of mirroring the whole thing, the idea is to mirror only
> >> > the database files (core.db etc) and download the packages on demand
> >> > from a Tier 1 mirror (and let nginx cache them). By doing it that way,
> >> > I only download requested packages from the Tier 1 mirrors, instead of
> >> > downloading the whole thing (saving Tier 1 bandwidth).
> >> >
> >> > To provide even better performance a CDN (ex: Cloudflare) could be
> >> > used to provide more caching. So we end up with a setup like this:
> >> > Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple
> upstream)
> >> >
> >> > Do I miss something? Is this a bad idea?
> >> > If I do setup a mirror like that, is there any chance it could be
> >> > added as a official mirror?
> >> >
> >> > Best regards
> >> > Kristian Klausen
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.archlinux.org/pipermail/arch-mirrors/attachments/20200304/12fb3217/attachment.htm>