Howdy everyone, first of all apologies for the wall of text, it starts with a brief motivation, some background info and then goes into the technical details of a proposed solution. If you want to chime in, please make sure to carefully read through all of it. Disclaimer: Might contain personal opinions! Due to my private local mirror on my NAS not being able to sync from mirror.pkgbuild.com via rsync anymore, I was made aware of a recent GeoIP change to this particular host: https://git.archlinux.org/infrastructure.git/commit/?id=04ef9fb819b12d750182... and https://git.archlinux.org/infrastructure.git/commit/?id=d0270fac570a12a308e8... I really like the idea of utilizing all the resources available to us and providing not only us but also our users and Tier 2 mirrors with the best experience (latency and throughput). Unfortunately aforementioned change essentially broke the rsync feature of that particular hostname, as the box to which it currently resolves (78.46.209.220 and 2a01:4f8:c2c:c62f::1) has no rsyncd running anymore and simply refuses connection attempts from rsync clients. My current workaround is to hardcode ger.mirror.pkgbuild.com as the mirror-url in my sync script. Obviously that is not ideal and I'm interested in having a proper solution for this. The two main issues that we are currently facing are: - broken rsync - essentially the box providing mirror.pkgbuild.com is still a SPOF, as all clients (no matter their location) will always connect to that one nginx instance Then, a brief discussion and brainstorming took place in our TU channel, during which I promised to write something up and post it to this list in order to gather some more thoughts on it and make the discussion public and easier to partake in. So here we go: It is my understanding that geoip based loadbalancing and redirection usually happens on DNS level, which I fully support and in the following I want to propose a possible scenario on how we could do that. Generally speaking, one should not run their own authoritative DNS servers, getting this right is not trivial and requires not only some hard brainwork but also the necessary infrastructure to do it properly. Therefore I am proposing a delegated child zone, leaving pkgbuild.com whereever it currently resides (guessing from the NS records, that is probably Hetzner). Let's assume, that particular child zone is called mirror.pkgbuild.com, because that is where we want to do the geoip stuff. The contents of pkgbuild.com would still be managed within the Hetzner interface and served by the Hetzner nameservers, no change there. The pkgbuild.com zone would need to be slightly adjusted so that it delegates the zone mirror.pkgbuild.com to a new set of nameservers, which we will host ourselves. For that purpose we could either spin up 2 or 3 little Hetzner Cloud instances, preferably in different locations, i.e. one in DE and one in FI. Or we could just add the DNS service to our existing machines, given that we already have multiple servers up and running. As I have grown fond of PowerDNS over the last couple of years due to their software simply being a bless to work with and their upstream being super helpful and quick to resolve issues, I would suggest their authoritative nameserver "pdns" as the right tool for the job. Not only does it support numerous backends but also GeoIP and even LUA records, with which you can do amazing and elaborate things. For demonstration purposes on how this would work I spent a couple of hours today sticking together a POC on my own setup, during which I stumled upon one actual bug and one unexpected challenge. The bug has been reported in https://github.com/PowerDNS/pdns/issues/8438 and already has a proposed fix in https://github.com/PowerDNS/pdns/pull/8439 , thanks again to Aki for quickly jumping in over on IRC and debugging this with me. For my POC I have currently rebuild our powerdns package locally with that patch applied and installed it on my machines. The challenge is a bit more complex: Currently PowerDNS has no full replication support for zones containing LUA records. Upstream has the intention to overcome that, but they are not sure on how to approach it properly yet. This means we cannot rely on PowerDNS doing the replication between our instances and need to come up with our own solution. Having spent a bit of time thinking about this, I managed to come up with two feasible solutions, at least from my perspective: - use PostgreSQL as backend and then setup postgres replication - use simple BIND zonefiles backend, template them with ansible and use ansible to push changes to all nodes running PowerDNS Obviously the postgres replication is more complex and needs some more work to set it up and also to maintain it (i.e. when adding or removing replication slaves). However it would allow using all features of the PowerDNS builtin tooling (e.g. pdnsutil) on the master node and changes would automatically be distributed to the replication slaves. On the other hand the BIND zonefiles (which are just textfiles), are way easier to setup and would fit right in with our ansible setup. Seeing that the current nginx geoip stuff is configured with ansible anyways, I personally think it would not be cumbersome to manage that one mirror.pkgbuild.com child zone within our ansible repo. Enough of the specifics, on to the fun part: My POC. You can try it out yourselves and I strongly encourage you to do that and let me know your results so that we can do some actual testing with my POC. The only thing you need is extra/bind-tools. The following DNS query should return the continent where your resolver (ISP, cloudflare, google, or whatever resolver you are currently using) is located in: # dig +short -t TXT continent.geotest.foxxx0.de "EU" That should hopefully match your actual continent and the response from the following query, otherwise it is likely that other GeoIP DNS stuff is not working properly for you at all: ! CAUTION: this will leak your client's public IP address. ! (I am not logging those though, just FYI) # dig +short -t TXT continent.geotest.foxxx0.de @ns1.nano-srv.net "EU" This will directly query my authoritative nameserver from your local client. I then went ahead and re-created our mirror scenario, which you can try out using the following query: (Again, you may repeat it with '@ns1.nano-srv.net') # host mirror.geotest.foxxx0.de mirror.geotest.foxxx0.de is an alias for ger.mirror.geotest.foxxx0.de. ger.mirror.geotest.foxxx0.de has address 127.0.0.1 ger.mirror.geotest.foxxx0.de has IPv6 address fe80::1 This will return the best suited mirror using a CNAME record, and then resolve that record to an IP. For my POC I have used some localhost and link-local addresses: ger.mirror.geotest.foxxx0.de IN A 127.0.0.1 ger.mirror.geotest.foxxx0.de IN AAAA fe80::1 jpn.mirror.geotest.foxxx0.de IN A 127.0.0.2 jpn.mirror.geotest.foxxx0.de IN AAAA fe80::2 mex.mirror.geotest.foxxx0.de IN A 127.0.0.3 mex.mirror.geotest.foxxx0.de IN AAAA fe80::3 sgp.mirror.geotest.foxxx0.de IN A 127.0.0.4 sgp.mirror.geotest.foxxx0.de IN AAAA fe80::4 I suppose the whole PowerDNS stuff could easily be condensed into a single ansible role and I would offer my assistance on that as I've done my fair share of ansible stuff in the past already. Thanks to everyone who kept reading until here and please let me know what you think about it, what you like, what you dislike, what you would change. Cheers -- Thore "foxxx0" Bödecker GPG ID: 0xEB763B4E9DB887A6 GPG FP: 051E AD6A 6155 389D 69DA 02E5 EB76 3B4E 9DB8 87A6