[arch-devops] Another attempt at (geolocation-aware) loadbalancing of our mirrors

Sat Oct 19 18:18:37 UTC 2019

Howdy everyone,

first of all apologies for the wall of text, it starts with a brief
motivation, some background info and then goes into the technical
details of a proposed solution. If you want to chime in, please
make sure to carefully read through all of it.

Disclaimer: Might contain personal opinions!

Due to my private local mirror on my NAS not being able to sync from
mirror.pkgbuild.com via rsync anymore, I was made aware of a recent
GeoIP change to this particular host:
https://git.archlinux.org/infrastructure.git/commit/?id=04ef9fb819b12d75018231cc63868267eba9059b
and
https://git.archlinux.org/infrastructure.git/commit/?id=d0270fac570a12a308e880cf290febad190b033c

I really like the idea of utilizing all the resources available to us
and providing not only us but also our users and Tier 2 mirrors with
the best experience (latency and throughput).

Unfortunately aforementioned change essentially broke the rsync
feature of that particular hostname, as the box to which it currently
resolves (78.46.209.220 and 2a01:4f8:c2c:c62f::1) has no rsyncd
running anymore and simply refuses connection attempts from rsync
clients.

My current workaround is to hardcode ger.mirror.pkgbuild.com as the
mirror-url in my sync script.
Obviously that is not ideal and I'm interested in having a proper
solution for this.

The two main issues that we are currently facing are:
    - broken rsync
    - essentially the box providing mirror.pkgbuild.com is still a
      SPOF, as all clients (no matter their location) will always
      connect to that one nginx instance

Then, a brief discussion and brainstorming took place in our TU
channel, during which I promised to write something up and post it to
this list in order to gather some more thoughts on it and make the
discussion public and easier to partake in.

So here we go:

It is my understanding that geoip based loadbalancing and redirection
usually happens on DNS level, which I fully support and in the
following I want to propose a possible scenario on how we could do
that.

Generally speaking, one should not run their own authoritative DNS
servers, getting this right is not trivial and requires not only
some hard brainwork but also the necessary infrastructure to do it
properly.
Therefore I am proposing a delegated child zone, leaving pkgbuild.com
whereever it currently resides (guessing from the NS records, that is
probably Hetzner).
Let's assume, that particular child zone is called mirror.pkgbuild.com,
because that is where we want to do the geoip stuff.
The contents of pkgbuild.com would still be managed within the Hetzner
interface and served by the Hetzner nameservers, no change there.
The pkgbuild.com zone would need to be slightly adjusted so that it
delegates the zone mirror.pkgbuild.com to a new set of nameservers,
which we will host ourselves.
For that purpose we could either spin up 2 or 3 little Hetzner Cloud
instances, preferably in different locations, i.e. one in DE and one
in FI. Or we could just add the DNS service to our existing machines,
given that we already have multiple servers up and running.

As I have grown fond of PowerDNS over the last couple of years due to
their software simply being a bless to work with and their upstream
being super helpful and quick to resolve issues, I would suggest their
authoritative nameserver "pdns" as the right tool for the job.
Not only does it support numerous backends but also GeoIP and even LUA
records, with which you can do amazing and elaborate things.

For demonstration purposes on how this would work I spent a couple of
hours today sticking together a POC on my own setup, during which I
stumled upon one actual bug and one unexpected challenge.
The bug has been reported in
https://github.com/PowerDNS/pdns/issues/8438 and already has a
proposed fix in https://github.com/PowerDNS/pdns/pull/8439 , thanks
again to Aki for quickly jumping in over on IRC and debugging this
with me.
For my POC I have currently rebuild our powerdns package locally with
that patch applied and installed it on my machines.

The challenge is a bit more complex:
Currently PowerDNS has no full replication support for zones containing
LUA records. Upstream has the intention to overcome that, but they are
not sure on how to approach it properly yet.
This means we cannot rely on PowerDNS doing the replication between
our instances and need to come up with our own solution.
Having spent a bit of time thinking about this, I managed to come up
with two feasible solutions, at least from my perspective:
    - use PostgreSQL as backend and then setup postgres replication
    - use simple BIND zonefiles backend, template them with ansible
      and use ansible to push changes to all nodes running PowerDNS

Obviously the postgres replication is more complex and needs some more
work to set it up and also to maintain it (i.e. when adding or
removing replication slaves). However it would allow using all
features of the PowerDNS builtin tooling (e.g. pdnsutil) on the master
node and changes would automatically be distributed to the replication
slaves.

On the other hand the BIND zonefiles (which are just textfiles), are
way easier to setup and would fit right in with our ansible setup.
Seeing that the current nginx geoip stuff is configured with ansible
anyways, I personally think it would not be cumbersome to manage that
one mirror.pkgbuild.com child zone within our ansible repo.

Enough of the specifics, on to the fun part: My POC.

You can try it out yourselves and I strongly encourage you to do that
and let me know your results so that we can do some actual
testing with my POC.
The only thing you need is extra/bind-tools.

The following DNS query should return the continent where your
resolver (ISP, cloudflare, google, or whatever resolver you are
currently using) is located in:

# dig +short -t TXT continent.geotest.foxxx0.de
"EU"

That should hopefully match your actual continent and the response
from the following query, otherwise it is likely that other GeoIP DNS
stuff is not working properly for you at all:
! CAUTION: this will leak your client's public IP address. !
(I am not logging those though, just FYI)

# dig +short -t TXT continent.geotest.foxxx0.de @ns1.nano-srv.net
"EU"

This will directly query my authoritative nameserver from your local
client.

I then went ahead and re-created our mirror scenario, which you can
try out using the following query:
(Again, you may repeat it with '@ns1.nano-srv.net')

# host mirror.geotest.foxxx0.de
mirror.geotest.foxxx0.de is an alias for ger.mirror.geotest.foxxx0.de.
ger.mirror.geotest.foxxx0.de has address 127.0.0.1
ger.mirror.geotest.foxxx0.de has IPv6 address fe80::1

This will return the best suited mirror using a CNAME record, and then
resolve that record to an IP. For my POC I have used some localhost
and link-local addresses:

ger.mirror.geotest.foxxx0.de  IN  A     127.0.0.1
ger.mirror.geotest.foxxx0.de  IN  AAAA  fe80::1
jpn.mirror.geotest.foxxx0.de  IN  A     127.0.0.2
jpn.mirror.geotest.foxxx0.de  IN  AAAA  fe80::2
mex.mirror.geotest.foxxx0.de  IN  A     127.0.0.3
mex.mirror.geotest.foxxx0.de  IN  AAAA  fe80::3
sgp.mirror.geotest.foxxx0.de  IN  A     127.0.0.4
sgp.mirror.geotest.foxxx0.de  IN  AAAA  fe80::4

I suppose the whole PowerDNS stuff could easily be condensed into a
single ansible role and I would offer my assistance on that as I've done
my fair share of ansible stuff in the past already.

Thanks to everyone who kept reading until here and please let me know
what you think about it, what you like, what you dislike, what you
would change.

Cheers

-- 
Thore "foxxx0" Bödecker

GPG ID: 0xEB763B4E9DB887A6
GPG FP: 051E AD6A 6155 389D 69DA  02E5 EB76 3B4E 9DB8 87A6
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20191019/60611d39/attachment.sig>