- Arch-devops - lists.archlinux.org

[arch-devops] Another attempt at (geolocation-aware) loadbalancing of our mirrors
by Thore Bödecker 25 Oct '19

25 Oct '19

Howdy everyone, first of all apologies for the wall of text, it starts with a brief motivation, some background info and then goes into the technical details of a proposed solution. If you want to chime in, please make sure to carefully read through all of it. Disclaimer: Might contain personal opinions! Due to my private local mirror on my NAS not being able to sync from mirror.pkgbuild.com via rsync anymore, I was made aware of a recent GeoIP change to this particular host: https://git.archlinux.org/infrastructure.git/commit/?id=04ef9fb819b12d75018… and https://git.archlinux.org/infrastructure.git/commit/?id=d0270fac570a12a308e… I really like the idea of utilizing all the resources available to us and providing not only us but also our users and Tier 2 mirrors with the best experience (latency and throughput). Unfortunately aforementioned change essentially broke the rsync feature of that particular hostname, as the box to which it currently resolves (78.46.209.220 and 2a01:4f8:c2c:c62f::1) has no rsyncd running anymore and simply refuses connection attempts from rsync clients. My current workaround is to hardcode ger.mirror.pkgbuild.com as the mirror-url in my sync script. Obviously that is not ideal and I'm interested in having a proper solution for this. The two main issues that we are currently facing are: - broken rsync - essentially the box providing mirror.pkgbuild.com is still a SPOF, as all clients (no matter their location) will always connect to that one nginx instance Then, a brief discussion and brainstorming took place in our TU channel, during which I promised to write something up and post it to this list in order to gather some more thoughts on it and make the discussion public and easier to partake in. So here we go: It is my understanding that geoip based loadbalancing and redirection usually happens on DNS level, which I fully support and in the following I want to propose a possible scenario on how we could do that. Generally speaking, one should not run their own authoritative DNS servers, getting this right is not trivial and requires not only some hard brainwork but also the necessary infrastructure to do it properly. Therefore I am proposing a delegated child zone, leaving pkgbuild.com whereever it currently resides (guessing from the NS records, that is probably Hetzner). Let's assume, that particular child zone is called mirror.pkgbuild.com, because that is where we want to do the geoip stuff. The contents of pkgbuild.com would still be managed within the Hetzner interface and served by the Hetzner nameservers, no change there. The pkgbuild.com zone would need to be slightly adjusted so that it delegates the zone mirror.pkgbuild.com to a new set of nameservers, which we will host ourselves. For that purpose we could either spin up 2 or 3 little Hetzner Cloud instances, preferably in different locations, i.e. one in DE and one in FI. Or we could just add the DNS service to our existing machines, given that we already have multiple servers up and running. As I have grown fond of PowerDNS over the last couple of years due to their software simply being a bless to work with and their upstream being super helpful and quick to resolve issues, I would suggest their authoritative nameserver "pdns" as the right tool for the job. Not only does it support numerous backends but also GeoIP and even LUA records, with which you can do amazing and elaborate things. For demonstration purposes on how this would work I spent a couple of hours today sticking together a POC on my own setup, during which I stumled upon one actual bug and one unexpected challenge. The bug has been reported in https://github.com/PowerDNS/pdns/issues/8438 and already has a proposed fix in https://github.com/PowerDNS/pdns/pull/8439 , thanks again to Aki for quickly jumping in over on IRC and debugging this with me. For my POC I have currently rebuild our powerdns package locally with that patch applied and installed it on my machines. The challenge is a bit more complex: Currently PowerDNS has no full replication support for zones containing LUA records. Upstream has the intention to overcome that, but they are not sure on how to approach it properly yet. This means we cannot rely on PowerDNS doing the replication between our instances and need to come up with our own solution. Having spent a bit of time thinking about this, I managed to come up with two feasible solutions, at least from my perspective: - use PostgreSQL as backend and then setup postgres replication - use simple BIND zonefiles backend, template them with ansible and use ansible to push changes to all nodes running PowerDNS Obviously the postgres replication is more complex and needs some more work to set it up and also to maintain it (i.e. when adding or removing replication slaves). However it would allow using all features of the PowerDNS builtin tooling (e.g. pdnsutil) on the master node and changes would automatically be distributed to the replication slaves. On the other hand the BIND zonefiles (which are just textfiles), are way easier to setup and would fit right in with our ansible setup. Seeing that the current nginx geoip stuff is configured with ansible anyways, I personally think it would not be cumbersome to manage that one mirror.pkgbuild.com child zone within our ansible repo. Enough of the specifics, on to the fun part: My POC. You can try it out yourselves and I strongly encourage you to do that and let me know your results so that we can do some actual testing with my POC. The only thing you need is extra/bind-tools. The following DNS query should return the continent where your resolver (ISP, cloudflare, google, or whatever resolver you are currently using) is located in: # dig +short -t TXT continent.geotest.foxxx0.de "EU" That should hopefully match your actual continent and the response from the following query, otherwise it is likely that other GeoIP DNS stuff is not working properly for you at all: ! CAUTION: this will leak your client's public IP address. ! (I am not logging those though, just FYI) # dig +short -t TXT continent.geotest.foxxx0.de @ns1.nano-srv.net "EU" This will directly query my authoritative nameserver from your local client. I then went ahead and re-created our mirror scenario, which you can try out using the following query: (Again, you may repeat it with '@ns1.nano-srv.net') # host mirror.geotest.foxxx0.de mirror.geotest.foxxx0.de is an alias for ger.mirror.geotest.foxxx0.de. ger.mirror.geotest.foxxx0.de has address 127.0.0.1 ger.mirror.geotest.foxxx0.de has IPv6 address fe80::1 This will return the best suited mirror using a CNAME record, and then resolve that record to an IP. For my POC I have used some localhost and link-local addresses: ger.mirror.geotest.foxxx0.de IN A 127.0.0.1 ger.mirror.geotest.foxxx0.de IN AAAA fe80::1 jpn.mirror.geotest.foxxx0.de IN A 127.0.0.2 jpn.mirror.geotest.foxxx0.de IN AAAA fe80::2 mex.mirror.geotest.foxxx0.de IN A 127.0.0.3 mex.mirror.geotest.foxxx0.de IN AAAA fe80::3 sgp.mirror.geotest.foxxx0.de IN A 127.0.0.4 sgp.mirror.geotest.foxxx0.de IN AAAA fe80::4 I suppose the whole PowerDNS stuff could easily be condensed into a single ansible role and I would offer my assistance on that as I've done my fair share of ansible stuff in the past already. Thanks to everyone who kept reading until here and please let me know what you think about it, what you like, what you dislike, what you would change. Cheers -- Thore "foxxx0" Bödecker GPG ID: 0xEB763B4E9DB887A6 GPG FP: 051E AD6A 6155 389D 69DA 02E5 EB76 3B4E 9DB8 87A6

2 2

[arch-devops] Resigning from the devops team
by Sébastien Luttringer 25 Oct '19

25 Oct '19

Hello, Although I'm reading the infrastructure topics with interest, I did not do anything on our infrastructures since a long time now. So, I would like to officially resign from the team. Regards, Sébastien "Seblu" Luttringer

3 2

[arch-devops] DevOps meeting
by Jelle van der Waa 24 Oct '19

24 Oct '19

The agenda for the meeting in 10 minutes is: - Gitlab - BBS migration - Backup offsite - Geo mirror location Greetings, Jelle van der Waa

1 0

[arch-devops] Devops meeting 19:30 - 20:30 UTC
by Jelle van der Waa 24 Oct '19

24 Oct '19

Hi all, The agenda for the meeting in 10 minutes is: - Gitlab - Forum migration - Backups offsite - Geo mirror location

1 0

[arch-devops] Devops meeting summary - 09/10
by Giancarlo Razzolini 23 Oct '19

23 Oct '19

Hi All, I'm sending the summary of yesterday's meeting: Mirroring of Arch Linux Archive runner2.archlinux.org becomes ger.mirror.pkgbuild.com use all PIA boxes as public mirrors for Archive AND repos Sven will be in handle of these Gitlab PoC Gitlab is running, we need to re-package keyclock for better security, anthraxx will handle this. Planet migration Make planet.archlinux.org part of archweb Mailman 3 Ask fedora regarding their mailman migration Create a package and test migration - Jerome will look into this Blocker https://gitlab.com/mailman/mailman/issues/343 Lack of overview of services/hosts Deferred Twitter Deferred Forum migration It was decided to migrate it to a VPS and then move to discourse. Possibly with SSO already. Soyuz retirement We are going to point internal mirroring to repos.archlinux.org Create new server and migrate content for public_html. Getting a storage box instead of vostok Move to storage box for backups (create a kanban ticket and figure out automation). We currently pay 30 euros Backup the archive Requires a larger backup server (currently 1.2TB free on vostok) Offsite backups Decide if borg will handle it or we are doubling the backup with another tool. Settle on external service: borgbase glacier online.net cold storage dedicated server I have summed up a few things for clarity, but I guess it covers what we've discussed. Regards, Giancarlo Razzolini

6 10

[arch-devops] Meeting Tomorrow 19:30 - 20:30 UTC
by Jelle van der Waa 11 Oct '19

11 Oct '19

Hi All, This is a reminder for our bi-weekly devops meeting tomorrow 10-10-2019 at 19:30 - 20:30 UTC. I propose to host this edition on mumble, as it allows for easier and more effective communication. (Sven can we use your mumble instance?). The agenda for the meeting is: * Mirroring the Arch Linux Archive * Gitlab PoC update * Planet migration * Mailman 2 status * Lack of overview of services/hosts Greetings, Jelle van der Waa

4 5

[arch-devops] Maintenance mode
by Giancarlo Razzolini 25 Sep '19

25 Sep '19

Hi All, Ever since I've done some of our migrations, I've used to put our services into maintenance mode manually, by copying the nginx.conf file over the service and reloading nginx. I have been toying with the idea of having a maintenance role for our services for some time, but I never came up with a good way of doing this. Yesterday I've drafted some stuff and now we have a maintenance role that can be "plugged" into other roles. I have added the maintenance mode for most of our roles already, and basically anything that has a nginx configuration can have this maintenance mode. I've added some documentation to the README file. But basically all you need to do is to pass -e maintenance=true to command line on the playbook run. If you don't pass it, and run the book again, then the normal nginx configuration is applied and things go back as they were. The next step is to add nginx configuration to allow the ip address of the person running the book through, to be able to test things, but keep the maintenance mode for everyone else. Just don't know yet how to glue the specifics. Regards, Giancarlo Razzolini

1 0

[arch-devops] Devops promotion process - Levente
by Giancarlo Razzolini 12 Sep '19

12 Sep '19

Hi Guys, Since we don't have a formal process for giving out the keys of the kingdom to people, I thought writing an email would be the best way to start this process. We recently removed root privileges of some guys and we can always use an extra (capable) hand. I have been trying to get Levente to join -devops for a while now, and it seems the message finally got through. I've talked with Levente and he wants to give a security perspective overview of our roles, it seems, as you might've noticed with the recent sudo change on our build machines. What are your thoughts? (If it were just up to me, I'd given him root access already, so +1). Regards, Giancarlo Razzolini

5 5

Re: [arch-devops] Devops promotion process - Levente
by Jan Alexander Steffens 12 Sep '19

12 Sep '19

+1 from me too; Levente is good.

1 0

[arch-devops] restricting sudoers PATH
by Jelle van der Waa 03 Sep '19

03 Sep '19

Hi all, Thanks to anthraxx, we now restrict the PATH which `sudo extra-x86_64-build` and other sudoers specific infra uses using restrict_path. To circumvent users overridding their own PATH with tools which are used in our build scripts which basically allows privilege escalation. [1] This shouldn't cause any issues, if they do contact me or anthraxx. [1] https://git.archlinux.org/infrastructure.git/commit/?id=1eb1dd41f8c734380a3… Greetings, Jelle

3 2