October 2019 - Arch-devops - lists.archlinux.org

[arch-devops] Backup Server
by Phillip Smith 30 Oct '19

30 Oct '19

Hi all, I have a spare IPv4 address (+IPv6) at my work that is useless (part of the /29 between our providers router and our firewalls). We have a 100/100 business fibre connection. If I provisioned some hardware, would it be useful as storage for backups? Cheers, ~p

2 1

[arch-devops] Fwd: Access to aur.git via git-recieve-pack
by Justin Capella 28 Oct '19

28 Oct '19

I think maybe this isn't meant to be accessed directly, and possibly may allow for large data amplification and high server load, intentional or otherwise. https://aur.archlinux.org/cgit/aur.git/info/refs?service=git-recieve-pack&h…

2 2

[arch-devops] Devops meeting summary - 24/10
by Jelle van der Waa 26 Oct '19

26 Oct '19

Hi all, Below are the notes of the meeting of thursday. # arch-devops meeting 2019-10-24 ## BBS migration ### State - The migration was set to happen last week, but was postponed. ### Who - Jelle - fukawi2 ### Actionable - Double check if avatars are migrated properly - Put cookie seed in the ansible vault - Set up new date for migration ## Gitlab ### State - Keycloak and GitLab mockup is working kind of well - Keycloak package's security needs to be improved ### Who - Jerome - Sven ### Actionable - Improve Keycloak package, set up automation for it and put it on a VPS - Connect GitLab to this Keycloak instance - Write mail to arch-devops and arch-projects ## archiso ### State - Gerardo stepped back as developer - Building/testing of archiso stuff isn't automated ### Who - Jelle ### Actionable - Revoke git access for gerardo - Figure out who can take over upstream development (accept patches, fixes, etc) ## Backup offsite ### State - Asked rsync.net and they are happy to sponsor 10TB for us - Waiting on reply from rsync.net ### Who - Sven - Jelle ### Actionable - Find out how borg restoration works and where the encryption keys are actually stored - Write documentation for adding a host to borg backups - Write documentation on how to restore on the host and if the host is lost (docs/borg.txt) ## Geo mirror location ### State - nginx-based approach is ok but only works for HTTP - Should work with rsync as well - foxxx0 made a patch and suggestion how to do this using PowerDNS ### Who - foxxx0 - Sven ### Actionable - Deferred ## Mirroring The Archive ### State - No mirrored archives ### Who - Sven - Jelle ### Actionable - Make a new ansible group which are allowed in the rsync config of the archive and make the PIA boxes archive. - Make new ansible role for rsync clients for archive mirrors - Make new ansible role for rsync server on orion for archive mirror provider ## Make archive of DevOps meeting notes ### Who - Jelle ### Actionable - Make wiki page with past devops meeting notes (on the devwiki)

1 0

[arch-devops] Another attempt at (geolocation-aware) loadbalancing of our mirrors
by Thore Bödecker 25 Oct '19

25 Oct '19

Howdy everyone, first of all apologies for the wall of text, it starts with a brief motivation, some background info and then goes into the technical details of a proposed solution. If you want to chime in, please make sure to carefully read through all of it. Disclaimer: Might contain personal opinions! Due to my private local mirror on my NAS not being able to sync from mirror.pkgbuild.com via rsync anymore, I was made aware of a recent GeoIP change to this particular host: https://git.archlinux.org/infrastructure.git/commit/?id=04ef9fb819b12d75018… and https://git.archlinux.org/infrastructure.git/commit/?id=d0270fac570a12a308e… I really like the idea of utilizing all the resources available to us and providing not only us but also our users and Tier 2 mirrors with the best experience (latency and throughput). Unfortunately aforementioned change essentially broke the rsync feature of that particular hostname, as the box to which it currently resolves (78.46.209.220 and 2a01:4f8:c2c:c62f::1) has no rsyncd running anymore and simply refuses connection attempts from rsync clients. My current workaround is to hardcode ger.mirror.pkgbuild.com as the mirror-url in my sync script. Obviously that is not ideal and I'm interested in having a proper solution for this. The two main issues that we are currently facing are: - broken rsync - essentially the box providing mirror.pkgbuild.com is still a SPOF, as all clients (no matter their location) will always connect to that one nginx instance Then, a brief discussion and brainstorming took place in our TU channel, during which I promised to write something up and post it to this list in order to gather some more thoughts on it and make the discussion public and easier to partake in. So here we go: It is my understanding that geoip based loadbalancing and redirection usually happens on DNS level, which I fully support and in the following I want to propose a possible scenario on how we could do that. Generally speaking, one should not run their own authoritative DNS servers, getting this right is not trivial and requires not only some hard brainwork but also the necessary infrastructure to do it properly. Therefore I am proposing a delegated child zone, leaving pkgbuild.com whereever it currently resides (guessing from the NS records, that is probably Hetzner). Let's assume, that particular child zone is called mirror.pkgbuild.com, because that is where we want to do the geoip stuff. The contents of pkgbuild.com would still be managed within the Hetzner interface and served by the Hetzner nameservers, no change there. The pkgbuild.com zone would need to be slightly adjusted so that it delegates the zone mirror.pkgbuild.com to a new set of nameservers, which we will host ourselves. For that purpose we could either spin up 2 or 3 little Hetzner Cloud instances, preferably in different locations, i.e. one in DE and one in FI. Or we could just add the DNS service to our existing machines, given that we already have multiple servers up and running. As I have grown fond of PowerDNS over the last couple of years due to their software simply being a bless to work with and their upstream being super helpful and quick to resolve issues, I would suggest their authoritative nameserver "pdns" as the right tool for the job. Not only does it support numerous backends but also GeoIP and even LUA records, with which you can do amazing and elaborate things. For demonstration purposes on how this would work I spent a couple of hours today sticking together a POC on my own setup, during which I stumled upon one actual bug and one unexpected challenge. The bug has been reported in https://github.com/PowerDNS/pdns/issues/8438 and already has a proposed fix in https://github.com/PowerDNS/pdns/pull/8439 , thanks again to Aki for quickly jumping in over on IRC and debugging this with me. For my POC I have currently rebuild our powerdns package locally with that patch applied and installed it on my machines. The challenge is a bit more complex: Currently PowerDNS has no full replication support for zones containing LUA records. Upstream has the intention to overcome that, but they are not sure on how to approach it properly yet. This means we cannot rely on PowerDNS doing the replication between our instances and need to come up with our own solution. Having spent a bit of time thinking about this, I managed to come up with two feasible solutions, at least from my perspective: - use PostgreSQL as backend and then setup postgres replication - use simple BIND zonefiles backend, template them with ansible and use ansible to push changes to all nodes running PowerDNS Obviously the postgres replication is more complex and needs some more work to set it up and also to maintain it (i.e. when adding or removing replication slaves). However it would allow using all features of the PowerDNS builtin tooling (e.g. pdnsutil) on the master node and changes would automatically be distributed to the replication slaves. On the other hand the BIND zonefiles (which are just textfiles), are way easier to setup and would fit right in with our ansible setup. Seeing that the current nginx geoip stuff is configured with ansible anyways, I personally think it would not be cumbersome to manage that one mirror.pkgbuild.com child zone within our ansible repo. Enough of the specifics, on to the fun part: My POC. You can try it out yourselves and I strongly encourage you to do that and let me know your results so that we can do some actual testing with my POC. The only thing you need is extra/bind-tools. The following DNS query should return the continent where your resolver (ISP, cloudflare, google, or whatever resolver you are currently using) is located in: # dig +short -t TXT continent.geotest.foxxx0.de "EU" That should hopefully match your actual continent and the response from the following query, otherwise it is likely that other GeoIP DNS stuff is not working properly for you at all: ! CAUTION: this will leak your client's public IP address. ! (I am not logging those though, just FYI) # dig +short -t TXT continent.geotest.foxxx0.de @ns1.nano-srv.net "EU" This will directly query my authoritative nameserver from your local client. I then went ahead and re-created our mirror scenario, which you can try out using the following query: (Again, you may repeat it with '@ns1.nano-srv.net') # host mirror.geotest.foxxx0.de mirror.geotest.foxxx0.de is an alias for ger.mirror.geotest.foxxx0.de. ger.mirror.geotest.foxxx0.de has address 127.0.0.1 ger.mirror.geotest.foxxx0.de has IPv6 address fe80::1 This will return the best suited mirror using a CNAME record, and then resolve that record to an IP. For my POC I have used some localhost and link-local addresses: ger.mirror.geotest.foxxx0.de IN A 127.0.0.1 ger.mirror.geotest.foxxx0.de IN AAAA fe80::1 jpn.mirror.geotest.foxxx0.de IN A 127.0.0.2 jpn.mirror.geotest.foxxx0.de IN AAAA fe80::2 mex.mirror.geotest.foxxx0.de IN A 127.0.0.3 mex.mirror.geotest.foxxx0.de IN AAAA fe80::3 sgp.mirror.geotest.foxxx0.de IN A 127.0.0.4 sgp.mirror.geotest.foxxx0.de IN AAAA fe80::4 I suppose the whole PowerDNS stuff could easily be condensed into a single ansible role and I would offer my assistance on that as I've done my fair share of ansible stuff in the past already. Thanks to everyone who kept reading until here and please let me know what you think about it, what you like, what you dislike, what you would change. Cheers -- Thore "foxxx0" Bödecker GPG ID: 0xEB763B4E9DB887A6 GPG FP: 051E AD6A 6155 389D 69DA 02E5 EB76 3B4E 9DB8 87A6

2 2

[arch-devops] Resigning from the devops team
by Sébastien Luttringer 25 Oct '19

25 Oct '19

Hello, Although I'm reading the infrastructure topics with interest, I did not do anything on our infrastructures since a long time now. So, I would like to officially resign from the team. Regards, Sébastien "Seblu" Luttringer

3 2

[arch-devops] DevOps meeting
by Jelle van der Waa 24 Oct '19

24 Oct '19

The agenda for the meeting in 10 minutes is: - Gitlab - BBS migration - Backup offsite - Geo mirror location Greetings, Jelle van der Waa

1 0

[arch-devops] Devops meeting 19:30 - 20:30 UTC
by Jelle van der Waa 24 Oct '19

24 Oct '19

Hi all, The agenda for the meeting in 10 minutes is: - Gitlab - Forum migration - Backups offsite - Geo mirror location

1 0

[arch-devops] Devops meeting summary - 09/10
by Giancarlo Razzolini 23 Oct '19

23 Oct '19

Hi All, I'm sending the summary of yesterday's meeting: Mirroring of Arch Linux Archive runner2.archlinux.org becomes ger.mirror.pkgbuild.com use all PIA boxes as public mirrors for Archive AND repos Sven will be in handle of these Gitlab PoC Gitlab is running, we need to re-package keyclock for better security, anthraxx will handle this. Planet migration Make planet.archlinux.org part of archweb Mailman 3 Ask fedora regarding their mailman migration Create a package and test migration - Jerome will look into this Blocker https://gitlab.com/mailman/mailman/issues/343 Lack of overview of services/hosts Deferred Twitter Deferred Forum migration It was decided to migrate it to a VPS and then move to discourse. Possibly with SSO already. Soyuz retirement We are going to point internal mirroring to repos.archlinux.org Create new server and migrate content for public_html. Getting a storage box instead of vostok Move to storage box for backups (create a kanban ticket and figure out automation). We currently pay 30 euros Backup the archive Requires a larger backup server (currently 1.2TB free on vostok) Offsite backups Decide if borg will handle it or we are doubling the backup with another tool. Settle on external service: borgbase glacier online.net cold storage dedicated server I have summed up a few things for clarity, but I guess it covers what we've discussed. Regards, Giancarlo Razzolini

6 10

[arch-devops] Meeting Tomorrow 19:30 - 20:30 UTC
by Jelle van der Waa 11 Oct '19

11 Oct '19

Hi All, This is a reminder for our bi-weekly devops meeting tomorrow 10-10-2019 at 19:30 - 20:30 UTC. I propose to host this edition on mumble, as it allows for easier and more effective communication. (Sven can we use your mumble instance?). The agenda for the meeting is: * Mirroring the Arch Linux Archive * Gitlab PoC update * Planet migration * Mailman 2 status * Lack of overview of services/hosts Greetings, Jelle van der Waa

4 5