I have a spare IPv4 address (+IPv6) at my work that is useless (part of the
/29 between our providers router and our firewalls).
We have a 100/100 business fibre connection. If I provisioned some
hardware, would it be useful as storage for backups?
Below are the notes of the meeting of thursday.
# arch-devops meeting 2019-10-24
## BBS migration
- The migration was set to happen last week, but was postponed.
- Double check if avatars are migrated properly
- Put cookie seed in the ansible vault
- Set up new date for migration
- Keycloak and GitLab mockup is working kind of well
- Keycloak package's security needs to be improved
- Improve Keycloak package, set up automation for it and put it on a VPS
- Connect GitLab to this Keycloak instance
- Write mail to arch-devops and arch-projects
- Gerardo stepped back as developer
- Building/testing of archiso stuff isn't automated
- Revoke git access for gerardo
- Figure out who can take over upstream development (accept patches, fixes, etc)
## Backup offsite
- Asked rsync.net and they are happy to sponsor 10TB for us
- Waiting on reply from rsync.net
- Find out how borg restoration works and where the encryption keys are actually stored
- Write documentation for adding a host to borg backups
- Write documentation on how to restore on the host and if the host is lost (docs/borg.txt)
## Geo mirror location
- nginx-based approach is ok but only works for HTTP
- Should work with rsync as well
- foxxx0 made a patch and suggestion how to do this using PowerDNS
## Mirroring The Archive
- No mirrored archives
- Make a new ansible group which are allowed in the rsync config of the archive and make the PIA boxes archive.
- Make new ansible role for rsync clients for archive mirrors
- Make new ansible role for rsync server on orion for archive mirror provider
## Make archive of DevOps meeting notes
- Make wiki page with past devops meeting notes (on the devwiki)
first of all apologies for the wall of text, it starts with a brief
motivation, some background info and then goes into the technical
details of a proposed solution. If you want to chime in, please
make sure to carefully read through all of it.
Disclaimer: Might contain personal opinions!
Due to my private local mirror on my NAS not being able to sync from
mirror.pkgbuild.com via rsync anymore, I was made aware of a recent
GeoIP change to this particular host:
I really like the idea of utilizing all the resources available to us
and providing not only us but also our users and Tier 2 mirrors with
the best experience (latency and throughput).
Unfortunately aforementioned change essentially broke the rsync
feature of that particular hostname, as the box to which it currently
resolves (22.214.171.124 and 2a01:4f8:c2c:c62f::1) has no rsyncd
running anymore and simply refuses connection attempts from rsync
My current workaround is to hardcode ger.mirror.pkgbuild.com as the
mirror-url in my sync script.
Obviously that is not ideal and I'm interested in having a proper
solution for this.
The two main issues that we are currently facing are:
- broken rsync
- essentially the box providing mirror.pkgbuild.com is still a
SPOF, as all clients (no matter their location) will always
connect to that one nginx instance
Then, a brief discussion and brainstorming took place in our TU
channel, during which I promised to write something up and post it to
this list in order to gather some more thoughts on it and make the
discussion public and easier to partake in.
So here we go:
It is my understanding that geoip based loadbalancing and redirection
usually happens on DNS level, which I fully support and in the
following I want to propose a possible scenario on how we could do
Generally speaking, one should not run their own authoritative DNS
servers, getting this right is not trivial and requires not only
some hard brainwork but also the necessary infrastructure to do it
Therefore I am proposing a delegated child zone, leaving pkgbuild.com
whereever it currently resides (guessing from the NS records, that is
Let's assume, that particular child zone is called mirror.pkgbuild.com,
because that is where we want to do the geoip stuff.
The contents of pkgbuild.com would still be managed within the Hetzner
interface and served by the Hetzner nameservers, no change there.
The pkgbuild.com zone would need to be slightly adjusted so that it
delegates the zone mirror.pkgbuild.com to a new set of nameservers,
which we will host ourselves.
For that purpose we could either spin up 2 or 3 little Hetzner Cloud
instances, preferably in different locations, i.e. one in DE and one
in FI. Or we could just add the DNS service to our existing machines,
given that we already have multiple servers up and running.
As I have grown fond of PowerDNS over the last couple of years due to
their software simply being a bless to work with and their upstream
being super helpful and quick to resolve issues, I would suggest their
authoritative nameserver "pdns" as the right tool for the job.
Not only does it support numerous backends but also GeoIP and even LUA
records, with which you can do amazing and elaborate things.
For demonstration purposes on how this would work I spent a couple of
hours today sticking together a POC on my own setup, during which I
stumled upon one actual bug and one unexpected challenge.
The bug has been reported in
https://github.com/PowerDNS/pdns/issues/8438 and already has a
proposed fix in https://github.com/PowerDNS/pdns/pull/8439 , thanks
again to Aki for quickly jumping in over on IRC and debugging this
For my POC I have currently rebuild our powerdns package locally with
that patch applied and installed it on my machines.
The challenge is a bit more complex:
Currently PowerDNS has no full replication support for zones containing
LUA records. Upstream has the intention to overcome that, but they are
not sure on how to approach it properly yet.
This means we cannot rely on PowerDNS doing the replication between
our instances and need to come up with our own solution.
Having spent a bit of time thinking about this, I managed to come up
with two feasible solutions, at least from my perspective:
- use PostgreSQL as backend and then setup postgres replication
- use simple BIND zonefiles backend, template them with ansible
and use ansible to push changes to all nodes running PowerDNS
Obviously the postgres replication is more complex and needs some more
work to set it up and also to maintain it (i.e. when adding or
removing replication slaves). However it would allow using all
features of the PowerDNS builtin tooling (e.g. pdnsutil) on the master
node and changes would automatically be distributed to the replication
On the other hand the BIND zonefiles (which are just textfiles), are
way easier to setup and would fit right in with our ansible setup.
Seeing that the current nginx geoip stuff is configured with ansible
anyways, I personally think it would not be cumbersome to manage that
one mirror.pkgbuild.com child zone within our ansible repo.
Enough of the specifics, on to the fun part: My POC.
You can try it out yourselves and I strongly encourage you to do that
and let me know your results so that we can do some actual
testing with my POC.
The only thing you need is extra/bind-tools.
The following DNS query should return the continent where your
resolver (ISP, cloudflare, google, or whatever resolver you are
currently using) is located in:
# dig +short -t TXT continent.geotest.foxxx0.de
That should hopefully match your actual continent and the response
from the following query, otherwise it is likely that other GeoIP DNS
stuff is not working properly for you at all:
! CAUTION: this will leak your client's public IP address. !
(I am not logging those though, just FYI)
# dig +short -t TXT continent.geotest.foxxx0.de @ns1.nano-srv.net
This will directly query my authoritative nameserver from your local
I then went ahead and re-created our mirror scenario, which you can
try out using the following query:
(Again, you may repeat it with '@ns1.nano-srv.net')
# host mirror.geotest.foxxx0.de
mirror.geotest.foxxx0.de is an alias for ger.mirror.geotest.foxxx0.de.
ger.mirror.geotest.foxxx0.de has address 127.0.0.1
ger.mirror.geotest.foxxx0.de has IPv6 address fe80::1
This will return the best suited mirror using a CNAME record, and then
resolve that record to an IP. For my POC I have used some localhost
and link-local addresses:
ger.mirror.geotest.foxxx0.de IN A 127.0.0.1
ger.mirror.geotest.foxxx0.de IN AAAA fe80::1
jpn.mirror.geotest.foxxx0.de IN A 127.0.0.2
jpn.mirror.geotest.foxxx0.de IN AAAA fe80::2
mex.mirror.geotest.foxxx0.de IN A 127.0.0.3
mex.mirror.geotest.foxxx0.de IN AAAA fe80::3
sgp.mirror.geotest.foxxx0.de IN A 127.0.0.4
sgp.mirror.geotest.foxxx0.de IN AAAA fe80::4
I suppose the whole PowerDNS stuff could easily be condensed into a
single ansible role and I would offer my assistance on that as I've done
my fair share of ansible stuff in the past already.
Thanks to everyone who kept reading until here and please let me know
what you think about it, what you like, what you dislike, what you
Thore "foxxx0" Bödecker
GPG ID: 0xEB763B4E9DB887A6
GPG FP: 051E AD6A 6155 389D 69DA 02E5 EB76 3B4E 9DB8 87A6
Although I'm reading the infrastructure topics with interest, I did not do
anything on our infrastructures since a long time now.
So, I would like to officially resign from the team.
Sébastien "Seblu" Luttringer
I'm sending the summary of yesterday's meeting:
Mirroring of Arch Linux Archive
runner2.archlinux.org becomes ger.mirror.pkgbuild.com
use all PIA boxes as public mirrors for Archive AND repos
Sven will be in handle of these
Gitlab is running, we need to re-package keyclock for better security, anthraxx will handle this.
Make planet.archlinux.org part of archweb
Ask fedora regarding their mailman migration
Create a package and test migration - Jerome will look into this
Lack of overview of services/hosts
It was decided to migrate it to a VPS and then move to discourse. Possibly with SSO already.
We are going to point internal mirroring to repos.archlinux.org
Create new server and migrate content for public_html.
Getting a storage box instead of vostok
Move to storage box for backups (create a kanban ticket and figure out automation). We currently pay 30 euros
Backup the archive
Requires a larger backup server (currently 1.2TB free on vostok)
Decide if borg will handle it or we are doubling the backup with another tool. Settle on external service:
online.net cold storage
I have summed up a few things for clarity, but I guess it covers what we've discussed.
This is a reminder for our bi-weekly devops meeting tomorrow 10-10-2019
at 19:30 - 20:30 UTC. I propose to host this edition on mumble, as it
allows for easier and more effective communication. (Sven can we use
your mumble instance?).
The agenda for the meeting is:
* Mirroring the Arch Linux Archive
* Gitlab PoC update
* Planet migration
* Mailman 2 status
* Lack of overview of services/hosts
Jelle van der Waa