- Arch-devops - lists.archlinux.org

[arch-devops] Meeting 2th July 18:30 UTC - 20:30 CEST
by Jelle van der Waa 30 Jun '20

30 Jun '20

Hi all, Another week, another meeting, topics: * Storagebox for backups * Gemini migration * Removal of hefurd torrent tracker * Security audit Gitlab/Keycloak * Resignation/Addition handling of members to DevOps Greetings, Jelle

1 0

[arch-devops] Broken disk on vostok
by Sven-Hendrik Haase 29 Jun '20

29 Jun '20

Hey all, During a small routine check I noticed that we have a broken disk on vostok. In fact, according to the log, we've had it for 7 months at least which is a bit embarrassing given that this is RAID1 only and it's also our primary backup box. This really goes to show that we need better monitoring/more meaningful alerts. Anyway, I had Hetzner replace the disk and the array is now rebuilding. Cheers, Sven

4 8

[arch-devops] repos.archlinux.org has been migrated to a new server
by Jelle van der Waa 26 Jun '20

26 Jun '20

Hi all, repos.archlinux.org, svn.archlinux.org and rsync.archlinux.org are now on a new server which has plenty of diskspace for us to continue packaging for a while (16T free). Some important notes: The ssh hostkeys changed for repos.archlinux.org which means if you use archco/communityco this will fail without a useful error. So take care of your .ssh/known_hosts. The ssh host keys for gemini/repos.archlinux.org are: # gemini.archlinux.org 1024 SHA256:F1Corf6i2U72yub+CIzzGHLOMVKVnjALh1YHM8gBjxE root(a)gemini.archlinux.org (DSA) 256 SHA256:If51DkTftUpDAFz65totgDfTd/ddu/2w/RBZIHtY74U root(a)gemini.archlinux.org (ECDSA) 256 SHA256:wUrJYf9+zOpIEUQ3ndgarK0PjzPICa1frmu7mpL4e14 root(a)gemini.archlinux.org (ED25519) 3072 SHA256:Rltnuln3bjsHJwVbys/LnYCj7hO6srPoa15JP8QhmlQ root(a)gemini.archlinux.org (RSA) https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/docs/ss… https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/docs/ss… If anything unexpected happens please either mail arch-devops(a)lists.archlinux.org or #archlinux-devops on irc. Greetings, Jelle

2 2

[arch-devops] Migrating hefur from orion
by Jelle van der Waa 25 Jun '20

25 Jun '20

As we now have moved the repos to gemini we have not yet migrated hefurd which is the torrent webseed service. As this is a random c++ application it might be nice to start moving it to a VPS? We can use the kitchensink to rsync the iso to a new vps and run /usr/bin/hefurd -disable-peers-page -torrent-dir /srv/ftp/iso As we used todo, opinions? Greetings, Jelle van der Waa

1 0

[arch-devops] Invalid SSL CN on tracker.archlinux.org
by Egor Kovetskiy 24 Jun '20

24 Jun '20

Hello, tracker.archlinux.org has an invalid certificate with CN = archive.archlinux.org $ openssl s_client -showcerts -connect tracker.archlinux.org:443 <<< "" 2>&1 | grep subject=CN subject=CN = archive.archlinux.org The link was found on https://github.com/archlinux/infrastructure

2 1

[arch-devops] Thursday 25th June 14:00 UTC repos.archlinux.org migration
by Jelle van der Waa 23 Jun '20

23 Jun '20

Hi al, On Thursday 25th June at 14:00 UTC work will begin on migrating the following services to a new machine: * repos.archlinux.org * svn.archlinux.org * Arch mirror infrastructure This will impact all packaging and svn activities as those services are not available during the migration period. The services will be migrated to a new server as diskspace is running low on the current server, the migration is expected to take at least four hours. When the migration is successfully completed an email will be send that the services are available and packaging can resume. TLDR: No packaging can be done on the 25th of June starting at 14:00 UTC till the migration is done. Greetings, Jelle

1 0

[arch-devops] archive.archlinux.org migrated to gemini.archlinux.org
by Jelle van der Waa 21 Jun '20

21 Jun '20

Hi all, In preparation of moving dbscripts/archive to gemini I've moved the archive nginx frontend and archive-uploader to gemini. The archive is still rsynced every hour so there is now a small delay which is temporary until dbscrpts is moved which should happen this week. Greetings, Jelle

1 0

[arch-devops] Meeting 18th June 18:30 UTC - 20:30 CEST
by Jelle van der Waa 17 Jun '20

17 Jun '20

Hi all, Tomorrow we will have another devops meeting at one hour earlier then our usual time. The topics should be: * Orion replacement/migration * Gitlab/keycloak Greetings, Jelle

1 0

[arch-devops] gemini.archlinux.org set up
by Sven-Hendrik Haase 17 Jun '20

17 Jun '20

Hey y'all, I just finished up setting up gemini.archlinux.org which will serve as the new repos and archive host as orion.archlinux.org is getting way too small for this. gemini should easily serve us for the next decade or two given our current rate of growth. It now needs the archive and repo data migrated to it and then we need to do the actual switchover. For users (packagers, that is), not much should actually change except for an SSH warning. We'll inform you about new host keys once this is switched over. After the initial data sync, we need to set up the actual tools that allow for repo operation (mainly dbscripts and I believe some parts of archweb) and then switch the DNS. We'll likely disallow package updating for a short critical switch over period to make sure we are perfectly synced but this is yet to be coordinated. The machine is an Intel E3-1270 v3 with 32GiB of ECC DDR3 and 4*10TiB in BTRFS RAID6 for an effective 20TiB of usable space. I know, this seems daring given BTRFS's history but we looked at the recent stability improvements of BTRFS RAID56 and it now seems stable enough for production usage. At any rate, we have backups. Cheers, Sven

2 1

[arch-devops] python-typer now required for ansible hcould inventory script
by Jelle van der Waa 17 Jun '20

17 Jun '20

With the refactoring of the hetzner cloud inventory script python-typer is now a dependency, don't forgot to install it before running Ansible :) Greetings, Jelle

1 0