June 2020 - Arch-devops - lists.archlinux.org

[arch-devops] Add Frederik Schwan to the devops team
by Sven-Hendrik Haase 15 Jul '20

15 Jul '20

Hey all, I propose adding Frederik Schwan to the devops team. He's eager to help Arch, he's knowledgeable, and does devops professionally. He's got a good amount of experience with mail stacks and cloud stuff and he'd like to help us out. In fact, he's already helped quite a bit with testing and some accessless devops tasks but the number of tasks requiring access that we currently have far outweigh the accessless tasks and I think he'd like to become more involved. Thoughts? Cheers, Sven

6 7

[arch-devops] Let's change our primary backup to a managed storage box
by Sven-Hendrik Haase 09 Jul '20

09 Jul '20

Hey all, I want to suggest we change our primary backup box to a managed box, namely a BX60 from Hetzner [0] for 40€/m with 10TB capacity. We currently have a SB35 with 2x3TB in a RAID1. The current box (vostok) is getting full and is sitting at 20% capacity remaining. The storage box supports Borg and so it's unlikely that we'll have to change our backup scheme. Why a BX60 and not a BX50 even though the capacity would currently suffice? I'd like to also start backing up the archive. The archive will become more relevant in Arch as reproducible builds gain more popularity and I think keeping the archive alive and well is an important part of that. We currently do not back up the archive for capacity reasons but we should start doing that. BX50 with its 5TB puts us too close to our projected needs (roughly 2.3TB (current effective backup size) + 2.3TB (archive)). Advantages of a BX60: - Less stuff to manage as Hetzner will handle disk failures and updates for us - More redundancy (FAQ says it can tolerate multiple disk failures while we can currently only handle a single disk failure) - Much higher capacity (about 3.4x as much) Disadvantages: - 10€ more per month - We'll have to migrate (though probably pretty easy to do as the backup stuff is pretty nicely ansibled) - Only 10 connections allowed at once (workable for us but worth mentioning) - We currently have no operating experience with storage boxes from Hetzner What do you think? Cheers, Sven [0] https://www.hetzner.de/storage/storage-box/bx60

2 5

[arch-devops] Resignation
by fukawi2 | Arch Linux 30 Jun '20

30 Jun '20

Hi all, For various reasons, I have to formally resign from the ArchLinux DevOps team. I will maintain my role as Forum Administrator, so I would appreciate keeping my @archlinux.org email address. Non-root access to the mail server would also be great to maintain if possible, for debugging issues for forum users (eg, missing password reset emails etc). If not, I'll just harass #-devops IRC when I need to get the logs for something ;D Cheers, ~Phil

2 1

[arch-devops] Meeting 2th July 18:30 UTC - 20:30 CEST
by Jelle van der Waa 30 Jun '20

30 Jun '20

Hi all, Another week, another meeting, topics: * Storagebox for backups * Gemini migration * Removal of hefurd torrent tracker * Security audit Gitlab/Keycloak * Resignation/Addition handling of members to DevOps Greetings, Jelle

1 0

[arch-devops] Broken disk on vostok
by Sven-Hendrik Haase 29 Jun '20

29 Jun '20

Hey all, During a small routine check I noticed that we have a broken disk on vostok. In fact, according to the log, we've had it for 7 months at least which is a bit embarrassing given that this is RAID1 only and it's also our primary backup box. This really goes to show that we need better monitoring/more meaningful alerts. Anyway, I had Hetzner replace the disk and the array is now rebuilding. Cheers, Sven

4 8

[arch-devops] repos.archlinux.org has been migrated to a new server
by Jelle van der Waa 26 Jun '20

26 Jun '20

Hi all, repos.archlinux.org, svn.archlinux.org and rsync.archlinux.org are now on a new server which has plenty of diskspace for us to continue packaging for a while (16T free). Some important notes: The ssh hostkeys changed for repos.archlinux.org which means if you use archco/communityco this will fail without a useful error. So take care of your .ssh/known_hosts. The ssh host keys for gemini/repos.archlinux.org are: # gemini.archlinux.org 1024 SHA256:F1Corf6i2U72yub+CIzzGHLOMVKVnjALh1YHM8gBjxE root(a)gemini.archlinux.org (DSA) 256 SHA256:If51DkTftUpDAFz65totgDfTd/ddu/2w/RBZIHtY74U root(a)gemini.archlinux.org (ECDSA) 256 SHA256:wUrJYf9+zOpIEUQ3ndgarK0PjzPICa1frmu7mpL4e14 root(a)gemini.archlinux.org (ED25519) 3072 SHA256:Rltnuln3bjsHJwVbys/LnYCj7hO6srPoa15JP8QhmlQ root(a)gemini.archlinux.org (RSA) https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/docs/ss… https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/docs/ss… If anything unexpected happens please either mail arch-devops(a)lists.archlinux.org or #archlinux-devops on irc. Greetings, Jelle

2 2

[arch-devops] Migrating hefur from orion
by Jelle van der Waa 25 Jun '20

25 Jun '20

As we now have moved the repos to gemini we have not yet migrated hefurd which is the torrent webseed service. As this is a random c++ application it might be nice to start moving it to a VPS? We can use the kitchensink to rsync the iso to a new vps and run /usr/bin/hefurd -disable-peers-page -torrent-dir /srv/ftp/iso As we used todo, opinions? Greetings, Jelle van der Waa

1 0

[arch-devops] Invalid SSL CN on tracker.archlinux.org
by Egor Kovetskiy 24 Jun '20

24 Jun '20

Hello, tracker.archlinux.org has an invalid certificate with CN = archive.archlinux.org $ openssl s_client -showcerts -connect tracker.archlinux.org:443 <<< "" 2>&1 | grep subject=CN subject=CN = archive.archlinux.org The link was found on https://github.com/archlinux/infrastructure

2 1

[arch-devops] Thursday 25th June 14:00 UTC repos.archlinux.org migration
by Jelle van der Waa 23 Jun '20

23 Jun '20

Hi al, On Thursday 25th June at 14:00 UTC work will begin on migrating the following services to a new machine: * repos.archlinux.org * svn.archlinux.org * Arch mirror infrastructure This will impact all packaging and svn activities as those services are not available during the migration period. The services will be migrated to a new server as diskspace is running low on the current server, the migration is expected to take at least four hours. When the migration is successfully completed an email will be send that the services are available and packaging can resume. TLDR: No packaging can be done on the 25th of June starting at 14:00 UTC till the migration is done. Greetings, Jelle

1 0

[arch-devops] archive.archlinux.org migrated to gemini.archlinux.org
by Jelle van der Waa 21 Jun '20

21 Jun '20

Hi all, In preparation of moving dbscripts/archive to gemini I've moved the archive nginx frontend and archive-uploader to gemini. The archive is still rsynced every hour so there is now a small delay which is temporary until dbscrpts is moved which should happen this week. Greetings, Jelle

1 0