June 2020 - Arch-devops - lists.archlinux.org

[arch-devops] Meeting 18th June 18:30 UTC - 20:30 CEST
by Jelle van der Waa 17 Jun '20

17 Jun '20

Hi all, Tomorrow we will have another devops meeting at one hour earlier then our usual time. The topics should be: * Orion replacement/migration * Gitlab/keycloak Greetings, Jelle

1 0

[arch-devops] gemini.archlinux.org set up
by Sven-Hendrik Haase 17 Jun '20

17 Jun '20

Hey y'all, I just finished up setting up gemini.archlinux.org which will serve as the new repos and archive host as orion.archlinux.org is getting way too small for this. gemini should easily serve us for the next decade or two given our current rate of growth. It now needs the archive and repo data migrated to it and then we need to do the actual switchover. For users (packagers, that is), not much should actually change except for an SSH warning. We'll inform you about new host keys once this is switched over. After the initial data sync, we need to set up the actual tools that allow for repo operation (mainly dbscripts and I believe some parts of archweb) and then switch the DNS. We'll likely disallow package updating for a short critical switch over period to make sure we are perfectly synced but this is yet to be coordinated. The machine is an Intel E3-1270 v3 with 32GiB of ECC DDR3 and 4*10TiB in BTRFS RAID6 for an effective 20TiB of usable space. I know, this seems daring given BTRFS's history but we looked at the recent stability improvements of BTRFS RAID56 and it now seems stable enough for production usage. At any rate, we have backups. Cheers, Sven

2 1

[arch-devops] python-typer now required for ansible hcould inventory script
by Jelle van der Waa 17 Jun '20

17 Jun '20

With the refactoring of the hetzner cloud inventory script python-typer is now a dependency, don't forgot to install it before running Ansible :) Greetings, Jelle

1 0

[arch-devops] secure-runner1.archlinux.org set up
by Sven-Hendrik Haase 17 Jun '20

17 Jun '20

Hey all, Just writing real quick to inform you that I just set up secure-runner1.archlinux.org which is a new hardware box that'll serve as a GitLab runner that we can trust. The idea is to never allow it to run unreviewed or untrusted code by limiting it to protected branches of specific projects (and only those projects, not their forks). Effectively, that'll be the master branch of selected Arch projects which need to produce trustable artifacts (like archiso for automatic ISO creation or VM images, for instance). The current idea is not to use it for any automatic package building. This is currently an experiment and will still need to be security audited (and we're not using it for producing any real artifacts until we feel comfortable with it). It might turn out we need one such secure runner per project, who knows. We'll control this runner rather tightly for obvious reasons and projects need to be hand-selected one-by-one. In case anyone's wondering, the box is a Ryzen 5 3600 with 64 GiB ECC DDR4 and 512 GiB BTRFS RAID1. Cheers, Sven

1 0

[arch-devops] Unexpected grub failure on orion
by Jelle van der Waa 15 Jun '20

15 Jun '20

Hello all, After updates on orion and a reboot we had a boot failure with the following error message. It is unknown what caused this, but grub was also updated: [2020-05-18T19:18:52+0000] [ALPM] upgraded grub (2:2.04-5 -> 2:2.04-7) The grub error: symbol `grub_file_filters` not found When trying to grub-install with the latest grub package it fails and results into the following bug FS#63656 We downgraded grub to grub-2:2.02-8-x86_64.pkg.tar.xz from archive.org, to get a package listing you can use the handy url. [2] This raised the question of grazzolini whether we should continue using a partionless btrfs setup and as we are about to setup a new orion.archlinux.org this might be relevant :) I wonder if the new box also has UEFI already and if we should switch away from grub? (although I think all our vps'es use grub) [1] https://bugs.archlinux.org/task/63656 [2] https://archive.org/download/archlinux_pkg_grub Greetings, Jelle

2 1

[arch-devops] intel-ucode update might cause issues
by Jelle van der Waa 11 Jun '20

11 Jun '20

Hi all, As we had an boot failure on orion, Foxboron pointed out it might be the recent intel-ucode update which caused issues on Ubuntu. [1] I've downgraded it on orion, but grub was the curlpit. I'll update a vps tommorow and see how that goes, but for our dedicated boxes be aware that the update might cause issues. [1] https://usn.ubuntu.com/4385-2/

1 0

[arch-devops] Is repos.archlinux.org down?
by Konstantin Gizdov 11 Jun '20

11 Jun '20

Hi, I've been having issues trying to login to repos.archlinux.org. First, it went offline suddenly, now it reports a different SSH key. I tried logging in and it no longer recognizes my key. I'm only asking cos I was in the middle of moving a package from community-testing to community that introduces breaking changes and now I'm not sure how to finish the process. -- Regards, Konstantin

1 0

[arch-devops] Is repos.archlinux.org down?
by Konstantin Gizdov 11 Jun '20

11 Jun '20

Hi, I've been having issues trying to login to repos.archlinux.org. First, it went offline suddenly, now it reports a different SSH key. I tried logging in and it no longer recognizes my key. I'm only asking cos I was in the middle of moving a package from community-testing to community that introduces breaking changes and now I'm not sure how to finish the process. -- Regards, Konstantin

1 0

[arch-devops] Diskspace getting low on orion or let's get a bigger archive server
by Jelle van der Waa 07 Jun '20

07 Jun '20

Hi All, We are running awfully low on diskspace on orion and need to start migrating the archive away ASAP as we can't keep hosting it on orion. The biggest offender is the archive which seems to be requiring around ~ 1TB per year, this might be due to the nature of how we archive as heftig pointed out we could use btrfs snapshots however no one has stepped up and I don't expect anyone to. I would propose to get us a new server, an AX51 which has 2 x 8TB disks which allows us to grow the archive. Currently we pay 35 / month for orion which has 2 x 3 TB disks. The hard question is if we tackle multiple problems at once and make this server only for the archive which might be tricky as it's now part of dbscripts to archive packages with "db-archive" which means we have to migrate our repos, svn server and sourceballs as that all sadly expects to be present on the machine. This would leave orion left with mail handling and hefur (for the arch iso torrent webseeds) which should be moved to a separate vps both shouldn't require a too big of a vps. https://www.hetzner.com/dedicated-rootserver/ex52-nvme Please respond ASAP, as we don't have too much time and Hetzner only deploys from Monday -Friday. Greetings, Jelle van der Waa

4 9

[arch-devops] Meeting 3th June 19:30 UTC - 21:30 CEST
by Jelle van der Waa 03 Jun '20

03 Jun '20

Hi all, Tomorrow we will have another devops meeting at the usual time, there are no topics in the agenda but I expect us to discuss Gitlab/Keyclaok progress. Greetings, Jelle

2 1