- Arch-devops - lists.archlinux.org

[arch-devops] Unexpected grub failure on orion
by Jelle van der Waa 15 Jun '20

15 Jun '20

Hello all, After updates on orion and a reboot we had a boot failure with the following error message. It is unknown what caused this, but grub was also updated: [2020-05-18T19:18:52+0000] [ALPM] upgraded grub (2:2.04-5 -> 2:2.04-7) The grub error: symbol `grub_file_filters` not found When trying to grub-install with the latest grub package it fails and results into the following bug FS#63656 We downgraded grub to grub-2:2.02-8-x86_64.pkg.tar.xz from archive.org, to get a package listing you can use the handy url. [2] This raised the question of grazzolini whether we should continue using a partionless btrfs setup and as we are about to setup a new orion.archlinux.org this might be relevant :) I wonder if the new box also has UEFI already and if we should switch away from grub? (although I think all our vps'es use grub) [1] https://bugs.archlinux.org/task/63656 [2] https://archive.org/download/archlinux_pkg_grub Greetings, Jelle

2 1

[arch-devops] intel-ucode update might cause issues
by Jelle van der Waa 11 Jun '20

11 Jun '20

Hi all, As we had an boot failure on orion, Foxboron pointed out it might be the recent intel-ucode update which caused issues on Ubuntu. [1] I've downgraded it on orion, but grub was the curlpit. I'll update a vps tommorow and see how that goes, but for our dedicated boxes be aware that the update might cause issues. [1] https://usn.ubuntu.com/4385-2/

1 0

[arch-devops] Is repos.archlinux.org down?
by Konstantin Gizdov 11 Jun '20

11 Jun '20

Hi, I've been having issues trying to login to repos.archlinux.org. First, it went offline suddenly, now it reports a different SSH key. I tried logging in and it no longer recognizes my key. I'm only asking cos I was in the middle of moving a package from community-testing to community that introduces breaking changes and now I'm not sure how to finish the process. -- Regards, Konstantin

1 0

[arch-devops] Is repos.archlinux.org down?
by Konstantin Gizdov 11 Jun '20

11 Jun '20

Hi, I've been having issues trying to login to repos.archlinux.org. First, it went offline suddenly, now it reports a different SSH key. I tried logging in and it no longer recognizes my key. I'm only asking cos I was in the middle of moving a package from community-testing to community that introduces breaking changes and now I'm not sure how to finish the process. -- Regards, Konstantin

1 0

[arch-devops] Diskspace getting low on orion or let's get a bigger archive server
by Jelle van der Waa 07 Jun '20

07 Jun '20

Hi All, We are running awfully low on diskspace on orion and need to start migrating the archive away ASAP as we can't keep hosting it on orion. The biggest offender is the archive which seems to be requiring around ~ 1TB per year, this might be due to the nature of how we archive as heftig pointed out we could use btrfs snapshots however no one has stepped up and I don't expect anyone to. I would propose to get us a new server, an AX51 which has 2 x 8TB disks which allows us to grow the archive. Currently we pay 35 / month for orion which has 2 x 3 TB disks. The hard question is if we tackle multiple problems at once and make this server only for the archive which might be tricky as it's now part of dbscripts to archive packages with "db-archive" which means we have to migrate our repos, svn server and sourceballs as that all sadly expects to be present on the machine. This would leave orion left with mail handling and hefur (for the arch iso torrent webseeds) which should be moved to a separate vps both shouldn't require a too big of a vps. https://www.hetzner.com/dedicated-rootserver/ex52-nvme Please respond ASAP, as we don't have too much time and Hetzner only deploys from Monday -Friday. Greetings, Jelle van der Waa

4 9

[arch-devops] Meeting 3th June 19:30 UTC - 21:30 CEST
by Jelle van der Waa 03 Jun '20

03 Jun '20

Hi all, Tomorrow we will have another devops meeting at the usual time, there are no topics in the agenda but I expect us to discuss Gitlab/Keyclaok progress. Greetings, Jelle

2 1

[arch-devops] AUR and SSO login
by Frédéric Mangano-Tarumi 20 May '20

20 May '20

Hello everyone, As a contributor to the AUR website, I got wind of the SSO migration project[1] from Lukas. The SSO migration was recently announced on arch-dev-public[2] and notably contains the following statement: > In order to still allow users to keep their old contributions in cases > where they can prove their identity via email, we'll build a new small > web application that allows them to connect their new Keycloak > identity to their other identities. The wiki article[1] mentions we need to verify the email addresses registered on AUR in order to merge them with something. I feel a bit concerned by these statements because I don’t think email addresses should be used to merge accounts, or that we should automatically merge accounts at all. We’ve always used username/password pairs as a primary authentication method, and suddenly altering the authentication method without explicit user consent doesn’t sound respectful. Some users may have used the same email address at different places but may want to keep their accounts separate. Other users may have used different email addresses but may want their accounts linked. Other may have deleted their email address but still regularly use a service without ever noticing they forgot to update their account information. I’d like to suggest a migration flow that should cover everyone’s case without making risky decisions, nor requiring prior email verification: First, we’d introduce an SSO button to the login page, next to or in place of the username/password form. When the user picks SSO, they’d be redirected to the SSO login page, where they may create an SSO account or input their existing SSO credentials. On successful login, they’ll be redirected back to the original website. If the website detects it’s the first time the SSO account has logged in, it would display both a registration form targeted at new users, and a legacy credentials form targeted at previously existing users. If the user fills in their legacy credentials, their account will be linked to the SSO. Note that email address verification would take place only in the SSO account creation, once and for all services. Also note that the user doesn’t need to visit an external website for linking accounts, since the first-authentication flow guides them throughout the migration process. This is most certainly not the first time the account linking topic is brought up, but these are my two cents as a non-infrastructure-y developer. Whether we take this road or not, I plan to contribute to the SSO integration at least as far as AUR is concerned. [1] https://wiki.archlinux.org/index.php/DeveloperWiki:SSOMigration [2] https://lists.archlinux.org/pipermail/arch-dev-public/2020-May/029971.html Regards, -- fmang

1 0

[arch-devops] Meeting 21th May 19:30 UTC - 21:30 CEST
by Jelle van der Waa 20 May '20

20 May '20

Hi all, Another important meeting is scheduled for tomorrow, the notes of the previous meeting can be found here. [1] Topics for tomorrow # Discuss previous meeting topic progress https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06 # Gitlab project creation We already have some projects on the Arch Linux namespace and some questions where raised about this: * Do we want to add an infrastructure namespace or other specific namespaces? * Do we already onboard projects, what is the procedure and what permissions do we set? * Do we always set up Github sync? * What to do with runners? # Onboarding people to the Arch Linux group Do we already onboard people, it is a manual task and error prone. Another problem is that we have onboarded users on keycloak which are now ex-developers or inactive, do we remove these accounts? How do we onboard new staff on keycloak? # Prometheus monitoring We have discussed changing our monitoring to Prometheus in the interest of further automation and having configuration in code. [2] # Discuss priority's We should discuss what our priorities are and blocking issues for Gitlab and ideally organize our Gitlab board to reflect this. [1] https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06 [2] https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/31

1 0

[arch-devops] infrastructure.git now on gitlab.archlinux.org
by Jelle van der Waa 18 May '20

18 May '20

As we are moving to gitlab.archlinux.org for our projects, the infrastructure repository has been moved to gitlab. The git.archlinux.org repository is disabled. Please switch your local git repository remote: git remote set-url origin ssh://git@gitlab.archlinux.org:222/archlinux/infrastructure.git The repository enforces that all commits should be signed and automatically pushes changes to our Github repository. For non-trivial changes we strongly recommend creating a merge request instead of pushing directly to master like you're used to. To get onto gitlab, browse to gitlab.archlinux.org, sign in with "Arch Linux SSO", follow the "forgot password" flow and use your archweb email address. Two factor authentication is enforced for all Devops users. Greetings, Jelle

4 3

[arch-devops] Meeting Thursday 7th May 19:30 UTC - 21:30 CEST
by Jelle van der Waa 06 May '20

06 May '20

Hi all, For this meeting the topics to discuss are: * Orion - I/O issues due to the archiving, disk space currently low and runs a ton of services which all generate I/O. The biggest storage usage offenders on orion are: - /srv/archive - 2.2 TB - /srv/ftp - 142 GiB - /srv/ftp/sources 64 GiB - /srv/svntogit 2.1 GiB The archive for packages is created by dbscripts, the real issue is the non-maintained "archivetools" for creating the daily repositories and iso releases. Heftig thought about reworking how archive should use btrfs subvolumes instead of rsync hardlinks, the archive is giant and rsync has to create an enormous amount of hardlinks, snapshot the previous day, then rsync over the kitchen sink, same thing just without link-dest or just keep the "kitchen sink" itself in a subvolume. Another idea is to later switch to a managed storagebox for the archive. * Periodically running btrfs scrub (heftig) * Clean up, re-order kanboard https://kanboard.archlinux.org/board/1 as requested by heftig * Fosshost.org wants to sponsor a box 8 vCPU, 8 GB ram. https://fosshost.org/platform/

1 0