- Arch-devops - lists.archlinux.org

[arch-devops] Is repos.archlinux.org down?
by Konstantin Gizdov 11 Jun '20

11 Jun '20

Hi, I've been having issues trying to login to repos.archlinux.org. First, it went offline suddenly, now it reports a different SSH key. I tried logging in and it no longer recognizes my key. I'm only asking cos I was in the middle of moving a package from community-testing to community that introduces breaking changes and now I'm not sure how to finish the process. -- Regards, Konstantin

1 0

[arch-devops] Diskspace getting low on orion or let's get a bigger archive server
by Jelle van der Waa 07 Jun '20

07 Jun '20

Hi All, We are running awfully low on diskspace on orion and need to start migrating the archive away ASAP as we can't keep hosting it on orion. The biggest offender is the archive which seems to be requiring around ~ 1TB per year, this might be due to the nature of how we archive as heftig pointed out we could use btrfs snapshots however no one has stepped up and I don't expect anyone to. I would propose to get us a new server, an AX51 which has 2 x 8TB disks which allows us to grow the archive. Currently we pay 35 / month for orion which has 2 x 3 TB disks. The hard question is if we tackle multiple problems at once and make this server only for the archive which might be tricky as it's now part of dbscripts to archive packages with "db-archive" which means we have to migrate our repos, svn server and sourceballs as that all sadly expects to be present on the machine. This would leave orion left with mail handling and hefur (for the arch iso torrent webseeds) which should be moved to a separate vps both shouldn't require a too big of a vps. https://www.hetzner.com/dedicated-rootserver/ex52-nvme Please respond ASAP, as we don't have too much time and Hetzner only deploys from Monday -Friday. Greetings, Jelle van der Waa

4 9

[arch-devops] Meeting 3th June 19:30 UTC - 21:30 CEST
by Jelle van der Waa 03 Jun '20

03 Jun '20

Hi all, Tomorrow we will have another devops meeting at the usual time, there are no topics in the agenda but I expect us to discuss Gitlab/Keyclaok progress. Greetings, Jelle

2 1

[arch-devops] AUR and SSO login
by Frédéric Mangano-Tarumi 20 May '20

20 May '20

Hello everyone, As a contributor to the AUR website, I got wind of the SSO migration project[1] from Lukas. The SSO migration was recently announced on arch-dev-public[2] and notably contains the following statement: > In order to still allow users to keep their old contributions in cases > where they can prove their identity via email, we'll build a new small > web application that allows them to connect their new Keycloak > identity to their other identities. The wiki article[1] mentions we need to verify the email addresses registered on AUR in order to merge them with something. I feel a bit concerned by these statements because I don’t think email addresses should be used to merge accounts, or that we should automatically merge accounts at all. We’ve always used username/password pairs as a primary authentication method, and suddenly altering the authentication method without explicit user consent doesn’t sound respectful. Some users may have used the same email address at different places but may want to keep their accounts separate. Other users may have used different email addresses but may want their accounts linked. Other may have deleted their email address but still regularly use a service without ever noticing they forgot to update their account information. I’d like to suggest a migration flow that should cover everyone’s case without making risky decisions, nor requiring prior email verification: First, we’d introduce an SSO button to the login page, next to or in place of the username/password form. When the user picks SSO, they’d be redirected to the SSO login page, where they may create an SSO account or input their existing SSO credentials. On successful login, they’ll be redirected back to the original website. If the website detects it’s the first time the SSO account has logged in, it would display both a registration form targeted at new users, and a legacy credentials form targeted at previously existing users. If the user fills in their legacy credentials, their account will be linked to the SSO. Note that email address verification would take place only in the SSO account creation, once and for all services. Also note that the user doesn’t need to visit an external website for linking accounts, since the first-authentication flow guides them throughout the migration process. This is most certainly not the first time the account linking topic is brought up, but these are my two cents as a non-infrastructure-y developer. Whether we take this road or not, I plan to contribute to the SSO integration at least as far as AUR is concerned. [1] https://wiki.archlinux.org/index.php/DeveloperWiki:SSOMigration [2] https://lists.archlinux.org/pipermail/arch-dev-public/2020-May/029971.html Regards, -- fmang

1 0

[arch-devops] Meeting 21th May 19:30 UTC - 21:30 CEST
by Jelle van der Waa 20 May '20

20 May '20

Hi all, Another important meeting is scheduled for tomorrow, the notes of the previous meeting can be found here. [1] Topics for tomorrow # Discuss previous meeting topic progress https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06 # Gitlab project creation We already have some projects on the Arch Linux namespace and some questions where raised about this: * Do we want to add an infrastructure namespace or other specific namespaces? * Do we already onboard projects, what is the procedure and what permissions do we set? * Do we always set up Github sync? * What to do with runners? # Onboarding people to the Arch Linux group Do we already onboard people, it is a manual task and error prone. Another problem is that we have onboarded users on keycloak which are now ex-developers or inactive, do we remove these accounts? How do we onboard new staff on keycloak? # Prometheus monitoring We have discussed changing our monitoring to Prometheus in the interest of further automation and having configuration in code. [2] # Discuss priority's We should discuss what our priorities are and blocking issues for Gitlab and ideally organize our Gitlab board to reflect this. [1] https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06 [2] https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/31

1 0

[arch-devops] infrastructure.git now on gitlab.archlinux.org
by Jelle van der Waa 18 May '20

18 May '20

As we are moving to gitlab.archlinux.org for our projects, the infrastructure repository has been moved to gitlab. The git.archlinux.org repository is disabled. Please switch your local git repository remote: git remote set-url origin ssh://git@gitlab.archlinux.org:222/archlinux/infrastructure.git The repository enforces that all commits should be signed and automatically pushes changes to our Github repository. For non-trivial changes we strongly recommend creating a merge request instead of pushing directly to master like you're used to. To get onto gitlab, browse to gitlab.archlinux.org, sign in with "Arch Linux SSO", follow the "forgot password" flow and use your archweb email address. Two factor authentication is enforced for all Devops users. Greetings, Jelle

4 3

[arch-devops] Meeting Thursday 7th May 19:30 UTC - 21:30 CEST
by Jelle van der Waa 06 May '20

06 May '20

Hi all, For this meeting the topics to discuss are: * Orion - I/O issues due to the archiving, disk space currently low and runs a ton of services which all generate I/O. The biggest storage usage offenders on orion are: - /srv/archive - 2.2 TB - /srv/ftp - 142 GiB - /srv/ftp/sources 64 GiB - /srv/svntogit 2.1 GiB The archive for packages is created by dbscripts, the real issue is the non-maintained "archivetools" for creating the daily repositories and iso releases. Heftig thought about reworking how archive should use btrfs subvolumes instead of rsync hardlinks, the archive is giant and rsync has to create an enormous amount of hardlinks, snapshot the previous day, then rsync over the kitchen sink, same thing just without link-dest or just keep the "kitchen sink" itself in a subvolume. Another idea is to later switch to a managed storagebox for the archive. * Periodically running btrfs scrub (heftig) * Clean up, re-order kanboard https://kanboard.archlinux.org/board/1 as requested by heftig * Fosshost.org wants to sponsor a box 8 vCPU, 8 GB ram. https://fosshost.org/platform/

1 0

[arch-devops] Gitlab Meeting Wednesday 5th May 18:30 UTC - 20:30 CEST
by Jelle van der Waa 05 May '20

05 May '20

Hi All, I propose we have a meeting about possibly using Gitlab, it's currently setup as a PoC on gitlab.archlinux.org There are however still some things we need to nail down and discuss: - GitLab <-> GitHub sync - GitLab security considerations - GitLab general user collaboration - Gitlab groups/roles? (on Github we have groups of "maintainers") - Gitlab backups * what to backup? - git.archlinux.org migrations? * can we abolish git.archlinux.org? (reduce maintenance) * do we archive old projects? * do we redirect old projects to a the new location? - What Gitlab runners to use and on what hardware? - what goes onto GitLab? - what goes onto Keycloak? - next projects to migrate to Keycloak? I also propose we should make it clear that the PoC Gitlab instance is not to be used for certain use cases as:- - building packages/signing etc. - onboarding external users for now Greetings, Jelle

3 4

[arch-devops] cloud-images
by Christian Rebischke 02 May '20

02 May '20

Dear devops team, Recently I have pushed cloud-init back to [community][1], one Arch Linux user did some improvements on the cloud-init codebase as well and they are going to officially support arch linux with version 19.3[2]. Therefore I have added support for cloud-images to the arch-boxes project[3]. These images are qcow2 images and are expected to be installed in cloud-environments like Openstack. The only thing what is missing is a way to distribute the images and release them. My current plan has been that I build and sign them locally and then upload them to a web directory (This is the most secure way as possible right now). For this I need answers for a few questions: 1. Can I have a web-directory, where only I have access to? 2. Which key should I use to sign them? A new cloud-image signing key or my personal key? (I think latter should be enough). 3. I expect to build them monthly like the ISOs, how many images do you want to keep? My current assumption is that 1 year of image backup of cloud-images (qcow2) only would cost us around 30-50GB. 4. Do we want to mirror the images? Best greetings Chris [1] https://www.archlinux.org/packages/community/any/cloud-init/ [2] https://git.launchpad.net/cloud-init/commit/?id=5d5a32e039782ce3e1c0843082f… [3] https://github.com/archlinux/arch-boxes/commit/9d9f04f9618367e2af8d92bef2d5…

2 3

[arch-devops] Refreshing borg backup keys
by Jelle van der Waa 01 May '20

01 May '20

Hi DevOps Team members, This evening I realized we added the offsite backups to rsync.net with borg but never updated the fetch-borg-keys playbook to also download the offsite backup keys (as they are different from the keys for vostok). I've updated the playbook to fetch the offiste borg backup key as $hostname-offsite.gpg. So please re-run them locally and keep the encrypted borg keys somewhere secure For further instructions for fetching the keys check the README. Greetings, Jelle

1 0