- Arch-devops - lists.archlinux.org

[arch-devops] Gitlab Meeting Wednesday 5th May 18:30 UTC - 20:30 CEST
by Jelle van der Waa 05 May '20

05 May '20

Hi All, I propose we have a meeting about possibly using Gitlab, it's currently setup as a PoC on gitlab.archlinux.org There are however still some things we need to nail down and discuss: - GitLab <-> GitHub sync - GitLab security considerations - GitLab general user collaboration - Gitlab groups/roles? (on Github we have groups of "maintainers") - Gitlab backups * what to backup? - git.archlinux.org migrations? * can we abolish git.archlinux.org? (reduce maintenance) * do we archive old projects? * do we redirect old projects to a the new location? - What Gitlab runners to use and on what hardware? - what goes onto GitLab? - what goes onto Keycloak? - next projects to migrate to Keycloak? I also propose we should make it clear that the PoC Gitlab instance is not to be used for certain use cases as:- - building packages/signing etc. - onboarding external users for now Greetings, Jelle

3 4

[arch-devops] cloud-images
by Christian Rebischke 02 May '20

02 May '20

Dear devops team, Recently I have pushed cloud-init back to [community][1], one Arch Linux user did some improvements on the cloud-init codebase as well and they are going to officially support arch linux with version 19.3[2]. Therefore I have added support for cloud-images to the arch-boxes project[3]. These images are qcow2 images and are expected to be installed in cloud-environments like Openstack. The only thing what is missing is a way to distribute the images and release them. My current plan has been that I build and sign them locally and then upload them to a web directory (This is the most secure way as possible right now). For this I need answers for a few questions: 1. Can I have a web-directory, where only I have access to? 2. Which key should I use to sign them? A new cloud-image signing key or my personal key? (I think latter should be enough). 3. I expect to build them monthly like the ISOs, how many images do you want to keep? My current assumption is that 1 year of image backup of cloud-images (qcow2) only would cost us around 30-50GB. 4. Do we want to mirror the images? Best greetings Chris [1] https://www.archlinux.org/packages/community/any/cloud-init/ [2] https://git.launchpad.net/cloud-init/commit/?id=5d5a32e039782ce3e1c0843082f… [3] https://github.com/archlinux/arch-boxes/commit/9d9f04f9618367e2af8d92bef2d5…

2 3

[arch-devops] Refreshing borg backup keys
by Jelle van der Waa 01 May '20

01 May '20

Hi DevOps Team members, This evening I realized we added the offsite backups to rsync.net with borg but never updated the fetch-borg-keys playbook to also download the offsite backup keys (as they are different from the keys for vostok). I've updated the playbook to fetch the offiste borg backup key as $hostname-offsite.gpg. So please re-run them locally and keep the encrypted borg keys somewhere secure For further instructions for fetching the keys check the README. Greetings, Jelle

1 0

[arch-devops] Refetch fetch-borg-keys
by Jelle van der Waa 01 May '20

01 May '20

Hi DevOps Team members, This evening I realized we added the offsite backups to rsync.net with borg but never updated the fetch-borg-keys playbook to also download the offsite backup keys (as they are different from the keys for vostok). I've updated the playbook to fetch the offiste borg backup key as $hostname-offsite.gpg. So please re-run them locally and keep the encrypted borg keys somewhere secure :-) For further instructions for fetching the keys check the README. Greetings, Jelle

1 0

[arch-devops] Arch Devops meeting tommorow 19:30 UTC
by Jelle van der Waa 23 Apr '20

23 Apr '20

Hi All, Tomorrow we have another DevOps meeting at 19:30 UTC or in your own timezone visit: https://time.is/compare/1930_23_Apr_2020_in_UTC * Keycloak * PIA mirrors * Mailman * AUR VPS * Reproducible builds infra -- Jelle van der Waa

2 2

[arch-devops] Hetzner billing handover to SPI
by Levente Polyak 13 Apr '20

13 Apr '20

Hey folks, we have been discussing this in a small round and want to transition the hetzner billing directly to SPI. This requires them to access the account once to insert the billing data in the accounting page. All this will be done in a secure manner, however even through we have two-factor in place we may want to change the password afterwards. cheers, Levente

4 5

[arch-devops] an Arch Linux status page
by Christian Rebischke 12 Apr '20

12 Apr '20

Dear Devops Team, I had a discussion with grazzolini around a status Arch Linux website. You may know, that this is on my wish list for several months now. But first let me reiterate the problem I want to solve with this: 1. this is not a replacement for your existing monitoring. 2. It should be available on status.archlinux.org for serving as simple status website, that people can inform theirself if a maintenance is running or if the AUR is down or whatever. 3. bonus points if we can access an API and use it as another monitoring channel (if our main monitoring goes down, we will at least get notified). 4. The status website should be HA, so even if the whole arch infrastructure explodes, people can still access this website and inform theirself through a safe channel. --------------- So I've looked around a bit and tested different providers. After digging through the web I landed on https://uptimerobot.org. I think uptimerobot is a nice option for us, because of several reasons: 1. they have a free plan. 2. they support multiple scans like HTTP(S), Keyword, Ping, Port 3. They provide a free status website (this is important, because status.io or statuspage.io don't do this). 4. They are HA (they are behind cloudflare) ------------------- Some bad facts about uptimerobot: 1. Maintenance mode is only supported in pro-mode (7$ per month), but we could hack around this via setting a different title on the page (but then the Arch Linux logo will disappear). 2. Heartbeat tests are only supported in pro-mode 3. They allow only one login, so we would need to setup a generic login (bonus point: they support 2FA at least). If you want to see how such a status website would look like, have a look here: https://stats.uptimerobot.com/vmM5ruWEAB (with Arch Linux Logo) This is one with just a text title: https://status.shibumi.dev/ All best Chris / shibumi

2 1

[arch-devops] Arch Devops meeting tommorow 20:30 UTC
by Jelle van der Waa 11 Mar '20

11 Mar '20

Hi All, Tommorow we have another DevOps meeting at 20:30 UTC or in your own timezone visit: https://time.is/compare/2030_12_Mar_2020_in_UTC The topics/updates to discuss are: * mailman 3 * aur-dev/aur * offsite backups -- Jelle van der Waa

1 1

[arch-devops] Offsite backups now in effect
by Sven-Hendrik Haase 11 Mar '20

11 Mar '20

Hey everyone, During my vacation, I found some time to tackle the offsite backups thing. So let me break it down: - We now have an offsite backup location provided by rsync.net and based in Switzerland. - Access the account data from the vault. - You all should have SSH access already via SSH keys (we have full authorized_keys file access) - All current machines were configured to use this offsite backup. - The offsite backup step runs as a separate backup right after the first one. - The offsite backups were tested and it appears to be working fine on all machines. - Payment-wise everything is managed by SPI directly (no reimbursements required) and I'm currently the rsync.net liaison and point of contact. This should be changed to a generic private mail that we can point to in order to mitigate the bus factor. - We should consider how to do 2FA with rsync.net in a DevOps meeting. - I had to do quite some Ansible gymnastics in order to configure the rsync.net machine as it doesn't have Python and is, in general, a pretty restricted environment so we have a new role next to borg-server called rsync_net. - During all of this I cleaned up a few things surrounding borg and I mostly/fully made it so that we can fairly easily add more different offsite locations. - Giancarlo reviewed my changes. Thanks! I might have forgotten somethings but this is pretty much the most important bit. Enjoy, Sven

1 0

[arch-devops] Arch Devops meeting today 20:30 UTC
by Jelle van der Waa 27 Feb '20

27 Feb '20

Hi All, Today we have another DevOps meeting at 20:30 UTC or in your own timezone visit: https://time.is/compare/2030_27_Feb_2020_in_UTC Since I didn't get any feedback on topics yet, I propose to hold a debug session on the terraform situation and discuss mailman 3. -- Jelle van der Waa

1 0