- Arch-devops - lists.archlinux.org

[arch-devops] Offsite backups now in effect
by Sven-Hendrik Haase 11 Mar '20

11 Mar '20

Hey everyone, During my vacation, I found some time to tackle the offsite backups thing. So let me break it down: - We now have an offsite backup location provided by rsync.net and based in Switzerland. - Access the account data from the vault. - You all should have SSH access already via SSH keys (we have full authorized_keys file access) - All current machines were configured to use this offsite backup. - The offsite backup step runs as a separate backup right after the first one. - The offsite backups were tested and it appears to be working fine on all machines. - Payment-wise everything is managed by SPI directly (no reimbursements required) and I'm currently the rsync.net liaison and point of contact. This should be changed to a generic private mail that we can point to in order to mitigate the bus factor. - We should consider how to do 2FA with rsync.net in a DevOps meeting. - I had to do quite some Ansible gymnastics in order to configure the rsync.net machine as it doesn't have Python and is, in general, a pretty restricted environment so we have a new role next to borg-server called rsync_net. - During all of this I cleaned up a few things surrounding borg and I mostly/fully made it so that we can fairly easily add more different offsite locations. - Giancarlo reviewed my changes. Thanks! I might have forgotten somethings but this is pretty much the most important bit. Enjoy, Sven

1 0

[arch-devops] Arch Devops meeting today 20:30 UTC
by Jelle van der Waa 27 Feb '20

27 Feb '20

Hi All, Today we have another DevOps meeting at 20:30 UTC or in your own timezone visit: https://time.is/compare/2030_27_Feb_2020_in_UTC Since I didn't get any feedback on topics yet, I propose to hold a debug session on the terraform situation and discuss mailman 3. -- Jelle van der Waa

1 0

[arch-devops] Devops meeting summary - 13-02-2020
by Jelle van der Waa 13 Feb '20

13 Feb '20

The notes can be found here https://wiki.archlinux.org/index.php?title=DeveloperWiki:DevopsMeetings/202… Greetings, Jelle

1 0

[arch-devops] Meeting tommorow 20:30 UTC
by Jelle van der Waa 12 Feb '20

12 Feb '20

Hi All, Tommorow we have another DevOps meeting at 20:30 UTC or in your own timezone visit: https://time.is/compare/2130_13_Feb_2020_in_UTC The topics for the meeting are: * AUR progress * Keycloak progress * Rsync.net offsite backups * Ansible testing -- Jelle van der Waa

1 0

[arch-devops] Devops meeting summary - 16-01-2020
by Jelle van der Waa 16 Jan '20

16 Jan '20

Hi, The meeting notes can be found here: https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-01-16 Greetings, Jelle van der Waa

1 0

[arch-devops] Meeting tommorow 20:30 UTC
by Jelle van der Waa 15 Jan '20

15 Jan '20

Hi all, Tommorow we have another DevOps meeting at 20:30 UTC or in your own timezone visit: https://time.is/compare/2130_16_Jan_2020_in_UTC The topics for the meeting are: * How to give anatolik a bugs.archlinux.org database dump without leaking personal information? * Backups of homedir * Planet.archlinux.org * AUR ansible-ification If you want to propose an additional topic, please reply to this email. Greetings, Jelle van der Waa

1 0

[arch-devops] Reminder about allowed activities on homedir.archlinux.org
by Sven-Hendrik Haase 06 Jan '20

06 Jan '20

Hey everyone, I'd like to remind everyone that homedir.archlinux.org is to be used as a host for files topically surrounding the Arch Linux project. It's not a personal file host. You can run small controlled scripts on it in case you want to generate public HTMLs or whatever but please keep it somewhat related to Arch. If in doubt, ask your nearest DevOps. The server is a shared resource and it'll only work if everyone plays by the rules voluntarily. Cheers, Sven

1 0

[arch-devops] Devops meeting summary - 02-01-2020
by Jelle van der Waa 03 Jan '20

03 Jan '20

Hi all, I forgot to send a notification out, but we had a short meeting yesterday night. Most of the discussions where follow ups and one new topic about a shared password manager for the Arch team. You can read the notes here: https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-01-02

1 0

[arch-devops] soyuz (pkgbuild.com) server move to homedir.archlinux.org
by Sven-Hendrik Haase 01 Jan '20

01 Jan '20

Hi all, I set up a new Hetzner VPS that is going to become our new homedir/public_html server available to all TUs and Devs like soyuz was. We decided to decommission soyuz and put the public_html stuff on its own server for security reasons, to cut costs, and so that we can compartmentalize further. The server uses a Hetzner Cloud Volume which we can scale if we want but for now, it's 100GiB of zstd-compressed btrfs. If possible, we'd like to keep it at this size for the time being. You can host your own repos there if you want and that's fine. Please talk to us beforehand if you see yourself exhausting the volume with what you want to do. If you had stuff hosted in the public_html of soyuz, I'd ask you to transfer stuff over to the new box which is already reachable at the names pkgbuild.com (you'll get an SSH error because of this) and homedir.archlinux.org. Please check if you can throw away some old stuff/junk that you might not necessarily need on the new server. This new box is *not for building*. That's what dragon is for. Please only put documents/files onto the new pkgbuild.com that you actually want people to be able to access publicly. The box has no building facilities and you get no sudo'd commands. soyuz is going to be decommissioned no sooner than 2020-01-01 but do not expect it to hang around for long after that. We will not transfer any files for you from soyuz. In other words: All data not explicitly transferred by you will be destroyed after 2020-01-01. Please bring up any problems you might have with this new server on the arch-devops list or IRC (#archlinux-devops). Enjoy, Sven

2 3

[arch-devops] Linux lockdown mode deployed
by Jelle van der Waa 23 Dec '19

23 Dec '19

Hi All, I've deployed a new Linux hardening setting on all our VPS'es which is available since 5.4. Which makes it harder for root to modify the running kernel by shielding off some functionality for userland. [1] No application should rely on this features so everything should still work as normal. Currently it is deployed as tmpfiles.d file which is suboptimal but adding it to our bootloader seems to be hard since we currently already enable btrfs via lineinfile. Maybe the grub configuration should live in our ansible repository? [1] https://git.archlinux.org/infrastructure.git/commit/?id=2c7538040f6353633ad… Greetings, Jelle

3 2