- Arch-devops - lists.archlinux.org

[arch-devops] Resignation
by Bartłomiej Piotrowski 06 Aug '20

06 Aug '20

Hi all, I think it's safe to say that I won't be able to find enough time to meaningfully contribute to DevOps team at this point. Thanks and good luck, Bart

2 1

[arch-devops] Stepping down from DevOps
by Florian Pritz 02 Aug '20

02 Aug '20

Hi, As you have probably noticed, my involvement in DevOps tasks has dwindled in recent months. I have given away a few tasks to others already, but since I hardly use my root access any more, I think it is finally time to officially open up the spot and let someone else take over. Back when we started the DevOps team nearly 5 years ago (good god, time flies), Allan asked me if I wanted to lead the "infrastructure team" (old name). I am glad that I have never needed to use the team leader hat since we could discuss pretty much every problem and reach an agreement. Given that things have worked so well in the past, I'm inclined to simply discontinue the (unused) team leader hat. Considering how little, well, actually not at all, I have used the hat, most people are probably not even aware of it, so this feel like the most logical choice. I think having a core team of active people that discuss the issues and then come to an agreement has worked well in the past and it is something worth keeping. I'm still amazed by what you guys accomplished over the last year and I hope that making space and letting others join will only allow the team to be even more awesome! All that said, please disable my root access and remove me from the various infrastructure/devops/root related email aliases and services (kanboard and gitlab come to mind). Please do keep me on the mirrors(a)archlinux.org alias for now though since I still handle mirror related requests. I also see that I am the only one on the dmarc-reports@ alias. If nobody is interested in the reports, please remove the relevant line from the DMARC DNS entry in the archlinux.org zone. Also, please remove my email address from the explicit hetzner contact list. If you think of anything else that I forgot, please disable it or, if in doubt, feel free to ask on IRC if I still need it. Thanks for the journey! Florian

2 1

[arch-devops] Meeting 30 July 18:30 UTC - 20:30 CEST
by Jelle van der Waa 28 Jul '20

28 Jul '20

Hi All, On Thursday we have another devops meetings with the following topics: * Gitlab go live, limits and monitoring * Grafana/prometheus * AUR migration notes/improvements * Next server(s) to migrate Greetings, Jelle

1 0

[arch-devops] Meeting 16 July 18:30 UTC - 20:30 CEST
by Jelle van der Waa 15 Jul '20

15 Jul '20

Hi all, Tommorow we have another devops meeting, topics: * prometheus monitoring * revoking access to inactive members * finer grained access * password manager * freswa Cya tommorow, Jelle

1 0

[arch-devops] Add Frederik Schwan to the devops team
by Sven-Hendrik Haase 15 Jul '20

15 Jul '20

Hey all, I propose adding Frederik Schwan to the devops team. He's eager to help Arch, he's knowledgeable, and does devops professionally. He's got a good amount of experience with mail stacks and cloud stuff and he'd like to help us out. In fact, he's already helped quite a bit with testing and some accessless devops tasks but the number of tasks requiring access that we currently have far outweigh the accessless tasks and I think he'd like to become more involved. Thoughts? Cheers, Sven

6 7

[arch-devops] Let's change our primary backup to a managed storage box
by Sven-Hendrik Haase 09 Jul '20

09 Jul '20

Hey all, I want to suggest we change our primary backup box to a managed box, namely a BX60 from Hetzner [0] for 40€/m with 10TB capacity. We currently have a SB35 with 2x3TB in a RAID1. The current box (vostok) is getting full and is sitting at 20% capacity remaining. The storage box supports Borg and so it's unlikely that we'll have to change our backup scheme. Why a BX60 and not a BX50 even though the capacity would currently suffice? I'd like to also start backing up the archive. The archive will become more relevant in Arch as reproducible builds gain more popularity and I think keeping the archive alive and well is an important part of that. We currently do not back up the archive for capacity reasons but we should start doing that. BX50 with its 5TB puts us too close to our projected needs (roughly 2.3TB (current effective backup size) + 2.3TB (archive)). Advantages of a BX60: - Less stuff to manage as Hetzner will handle disk failures and updates for us - More redundancy (FAQ says it can tolerate multiple disk failures while we can currently only handle a single disk failure) - Much higher capacity (about 3.4x as much) Disadvantages: - 10€ more per month - We'll have to migrate (though probably pretty easy to do as the backup stuff is pretty nicely ansibled) - Only 10 connections allowed at once (workable for us but worth mentioning) - We currently have no operating experience with storage boxes from Hetzner What do you think? Cheers, Sven [0] https://www.hetzner.de/storage/storage-box/bx60

2 5

[arch-devops] Resignation
by fukawi2 | Arch Linux 30 Jun '20

30 Jun '20

Hi all, For various reasons, I have to formally resign from the ArchLinux DevOps team. I will maintain my role as Forum Administrator, so I would appreciate keeping my @archlinux.org email address. Non-root access to the mail server would also be great to maintain if possible, for debugging issues for forum users (eg, missing password reset emails etc). If not, I'll just harass #-devops IRC when I need to get the logs for something ;D Cheers, ~Phil

2 1

[arch-devops] Meeting 2th July 18:30 UTC - 20:30 CEST
by Jelle van der Waa 30 Jun '20

30 Jun '20

Hi all, Another week, another meeting, topics: * Storagebox for backups * Gemini migration * Removal of hefurd torrent tracker * Security audit Gitlab/Keycloak * Resignation/Addition handling of members to DevOps Greetings, Jelle

1 0

[arch-devops] Broken disk on vostok
by Sven-Hendrik Haase 29 Jun '20

29 Jun '20

Hey all, During a small routine check I noticed that we have a broken disk on vostok. In fact, according to the log, we've had it for 7 months at least which is a bit embarrassing given that this is RAID1 only and it's also our primary backup box. This really goes to show that we need better monitoring/more meaningful alerts. Anyway, I had Hetzner replace the disk and the array is now rebuilding. Cheers, Sven

4 8

[arch-devops] repos.archlinux.org has been migrated to a new server
by Jelle van der Waa 26 Jun '20

26 Jun '20

Hi all, repos.archlinux.org, svn.archlinux.org and rsync.archlinux.org are now on a new server which has plenty of diskspace for us to continue packaging for a while (16T free). Some important notes: The ssh hostkeys changed for repos.archlinux.org which means if you use archco/communityco this will fail without a useful error. So take care of your .ssh/known_hosts. The ssh host keys for gemini/repos.archlinux.org are: # gemini.archlinux.org 1024 SHA256:F1Corf6i2U72yub+CIzzGHLOMVKVnjALh1YHM8gBjxE root(a)gemini.archlinux.org (DSA) 256 SHA256:If51DkTftUpDAFz65totgDfTd/ddu/2w/RBZIHtY74U root(a)gemini.archlinux.org (ECDSA) 256 SHA256:wUrJYf9+zOpIEUQ3ndgarK0PjzPICa1frmu7mpL4e14 root(a)gemini.archlinux.org (ED25519) 3072 SHA256:Rltnuln3bjsHJwVbys/LnYCj7hO6srPoa15JP8QhmlQ root(a)gemini.archlinux.org (RSA) https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/docs/ss… https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/docs/ss… If anything unexpected happens please either mail arch-devops(a)lists.archlinux.org or #archlinux-devops on irc. Greetings, Jelle

2 2