- Arch-devops - lists.archlinux.org

[arch-devops] Devops meeting 19:30 - 20:30 UTC
by Jelle van der Waa 24 Oct '19

24 Oct '19

Hi all, The agenda for the meeting in 10 minutes is: - Gitlab - Forum migration - Backups offsite - Geo mirror location

1 0

[arch-devops] Devops meeting summary - 09/10
by Giancarlo Razzolini 23 Oct '19

23 Oct '19

Hi All, I'm sending the summary of yesterday's meeting: Mirroring of Arch Linux Archive runner2.archlinux.org becomes ger.mirror.pkgbuild.com use all PIA boxes as public mirrors for Archive AND repos Sven will be in handle of these Gitlab PoC Gitlab is running, we need to re-package keyclock for better security, anthraxx will handle this. Planet migration Make planet.archlinux.org part of archweb Mailman 3 Ask fedora regarding their mailman migration Create a package and test migration - Jerome will look into this Blocker https://gitlab.com/mailman/mailman/issues/343 Lack of overview of services/hosts Deferred Twitter Deferred Forum migration It was decided to migrate it to a VPS and then move to discourse. Possibly with SSO already. Soyuz retirement We are going to point internal mirroring to repos.archlinux.org Create new server and migrate content for public_html. Getting a storage box instead of vostok Move to storage box for backups (create a kanban ticket and figure out automation). We currently pay 30 euros Backup the archive Requires a larger backup server (currently 1.2TB free on vostok) Offsite backups Decide if borg will handle it or we are doubling the backup with another tool. Settle on external service: borgbase glacier online.net cold storage dedicated server I have summed up a few things for clarity, but I guess it covers what we've discussed. Regards, Giancarlo Razzolini

6 10

[arch-devops] Meeting Tomorrow 19:30 - 20:30 UTC
by Jelle van der Waa 11 Oct '19

11 Oct '19

Hi All, This is a reminder for our bi-weekly devops meeting tomorrow 10-10-2019 at 19:30 - 20:30 UTC. I propose to host this edition on mumble, as it allows for easier and more effective communication. (Sven can we use your mumble instance?). The agenda for the meeting is: * Mirroring the Arch Linux Archive * Gitlab PoC update * Planet migration * Mailman 2 status * Lack of overview of services/hosts Greetings, Jelle van der Waa

4 5

[arch-devops] Maintenance mode
by Giancarlo Razzolini 25 Sep '19

25 Sep '19

Hi All, Ever since I've done some of our migrations, I've used to put our services into maintenance mode manually, by copying the nginx.conf file over the service and reloading nginx. I have been toying with the idea of having a maintenance role for our services for some time, but I never came up with a good way of doing this. Yesterday I've drafted some stuff and now we have a maintenance role that can be "plugged" into other roles. I have added the maintenance mode for most of our roles already, and basically anything that has a nginx configuration can have this maintenance mode. I've added some documentation to the README file. But basically all you need to do is to pass -e maintenance=true to command line on the playbook run. If you don't pass it, and run the book again, then the normal nginx configuration is applied and things go back as they were. The next step is to add nginx configuration to allow the ip address of the person running the book through, to be able to test things, but keep the maintenance mode for everyone else. Just don't know yet how to glue the specifics. Regards, Giancarlo Razzolini

1 0

[arch-devops] Devops promotion process - Levente
by Giancarlo Razzolini 12 Sep '19

12 Sep '19

Hi Guys, Since we don't have a formal process for giving out the keys of the kingdom to people, I thought writing an email would be the best way to start this process. We recently removed root privileges of some guys and we can always use an extra (capable) hand. I have been trying to get Levente to join -devops for a while now, and it seems the message finally got through. I've talked with Levente and he wants to give a security perspective overview of our roles, it seems, as you might've noticed with the recent sudo change on our build machines. What are your thoughts? (If it were just up to me, I'd given him root access already, so +1). Regards, Giancarlo Razzolini

5 5

Re: [arch-devops] Devops promotion process - Levente
by Jan Alexander Steffens 12 Sep '19

12 Sep '19

+1 from me too; Levente is good.

1 0

[arch-devops] restricting sudoers PATH
by Jelle van der Waa 03 Sep '19

03 Sep '19

Hi all, Thanks to anthraxx, we now restrict the PATH which `sudo extra-x86_64-build` and other sudoers specific infra uses using restrict_path. To circumvent users overridding their own PATH with tools which are used in our build scripts which basically allows privilege escalation. [1] This shouldn't cause any issues, if they do contact me or anthraxx. [1] https://git.archlinux.org/infrastructure.git/commit/?id=1eb1dd41f8c734380a3… Greetings, Jelle

3 2

[arch-devops] (no subject)
by Pierre Schmitz 02 Sep '19

02 Sep '19

Hi all, I was wondering if there were any objections if we move the MediaWiki config from the archwiki repo into infrastructure. ATM there is a file called LocalSettings.archlinux.org.php in the archwiki repo which gets included by the file LocalSettings.php that is store in the infrastructure repo. Changing the Wiki configuration is kind of inconvenient. You have to decide which config file to use, if you choose LocalSettings.archlinux.org.php you have to tag a new release and update the infrastructure repo as well. To sum things up: the only site specific configuration LocalSettings.php should be stored in the infrastructure repo. Greetings, Pierre

2 1

[arch-devops] fail2ban deployed for the wiki
by Jelle van der Waa 01 Sep '19

01 Sep '19

Hi All, Today and earlier we suffered a huge influx of bots from china crawling the wiki. We have taken a few measures to reduce the impact: - fastcgi_cache for /load.php which caches the load.php page for 10 minutes which contains assets such as css/js. This should offload php-fpm which was overloaded. - Enabled a plugin to disable viewing wiki page revisions for anonymous users this reduces some heavy requests and hopefully the impact of bots The last step was enabling fail2ban for HTTP/HTTPS requests, only for the wiki now. It blocks every ip doing more then 300 requests in 30 minutes. This might be a bit too aggressive but for now it dropped our load from ~ 20 -> ~ 2/3 and blocks 85 ips. This can be tweaked later, maybe it should be 400/500? To view the blocked ips execute: fail2ban-client status wiki-nginx-dos To unban a valid IP: fail2ban-client unban $ip fail2ban does use a lot of CPU which we should look into tuning, but maybe it will get better over time when the log files are smaller due to less bots coming through :-) The fail2ban role is in ansible, but not suited yet to be re-used on other hosts. Greetings, Jelle

1 0

[arch-devops] Bi-weekly infrastructure meeting
by Jelle van der Waa 21 Aug '19

21 Aug '19

Hi All, We currently have a lot of things which are open and in progress. It would be nice if we where a bit more active in getting things done. I propose we hold a mumble or irc meeting bi-weekly on a to be scheduled time. If no one really objects, I'll create a time picker for when we can hold these meetings! Greetings, -- Jelle van der Waa

7 9