- Arch-devops - lists.archlinux.org

[arch-devops] New devops member - Phillip Smith (fukawi2)
by Florian Pritz 01 Feb '18

01 Feb '18

Hi, We have a new member joining the sysadmin/devops team by recommendation of Allan. I've already rolled out root and git access. Welcome Phillip, let's break stuff! (slightly) ;) Florian

4 6

[arch-devops] Secondary backup ideas
by Florian Pritz 12 Jan '18

12 Jan '18

Hi, Bartłomiej pointed out that we somehow stopped using the mailing list for discussion so here we go. Currently we only create backups on vostok using borg. A possibly quite big problem with this is that if an attacker gains access to a server, they also have sufficient access to remove all backups of that server. We could restrict that in borg, but then borg wouldn't be able to clean up old backups regularly and our repos would grow forever. A better solution is to create backups of the backups in a way that the front end servers can not delete any of these secondary backups. Possible solutions include: - Creation of a second layer of backups (backups of the backups) on vostok. This roughly doubles our space requirement and we are currently at 44% usage so this won't work well/for long. Unless we can use file system level snapshots or similar for this to reduce the required space, it's out. - Put the secondary backups on a different, possibly new, machine using borg. The backup would be created on vostok from the existing backup data. - Put them on AWS glacier. Roughly 4€/month and TB; suggested by Tyler from Arch Linux 32 Using glacier would require that we export tarballs (supported by borg) and then upload them. However, since the backups are encrypted and vostok is not supposed to be able to read them, the tarballs need to be created on and uploaded from the servers themselves. This may become a cpu/bandwidth/traffic concern if done often. Tyler is currently investigating this for Arch Linux 32's backups AFAIK. Does anyone have other ideas what we could do here to ensure that we have backups of the backups? The most important requirements are that no matter which server an attacker manages to get access to, they cannot read any user data from other servers and that they cannot remove all backups from that server alone. Florian

4 8

[arch-devops] Status of the Arch Twitter Account
by D E 09 Sep '17

09 Sep '17

Good morning, All! As per [the brief discussion on r/archlinux](https://www.reddit.com/r/archlinux/comments/6ykr93/is_the_arch…, could you please tell me the status of the Arch Twitter account? I don't know about anyone else, but I've grown fond of it over the past year. Regards, --A

2 1

[arch-devops] Migrating mail server on Sunday
by Florian Pritz 03 Sep '17

03 Sep '17

Hi, I will migrate our mail server from nymeria to orion on Sunday (tomorrow, September 3rd) starting around 12:00 UTC. The domains will stay the same, user names will stay the same expect for one user. The server may be unavailable for some time during the migration. User passwords are not migrated so you will have to set the password again on orion using `passwd`. If you have already set a password previously but forgot it, I can remove the password for you. Mailing lists will continue to work normally if the @lists.archlinux.org addresses are used. @archlinux.org addresses will work again after the move. If you have any questions or notice any problems, please contact me via mail or on IRC. Florian

1 1

[arch-devops] Caching problem with http://www.archlinux.org/check_network_status.txt
by Daniel Aleksandersen 21 Jul '17

21 Jul '17

Hi, Just wanted to report a Arch Linux infrastrucutre problem, and hope this was the right spot. The NetworkManager test probe page, located at http://www.archlinux.org /check_network_status.txt, should be made cache-aware. Currently, users are testing the connection to their web accelerator or intermediary proxy cache and not their internet connection as expected. Please add a `Cache-Control: max-age=0, must-revalidate` HTTP response header to this page to resolve this problem. I wrote a blog post earlier this week about the problem, but had some problem finding the appropriate place to report this to Arch. https://ctrl.blog/entry/network-connection-http-checks -- Daniel

3 2

[arch-devops] Cleaning up archive.archlinux.org
by Florian 11 Jul '17

11 Jul '17

Hi, While I like the idea of the archive, it currently does not implement any kind of cleanup and forever grows in size. Right now is is somewhere around 50 million files with a total size of 1.7TB. Those are very rough numbers since calculating them for real takes considerable time. This creates multiple problems: - The disks only have ~700gb more space before they are full. We will reach that eventually - The backup creation takes somewhere between 4 to 5 hours depending on machine load. I am thinking about increasing the backup frequency where possible, but if one backup takes 4 hours that won't happen. I am also worried that if we ever need to restore, this could easily take well over 24 hours. The initial backup (which read all data rather than ignoring unchanged data) took something around 24 hours, but that was with less files. Essentially this means we need to put some kind of automatic deletion of old data in place and reduce the size of the archive. What would be a good time frame here? Have we defined a clear goal for the archive that we can use as a guideline? My gut says 6 months should be enough, but feel free to disagree. Once we have decided how long data should be kept, we also need something to actually delete it. Deleting files from the ./repos directory should be simple, but the ./packages directory is more complicated because it is not nicely separated into directories per day. Sebastien, do you have an idea about how we could delete old data or do you have a script for that? Florian

6 8

[arch-devops] Maintenance window for luna
by Bartłomiej Piotrowski 17 Jun '17

17 Jun '17

Hi all, tomorrow around 22:00 CET, the server that hosts forums, wiki and AUR will be offline for planned maintenance. The downtime will take up to 2 hours; the plan is to reboot it into linux-hardened, then see if it is possible to switch from Percona to MariaDB (and if the mysterious crash from the past still happens, gather debug data for reporting it upstream). Sorry for the inconvenience, Bartłomiej

1 1

[arch-devops] Webserver maintenance downtime
by Florian Pritz 12 Apr '17

12 Apr '17

Hi, Our web server (apollo.archlinux.org) appears to experience some hardware issues and will be checked by the ISP shortly. Please expect a downtime of up to 14 hours. The following services will be affected: bugs, planet, security, www (main website) Other services (wiki, bbs, mailinglists) are not affected by this. I'm sorry for the inconvenience. Florian

1 0

[arch-devops] archive.archlinux.org
by Sébastien Luttringer 01 Apr '17

01 Apr '17

Hello, Archive is now on Orion. DNS has been updated and the ansible config pushed. Cheers, -- Sébastien "Seblu" Luttringer GPG: 0x2072D77A

1 0

[arch-devops] Monitoring
by Florian Pritz 20 Mar '17

20 Mar '17

Hi, I've finally started setting up a monitoring solution (zabbix). Thanks to Bartłomiej for getting zabbix packages into our repos! If anyone from the devops team wants accounts on zabbix, please send me a mail. Right now it doesn't monitor too much (items and hosts), but I plan to increase coverage over time. I'll likely also set up grafana later for pretty graphs. Florian

1 0