January 2019 - Arch-devops - lists.archlinux.org

Re: [arch-devops] [arch-dev-public] Uploading old packages on archive.org (Was: Archive cleanup)
by Florian Pritz 04 Mar '19

04 Mar '19

Hi Baptiste, On Thu, Jun 14, 2018 at 10:28:17AM +0200, Florian Pritz via arch-dev-public <arch-dev-public(a)archlinux.org> wrote: > On 10.06.2018 01:35, Baptiste Jonglez wrote: > > Archival of all packages between September 2013 and December 2016 is finished: > > > > https://archive.org/details/archlinuxarchive > > > > Here is some documentation on this "Historical Archive" hosted on archive.org: > > > > https://wiki.archlinux.org/index.php/Arch_Linux_Archive#Historical_Archive So the archive is still growing and we'll need to clean it up again soonish. Where do you have the archive.org uploader and is it in a state where we (devops team) can also easily use it? If not, could you help us getting it to that point? We also have an archive cleanup script here[1]. Maybe the uploader can be integrated there? I don't know how complicated it is. [1] https://github.com/archlinux/archivetools/blob/master/archive-cleaner Florian

2 6

[arch-devops] Let's get a big build box
by Sven-Hendrik Haase 25 Jan '19

25 Jan '19

Hi all, so this has been a long time coming as you know from IRC but now I'm actually taking the time to write an email. :P ## Suggested new server and finances So I'd like us to get a big build box. Specifically this one: https://www.hetzner.de/dedicated-rootserver/dell/dx292 This would be an upgrade to soyuz (and the current soyuz would go away). Total cost with 2x1.92TiB NVMe disks and 256GiB of RAM is € 461.00/month + € 219.00 setup. soyuz currently costs us € 54.00 so we'd be paying € 407.00/month extra. This is a big step up in cost but 1) our infra costs are very low all in all otherwise and 2) frankly we just have a ton of money laying around doing nothing and while that doesn't mean we have to spend it needlessly, I believe that this is a useful thing to do with the money. ## Performance ### Processors The suggested DX292 has two Intel Xeon Gold 6130 16-Core processors while the current one has a single Intel Xeon CPU E3-1275 v5. From benchmarks, I'm estimating the compute power to be almost exactly 4 times as good in the suggested server [0][1] for our workloads. ### Disks We currently have spinning disks in soyuz and that isn't great for building. While I believe soyuz instead puts chroots onto a tmpfs to mitigate this, it takes away from the usable RAM that we have. This is actually a problem as the server has ran out of memory a few times before. Using RAID1 NVMs (as in the suggested new server) for building would make that workaround unnecessary as these should just generally be fast enough for building. ## Reasoning I believe that the current soyuz is too small for bigger rebuilds and big packages for them to get done quickly. I've heard some members of the team complain about rebuild times of C++-based rebuilds in the past as well. I know that soyuz sits mostly idle currently but I suppose the reason for that is that some people build big packages on their own, faster machines (I know that I do this and some TUs as well). On my machine (12 threads), tensorflow takes ~10h to compile while pytorch and arrayfire are at 2-3h. Yes, these are certainly outliers but imagine we have quite a few more of these packages that I don't know about. Also big rebuilds like KDE, boost would benefit. Ultimately, we all want Arch CI and then we could theoretically dynamically spin up/down big build slaves automatically as we need. However, this is currently blocked by reproducible builds AND the svn-git migration. Therefore, I don't see that happening any time soon. This proposal is for getting a practical solution now and not in a few months/years. Additionally, this big server could also serve as a testbed for the CI. ### Alternatives People have suggested this [2] alternative in the past and while it's quite a bit cheaper, it's also only about half as powerful. While the CPU is about the same speed [3], it only has one of them. ## Closing I know that some people have been skeptical about getting a big, expensive server but I hope I made a good case for why I think we should get one. If not, well, at least we'll have it in the archive. Sven [0] cpubenchmark.net shows only the single processor version but we can roughly double the performance given our workload to estimate dual processor performance: https://www.cpubenchmark.net/compare/Intel-Xeon-Gold-6130-vs-Intel-Xeon-E3-… [1] geekbench.com has whole systems and I actually found a DELL R740 which has the exact same processor configuration as the R640 DX292 from Hetzner that I'm suggesting. From those numbers, 4x the compute power seems about right: https://browser.geekbench.com/v4/cpu/11406589 vs https://browser.geekbench.com/v4/cpu/11568488 [2] https://www.hetzner.de/dedicated-rootserver/ax160 [3] https://www.cpubenchmark.net/compare/AMD-EPYC-7401P-vs-Intel-Xeon-Gold-6130…

5 6

[arch-devops] Archweb Python 3 update
by Jelle van der Waa 25 Jan '19

25 Jan '19

Hi All, After nearly a year? I've finally deployed the Python 3 version of Archweb with Django 2.1.5 on nymeria (staging server). So far I haven't found any issues but I will do more testing over the weekend. I want to merge the Python 3 branch to master soon and switch archlinux.org. Since next weekend is FOSDEM I think I'll do it the week after. So ~4 February maybe. Feedback is welcome, I'm kind of nervous as this is a big release with many changes. But this will open the ability to add CSP and other security headers / 2FA for login. After those changes the biggest feature wanted is probably automatic out of date flagging of packages, which could be achieved with repology or another API. But repology would be preferable since Archweb does not know about source urls :-) (input welcome) -- Jelle van der Waa

3 3

[arch-devops] Reproducible Build nodes
by Jelle van der Waa 14 Jan '19

14 Jan '19

Hi all, We promised two PIA boxes to be used for reproducible build rebuilding nodes. [1] The boxes are already defined in ansible's hosts and don't have to be converted to Debian if we can get them working with Arch. [2] [3] The boxes should be added to their Jenkins server via ssh. We should setup a user with sudo rights and ssh key access to set the box up as slave. My plan if no one objects is: 1) Update the boxes and hope they come back up (~ 200 updates). 2) Add DNS entries for the two boxes {repro1,2}.pkgbuild.com. 3) add holger to this box with sudo rights (an ansible role probably). [4] 4) Add them to monitoring I suppose. Some background about reproducible builds and the usage of the boxes. The boxes will detect package updates, rebuild these packages twice and check if they are reproducible. These rebuilds have already found failing to build from source packages, 404 urls and other issues which improves are packages! [1] https://tests.reproducible-builds.org/archlinux/archlinux.html [2] https://git.archlinux.org/infrastructure.git/tree/hosts [3] https://salsa.debian.org/qa/jenkins.debian.net/blob/master/bin/reproducible… [4] https://salsa.debian.org/qa/jenkins.debian.net/blob/master/authorized_keys/… -- Jelle van der Waa

2 2