- Arch-devops - lists.archlinux.org

[arch-devops] Archweb migration
by Jelle van der Waa 08 Feb '19

08 Feb '19

Hi All, Tommorow night (CET) I will update archweb (archlinux.org) to Python 3. We've already tested it on a staging server and it seems to work well so tommorow I'll update it on our main server. This will mean that archlinux.org will be unavailabe for ~ 30 minutes since the upgrade requires some manual work with removing and creating a new virtualenv. -- Jelle van der Waa

1 1

[arch-devops] Secure PGP key on our servers
by Jelle van der Waa 06 Feb '19

06 Feb '19

Hi all, The security team wants to setup a secure security(a)archlinux.org address with schleuder which decrypts an encrypted mail and re-encrypts it to recipients of this address. We would like to use a PGP token for it, so I've contacted Hetzner about the option to install it. If it can't happen in Hetzner, we'll have to discuss a new solution. [1] [1] https://kanboard.archlinux.org/project/1/task/98 -- Jelle van der Waa

1 0

[arch-devops] Let's get a big build box
by Sven-Hendrik Haase 25 Jan '19

25 Jan '19

Hi all, so this has been a long time coming as you know from IRC but now I'm actually taking the time to write an email. :P ## Suggested new server and finances So I'd like us to get a big build box. Specifically this one: https://www.hetzner.de/dedicated-rootserver/dell/dx292 This would be an upgrade to soyuz (and the current soyuz would go away). Total cost with 2x1.92TiB NVMe disks and 256GiB of RAM is € 461.00/month + € 219.00 setup. soyuz currently costs us € 54.00 so we'd be paying € 407.00/month extra. This is a big step up in cost but 1) our infra costs are very low all in all otherwise and 2) frankly we just have a ton of money laying around doing nothing and while that doesn't mean we have to spend it needlessly, I believe that this is a useful thing to do with the money. ## Performance ### Processors The suggested DX292 has two Intel Xeon Gold 6130 16-Core processors while the current one has a single Intel Xeon CPU E3-1275 v5. From benchmarks, I'm estimating the compute power to be almost exactly 4 times as good in the suggested server [0][1] for our workloads. ### Disks We currently have spinning disks in soyuz and that isn't great for building. While I believe soyuz instead puts chroots onto a tmpfs to mitigate this, it takes away from the usable RAM that we have. This is actually a problem as the server has ran out of memory a few times before. Using RAID1 NVMs (as in the suggested new server) for building would make that workaround unnecessary as these should just generally be fast enough for building. ## Reasoning I believe that the current soyuz is too small for bigger rebuilds and big packages for them to get done quickly. I've heard some members of the team complain about rebuild times of C++-based rebuilds in the past as well. I know that soyuz sits mostly idle currently but I suppose the reason for that is that some people build big packages on their own, faster machines (I know that I do this and some TUs as well). On my machine (12 threads), tensorflow takes ~10h to compile while pytorch and arrayfire are at 2-3h. Yes, these are certainly outliers but imagine we have quite a few more of these packages that I don't know about. Also big rebuilds like KDE, boost would benefit. Ultimately, we all want Arch CI and then we could theoretically dynamically spin up/down big build slaves automatically as we need. However, this is currently blocked by reproducible builds AND the svn-git migration. Therefore, I don't see that happening any time soon. This proposal is for getting a practical solution now and not in a few months/years. Additionally, this big server could also serve as a testbed for the CI. ### Alternatives People have suggested this [2] alternative in the past and while it's quite a bit cheaper, it's also only about half as powerful. While the CPU is about the same speed [3], it only has one of them. ## Closing I know that some people have been skeptical about getting a big, expensive server but I hope I made a good case for why I think we should get one. If not, well, at least we'll have it in the archive. Sven [0] cpubenchmark.net shows only the single processor version but we can roughly double the performance given our workload to estimate dual processor performance: https://www.cpubenchmark.net/compare/Intel-Xeon-Gold-6130-vs-Intel-Xeon-E3-… [1] geekbench.com has whole systems and I actually found a DELL R740 which has the exact same processor configuration as the R640 DX292 from Hetzner that I'm suggesting. From those numbers, 4x the compute power seems about right: https://browser.geekbench.com/v4/cpu/11406589 vs https://browser.geekbench.com/v4/cpu/11568488 [2] https://www.hetzner.de/dedicated-rootserver/ax160 [3] https://www.cpubenchmark.net/compare/AMD-EPYC-7401P-vs-Intel-Xeon-Gold-6130…

5 6

[arch-devops] Archweb Python 3 update
by Jelle van der Waa 25 Jan '19

25 Jan '19

Hi All, After nearly a year? I've finally deployed the Python 3 version of Archweb with Django 2.1.5 on nymeria (staging server). So far I haven't found any issues but I will do more testing over the weekend. I want to merge the Python 3 branch to master soon and switch archlinux.org. Since next weekend is FOSDEM I think I'll do it the week after. So ~4 February maybe. Feedback is welcome, I'm kind of nervous as this is a big release with many changes. But this will open the ability to add CSP and other security headers / 2FA for login. After those changes the biggest feature wanted is probably automatic out of date flagging of packages, which could be achieved with repology or another API. But repology would be preferable since Archweb does not know about source urls :-) (input welcome) -- Jelle van der Waa

3 3

[arch-devops] Reproducible Build nodes
by Jelle van der Waa 14 Jan '19

14 Jan '19

Hi all, We promised two PIA boxes to be used for reproducible build rebuilding nodes. [1] The boxes are already defined in ansible's hosts and don't have to be converted to Debian if we can get them working with Arch. [2] [3] The boxes should be added to their Jenkins server via ssh. We should setup a user with sudo rights and ssh key access to set the box up as slave. My plan if no one objects is: 1) Update the boxes and hope they come back up (~ 200 updates). 2) Add DNS entries for the two boxes {repro1,2}.pkgbuild.com. 3) add holger to this box with sudo rights (an ansible role probably). [4] 4) Add them to monitoring I suppose. Some background about reproducible builds and the usage of the boxes. The boxes will detect package updates, rebuild these packages twice and check if they are reproducible. These rebuilds have already found failing to build from source packages, 404 urls and other issues which improves are packages! [1] https://tests.reproducible-builds.org/archlinux/archlinux.html [2] https://git.archlinux.org/infrastructure.git/tree/hosts [3] https://salsa.debian.org/qa/jenkins.debian.net/blob/master/bin/reproducible… [4] https://salsa.debian.org/qa/jenkins.debian.net/blob/master/authorized_keys/… -- Jelle van der Waa

2 2

Re: [arch-devops] [arch-projects] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
by Eli Schwartz 20 Dec '18

20 Dec '18

On 12/4/18 1:09 PM, Eli Schwartz wrote: > Whenever adding new package files to the pool of distributed packages, > hardlink a copy of every package it was built with, into a > "reproducible" pool, and log which file required it. The question becomes, where can I store these? As-is, this will burden the mirror network as well. Unsure how to handle this. Could this be configurable by the mirror, as ISOs are now? Should we exclusively self-host this, and if so, where? archive.archlinux.org is managed by another service with its own exclusively writable location. > Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> > --- > config | 1 + > config.local.svn-community | 1 + > config.local.svn-packages | 1 + > db-functions | 49 +++++++++++++++++++++++++++++++------- > db-update | 4 ++++ > 5 files changed, 48 insertions(+), 8 deletions(-) > > diff --git a/config b/config > index 1cfe11f4..5144fca7 100644 > --- a/config > +++ b/config > @@ -3,6 +3,7 @@ > FTP_BASE="/srv/ftp" > PKGREPOS=() > PKGPOOL='' > +EXTRA_PKGPOOLS=() > SRCPOOL='' > TESTING_REPO='' > STABLE_REPOS=() > diff --git a/config.local.svn-community b/config.local.svn-community > index 5d61b5ea..15bcc17f 100644 > --- a/config.local.svn-community > +++ b/config.local.svn-community > @@ -2,6 +2,7 @@ > > PKGREPOS=('community' 'community-testing' 'community-staging' 'multilib' 'multilib-testing' 'multilib-staging') > PKGPOOL='pool/community' > +EXTRA_PKGPOOLS=('pool/packages') > SRCPOOL='sources/community' > SVNREPO='file:///srv/repos/svn-community/svn' > SVNUSER='svn-community' > diff --git a/config.local.svn-packages b/config.local.svn-packages > index 34aab35c..75986b65 100644 > --- a/config.local.svn-packages > +++ b/config.local.svn-packages > @@ -2,6 +2,7 @@ > > PKGREPOS=('core' 'extra' 'testing' 'staging' 'kde-unstable' 'gnome-unstable') > PKGPOOL='pool/packages' > +EXTRA_PKGPOOLS=('pool/community') > SRCPOOL='sources/packages' > SVNREPO='file:///srv/repos/svn-packages/svn' > SVNUSER='svn-packages' > diff --git a/db-functions b/db-functions > index 7aeedced..2b1ae87a 100644 > --- a/db-functions > +++ b/db-functions > @@ -165,20 +165,23 @@ repo_unlock () { #repo_unlock <repo-name> <arch> > fi > } > > +# usage: _grep_all_info pkgfile infofile key > +_grep_all_info() { > + local _ret=() > + > + mapfile -t _ret < <(/usr/bin/bsdtar -xOqf "$1" "${2}" | grep "^${3} = ") > + > + printf '%s\n' "${_ret[@]#${3} = }" > +} > + > # usage: _grep_pkginfo pkgfile pattern > _grep_pkginfo() { > - local _ret > - > - _ret="$(/usr/bin/bsdtar -xOqf "$1" .PKGINFO | grep "^${2} = " | tail -1)" > - echo "${_ret#${2} = }" > + _grep_all_info "${1}" .PKGINFO "${2}" | tail -1 > } > > # usage: _grep_buildinfo pkgfile pattern > _grep_buildinfo() { > - local _ret > - > - _ret="$(/usr/bin/bsdtar -xOqf "$1" .BUILDINFO | grep "^${2} = " | tail -1)" > - echo "${_ret#${2} = }" > + _grep_all_info "${1}" .BUILDINFO "${2}" | tail -1 > } > > # Get the package base or name as fallback > @@ -444,4 +447,34 @@ arch_repo_modify() { > REPO_MODIFIED=1 > } > > +# Build an index of dependent packages needed by a given pkgfile > +# usage: make_reproducible pkgfile [check] > +make_reproducible() { > + local pkg dir pkgs=() pkgfile pkgfiles=() > + > + mapfile -t pkgs < <(_grep_all_info "${1}" .BUILDINFO installed) > + > + for pkg in "${pkgs[@]}"; do > + for dir in "${FTP_BASE}/${PKGPOOL}" "${EXTRA_PKGPOOLS[@]/#/${FTP_BASE}/}" "${STAGING}"/**/; do > + if pkgfile="$(getpkgfile "${dir}/${pkg}"${PKGEXTS} 2>/dev/null)"; then > + pkgfiles+=("${pkgfile}") > + continue 2 > + fi > + done > + error "could not find existing package for %s" "${pkg}" > + return 1 > + done > + > + if [[ ${2} = check ]]; then > + return 0 > + fi > + > + for pkg in "${pkgfiles[@]}"; do > + if [[ ! -f ${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/} ]]; then > + ln -L "${pkg}" "${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/}" > + fi > + echo "${1}" >> "${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/}.buildlinks" > + done > +} > + > . "$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")/db-functions-${VCS}" > diff --git a/db-update b/db-update > index 313fb999..11ec185f 100755 > --- a/db-update > +++ b/db-update > @@ -61,6 +61,9 @@ for repo in "${repos[@]}"; do > if ! check_builddir "${pkg}"; then > die "Package %s was not built in a chroot" "$repo/${pkg##*/}" > fi > + if ! make_reproducible "${pkg}" "check"; then > + die "Package %s is not reproducible" "${pkg}" > + fi > done > if ! check_splitpkgs "${repo}" "${pkgs[@]}"; then > die "Missing split packages for %s" "$repo" > @@ -82,6 +85,7 @@ for repo in "${repos[@]}"; do > # any packages might have been moved by the previous run > if [[ -f ${pkg} ]]; then > mv "${pkg}" "$FTP_BASE/${PKGPOOL}" > + make_reproducible "${FTP_BASE}/${PKGPOOL}${pkg##*/}" > fi > ln -s "../../../${PKGPOOL}/${pkgfile}" "$FTP_BASE/$repo/os/${pkgarch}" > # also move signatures > -- Eli Schwartz Bug Wrangler and Trusted User

3 4

[arch-devops] Switch wiki git repo from tarball based updates to git merges?
by Florian Pritz 05 Nov '18

05 Nov '18

Hi, AFAICT, our wiki repo has been created with an extracted tarball and updated by just extracting the new tarballs on top of it and committing the result. Due to that we've once lost a change (a2bbd243c85ea0e425ee3e8c380aba9f254cee61) when an update had been commited (7c2a4c95e1d96bc56d00fd133fa537acc1346f5b). I've noticed that we have this open pull request[1] and since the Vector skin is part of the upstream tarball I'm worried that if I were to apply the change, it would again be lost with the next update. [1] https://github.com/archlinux/archwiki/pull/18 I've thus prepared a local branch where I merged our repo with the upstream repo. That means that we can update by merging the upstream tags and thus git can track our changes we get merge conflicts when necessary. The only issue I see with this setup is that we also pull in upstream tags, but those obviously refer to the upstream versions which do not include our changes and thus won't work. We could disable tag fetching, but this has to be done each time someone configures the upstream repo in their local clone and it can't be done automatically. We could either ship a small script that configures the remote correctly, but it has to be executed manually by the user, or we just live with the tags and prefix our own tags with 'archwiki-' or similar in the future. We could also stay with the tarball update method, but I think that's unnecessarily complicated and error prone. Thoughts? Florian

2 2

[arch-devops] Build tools and repo management
by Dumont, Joey 01 Nov '18

01 Nov '18

Hi everyone! Our group is thinking of maintaining a subset of the msys2 repo to be used internally. I am wondering if there is documentation on the Arch Linux side on what tools you are currently using to automatically build the packages in the officially supported repos. I believe there must be some kind of dependency management as well, since official repos are never cursed by soname bump issues. Is there documentation on this as well? Specifically, is there a tool similar to apt-mirror? Thank you, and please don't hesistate to ask questions if I didn't make myself clear. Joey Dumont Scientific Computing Technical Advisor National Research Council Canada joey.dumont(a)nrc-cnrc.gc.ca<mailto:joey.dumont@nrc-cnrc.gc.ca>

3 3

[arch-devops] Mirrorbrain
by Phillip Smith 24 Sep '18

24 Sep '18

It's been discussed in the past, but I thought it worth having a semi-formal and documented discussion about it. Do we think setting up mirrorbrain[0] would be a worthwhile service? For those not familiar, the elevator pitch is that it's an open-source download redirector to coordinate a simple CDN. It is used by a number of open-source projects such as vlc, Gnome, KDE, LibreOffice and OpenSUSE. Architecturally speaking, it maintains a list of our mirrors and local copy of our repos, and monitors those mirrors for availability and staleness. HTTP(s) clients (ie, pacman) are then redirected to an appropriate mirror based on the clients geographic location and mirror health. Benefits include: 1. Ensuring users always receive up-to-date packages (mirrorbrain won't redirect to a mirror if that mirrors version of that package is outdated compared to the authoritative repository). 2. Automated monitoring of our mirror network to proactively detect stale or broken mirrors. Mirrorlist files (ie, /etc/pacman.d/mirrorlist) can be automatically generated based on MirrorBrain's data. 3. Reduced load on core mirrors, and load-balancing (for example, a 1Gbit mirror can be weighted to receive 10x the traffic of a 100MBps mirror). 4. Automatic MetaLink and Torrent file generation, with web-seeds (currently handled by hefur on luna?). Exceptions to redirection can be applied, for example to ensure security-sensitive files (checksum files perhaps) are always served directly from the authoritative repo. Requirements are pretty basic; apache with mod_asn, postgresql, and some python+perl modules. It can be run behind a reverse-proxy if we wanted to hide apache behind nginx. It's not a particular fast-moving project (last release 2014) but that's a reflection of it's stability IMHO. What do others think? I'm happy to take on the implementation project. Cheers, ~p [0] http://mirrorbrain.org/

4 5

[arch-devops] Centralized log monitoring and alerting?
by Andrew Crerar 11 Sep '18

11 Sep '18

Hi all, On 2018-07-28 there were some discussions in #archlinux-devops around setting up some sort of centralized logging/monitoring/alerting solution for the various services on Apollo (and maybe other?) server(s). I had mentioned possibly using the ELK[1] stack for this task. There was some back and forth about it potentially being a bit heavy handed for what was needed and how we would most likely need to repurpose/dedicate something like nymeria to handle the stack. There was also the suggestion of possibly using something like tenshi[2] if we're aiming for a low overhead solution, however, there would be much writing of the regexes. With that being said, the purpose of this email is to have a more formal discussion around what we're trying to capture from the logs, the actions we want to have taken with what ends up being captured, and possibly come to a consensus on what tool(s) we could leverage. Thoughts? Regards, Andrew [1] https://www.elastic.co/de/elk-stack [2]https://github.com/inversepath/tenshi

6 5