[arch-devops] Build tools and repo management

[Dumont, Joey]

Hi Florian and Bruno, 

This helps a lot, thank you. This gives a pretty good idea of what can be done with existing Arch Linux tools. I'll check what msys2 has ported over to see if we can use the same pipeline on that side.

As for documentation, I was simply wondering if there were pages on the DevWiki describing the details of maintaining an official repo, including generating the list of packages that need to be rebuilt. You answered my question!

Thanks!

-----Original Message-----
From: Florian Pritz [mailto:bluewind at xinu.at] 
Sent: November 1, 2018 6:49 AM
To: Arch Linux Infrastructure development discussion <arch-devops at lists.archlinux.org>
Cc: Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>; Bruno Pagani <bruno.n.pagani at gmail.com>
Subject: Re: [arch-devops] Build tools and repo management

On Wed, Oct 31, 2018 at 08:31:58PM +0100, Bruno Pagani via arch-devops <arch-devops at lists.archlinux.org> wrote:
> > I believe there must be some kind of dependency management as well, 
> > since official repos are never cursed by soname bump issues.
> >
> When a package has a soname dump (something we can detect with 
> checkpkg for instance), we generate a todo list 
> (https://www.archlinux.org/todo/) of depending packages to rebuild (using a script called sogrep).

To clarify, you can find checkpkg here[1] and sogrep (which uses a database created by `createlinks`) is part of our ansible repository[2]. When you built a package, run checkpkg to check for changed files/libraries in the new package. If a soname changed, you'll get a message and then you can use sogrep to generate a list of packages that need to be rebuilt. We use a [staging] repository for this, but you can also just do all the rebuilds locally and then release the packages all at once. Just make sure you have the new versions of everything installed in the chroot when you build higher-level packages.
Using a [staging] repository that is updated after each package build makes this easier since you can start with a fresh chroot each time and pacman will fetch the newest packages from [staging]. You can also check the built packages' .BUILDINFO file to verify that they were built with the correct dependency versions.

[1] https://git.archlinux.org/devtools.git/
[2] https://git.archlinux.org/infrastructure.git/tree/roles/sogrep

If you don't want to use createlinks and sogrep, you can also use makepkg's libprovides/libdepends support to track these dependencies directly in each package. Look in `man PKGBUILD` in the documentation of the provides/depends arrays for details on how to use it. I'm not sure if this works well when integrating with an external repo though so you might be better off with sogrep.

> > Is there documentation on this as well? Specifically, is there a 
> > tool similar to apt-mirror?

I'm not exactly sure what you are looking for, but we have a mirror script[3] which is linked on our mirror wiki page[4]. It's mostly just some fanciness around rsync to mirror our repo. I'm not sure how helpful that is with msys2. If you are looking for our PKGBUILDS, this page should get you started[5].

[3] https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh
[4] https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors
[5] https://wiki.archlinux.org/index.php/Arch_Build_System

Florian