[arch-devops] Build tools and repo management
Hi everyone! Our group is thinking of maintaining a subset of the msys2 repo to be used internally. I am wondering if there is documentation on the Arch Linux side on what tools you are currently using to automatically build the packages in the officially supported repos. I believe there must be some kind of dependency management as well, since official repos are never cursed by soname bump issues. Is there documentation on this as well? Specifically, is there a tool similar to apt-mirror? Thank you, and please don't hesistate to ask questions if I didn't make myself clear. Joey Dumont Scientific Computing Technical Advisor National Research Council Canada joey.dumont@nrc-cnrc.gc.ca<mailto:joey.dumont@nrc-cnrc.gc.ca>
Hi, Le 31/10/2018 à 20:05, Dumont, Joey a écrit :
Hi everyone!
Our group is thinking of maintaining a subset of the msys2 repo to be used internally. I am wondering if there is documentation on the Arch Linux side on what tools you are currently using to automatically build the packages in the officially supported repos.
Almost none, we mostly build by hand. We have http://rebuilds.foutrelis.com/ for huge rebuilds though. Automation of rebuilds is somewhere on the agenda (but we would prefer to have reproducible builds first I guess).
I believe there must be some kind of dependency management as well, since official repos are never cursed by soname bump issues.
When a package has a soname dump (something we can detect with checkpkg for instance), we generate a todo list (https://www.archlinux.org/todo/) of depending packages to rebuild (using a script called sogrep).
Is there documentation on this as well? Specifically, is there a tool similar to apt-mirror?
I don’t know the specifics about mirrors/repos, but I think it’s pretty easy to setup. Regards, Bruno
On Wed, Oct 31, 2018 at 08:31:58PM +0100, Bruno Pagani via arch-devops <arch-devops@lists.archlinux.org> wrote:
I believe there must be some kind of dependency management as well, since official repos are never cursed by soname bump issues.
When a package has a soname dump (something we can detect with checkpkg for instance), we generate a todo list (https://www.archlinux.org/todo/) of depending packages to rebuild (using a script called sogrep).
To clarify, you can find checkpkg here[1] and sogrep (which uses a database created by `createlinks`) is part of our ansible repository[2]. When you built a package, run checkpkg to check for changed files/libraries in the new package. If a soname changed, you'll get a message and then you can use sogrep to generate a list of packages that need to be rebuilt. We use a [staging] repository for this, but you can also just do all the rebuilds locally and then release the packages all at once. Just make sure you have the new versions of everything installed in the chroot when you build higher-level packages. Using a [staging] repository that is updated after each package build makes this easier since you can start with a fresh chroot each time and pacman will fetch the newest packages from [staging]. You can also check the built packages' .BUILDINFO file to verify that they were built with the correct dependency versions. [1] https://git.archlinux.org/devtools.git/ [2] https://git.archlinux.org/infrastructure.git/tree/roles/sogrep If you don't want to use createlinks and sogrep, you can also use makepkg's libprovides/libdepends support to track these dependencies directly in each package. Look in `man PKGBUILD` in the documentation of the provides/depends arrays for details on how to use it. I'm not sure if this works well when integrating with an external repo though so you might be better off with sogrep.
Is there documentation on this as well? Specifically, is there a tool similar to apt-mirror?
I'm not exactly sure what you are looking for, but we have a mirror script[3] which is linked on our mirror wiki page[4]. It's mostly just some fanciness around rsync to mirror our repo. I'm not sure how helpful that is with msys2. If you are looking for our PKGBUILDS, this page should get you started[5]. [3] https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh [4] https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors [5] https://wiki.archlinux.org/index.php/Arch_Build_System Florian
Hi Florian and Bruno, This helps a lot, thank you. This gives a pretty good idea of what can be done with existing Arch Linux tools. I'll check what msys2 has ported over to see if we can use the same pipeline on that side. As for documentation, I was simply wondering if there were pages on the DevWiki describing the details of maintaining an official repo, including generating the list of packages that need to be rebuilt. You answered my question! Thanks! -----Original Message----- From: Florian Pritz [mailto:bluewind@xinu.at] Sent: November 1, 2018 6:49 AM To: Arch Linux Infrastructure development discussion <arch-devops@lists.archlinux.org> Cc: Dumont, Joey <Joey.Dumont@nrc-cnrc.gc.ca>; Bruno Pagani <bruno.n.pagani@gmail.com> Subject: Re: [arch-devops] Build tools and repo management On Wed, Oct 31, 2018 at 08:31:58PM +0100, Bruno Pagani via arch-devops <arch-devops@lists.archlinux.org> wrote:
I believe there must be some kind of dependency management as well, since official repos are never cursed by soname bump issues.
When a package has a soname dump (something we can detect with checkpkg for instance), we generate a todo list (https://www.archlinux.org/todo/) of depending packages to rebuild (using a script called sogrep).
To clarify, you can find checkpkg here[1] and sogrep (which uses a database created by `createlinks`) is part of our ansible repository[2]. When you built a package, run checkpkg to check for changed files/libraries in the new package. If a soname changed, you'll get a message and then you can use sogrep to generate a list of packages that need to be rebuilt. We use a [staging] repository for this, but you can also just do all the rebuilds locally and then release the packages all at once. Just make sure you have the new versions of everything installed in the chroot when you build higher-level packages. Using a [staging] repository that is updated after each package build makes this easier since you can start with a fresh chroot each time and pacman will fetch the newest packages from [staging]. You can also check the built packages' .BUILDINFO file to verify that they were built with the correct dependency versions. [1] https://git.archlinux.org/devtools.git/ [2] https://git.archlinux.org/infrastructure.git/tree/roles/sogrep If you don't want to use createlinks and sogrep, you can also use makepkg's libprovides/libdepends support to track these dependencies directly in each package. Look in `man PKGBUILD` in the documentation of the provides/depends arrays for details on how to use it. I'm not sure if this works well when integrating with an external repo though so you might be better off with sogrep.
Is there documentation on this as well? Specifically, is there a tool similar to apt-mirror?
I'm not exactly sure what you are looking for, but we have a mirror script[3] which is linked on our mirror wiki page[4]. It's mostly just some fanciness around rsync to mirror our repo. I'm not sure how helpful that is with msys2. If you are looking for our PKGBUILDS, this page should get you started[5]. [3] https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh [4] https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors [5] https://wiki.archlinux.org/index.php/Arch_Build_System Florian
participants (3)
-
Bruno Pagani
-
Dumont, Joey
-
Florian Pritz