[aur-general] Continuous integration of AUR packages
Hi, I would like to run some kind of continuous integration of my AUR packages. The goal is to know when a package I maintain fails to build because either: 1/ its dependencies have been updated (new API, new incompatible version of GCC, ...) 2/ for -git packages, changes made upstream broke something (new dependency needed, new build system...) The scripts in devtools [1] look like they should work just fine to automate these kind of builds. After all, they are used to build the official Archlinux packages. However, I found that the build scripts do not really handle dependencies. When building a given package, they just install deps and makedeps from the existing Archlinux repositories. This is an issue when AUR packages depend on each other, because then dependencies cannot be installed from the Archlinux repositories... It *is* possible to manually pass packages to install in the chroot before building, but this is far from convenient. For instance, ring-daemon depends on opendht, so I would need to do this: cd opendht extra-x86_64-build cd ../ring-daemon extra-x86_64-build -- -I ../opendht/opendht-0.5.1-1-x86_64.pkg.tar.xz When there are multiple dependencies, it quickly becomes a nightmare to automate (especially because dependencies need to be passed in the right order). Is there any script that automates dependency handling when building? Basically, it would probably need to perform a topological sort, build packages in this order, and push them to a local repository to be able to build later packages. Or did I take the wrong approach entirely? Thanks, Baptiste [1] https://projects.archlinux.org/devtools.git/ PS : Some existing efforts I found about CI with Arch: - https://lists.archlinux.org/pipermail/arch-dev-public/2014-November/026757.h... https://jenkins.arch-ci.org/ No script provided, site appears to be down - http://openbuildservice.org/2012/09/10/arch-linux-support/ Source code is unreadable (enormous Perl scripts, no modularity)
I actually have a Jenkins setup for my packages that does almost exactly that, I have a job generator build that I give an AUR package name to and that's all the config for new packages, and it builds them in a docker container. You can obviously set up dependencies between the jobs to build in order, but that part isn't automatic yet. Its been pretty reliable for me over the past year or so since I last really changed anything. If you're interested, let me know and I can share the scripts and stuff. Just on my phone at the moment or I would link them here. Cheers, Justin On Thu, Feb 18, 2016, 9:51 PM Baptiste Jonglez <baptiste@bitsofnetworks.org> wrote:
Hi,
I would like to run some kind of continuous integration of my AUR packages. The goal is to know when a package I maintain fails to build because either:
1/ its dependencies have been updated (new API, new incompatible version of GCC, ...)
2/ for -git packages, changes made upstream broke something (new dependency needed, new build system...)
The scripts in devtools [1] look like they should work just fine to automate these kind of builds. After all, they are used to build the official Archlinux packages.
However, I found that the build scripts do not really handle dependencies. When building a given package, they just install deps and makedeps from the existing Archlinux repositories. This is an issue when AUR packages depend on each other, because then dependencies cannot be installed from the Archlinux repositories...
It *is* possible to manually pass packages to install in the chroot before building, but this is far from convenient. For instance, ring-daemon depends on opendht, so I would need to do this:
cd opendht extra-x86_64-build cd ../ring-daemon extra-x86_64-build -- -I ../opendht/opendht-0.5.1-1-x86_64.pkg.tar.xz
When there are multiple dependencies, it quickly becomes a nightmare to automate (especially because dependencies need to be passed in the right order).
Is there any script that automates dependency handling when building? Basically, it would probably need to perform a topological sort, build packages in this order, and push them to a local repository to be able to build later packages. Or did I take the wrong approach entirely?
Thanks, Baptiste
[1] https://projects.archlinux.org/devtools.git/
PS : Some existing efforts I found about CI with Arch:
- https://lists.archlinux.org/pipermail/arch-dev-public/2014-November/026757.h... https://jenkins.arch-ci.org/ No script provided, site appears to be down
- http://openbuildservice.org/2012/09/10/arch-linux-support/ Source code is unreadable (enormous Perl scripts, no modularity)
Hi, On Thu, Feb 18, 2016 at 12:25:53PM +0000, Justin Dray wrote:
I actually have a Jenkins setup for my packages that does almost exactly that, I have a job generator build that I give an AUR package name to and that's all the config for new packages, and it builds them in a docker container.
You can obviously set up dependencies between the jobs to build in order, but that part isn't automatic yet. Its been pretty reliable for me over the past year or so since I last really changed anything.
If you're interested, let me know and I can share the scripts and stuff. Just on my phone at the moment or I would link them here.
Yes, I would be interested! Out of curiosity, why do you use docker containers instead of the "official" method with chroots? https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroo...
Cheers, Justin
On Thu, Feb 18, 2016, 9:51 PM Baptiste Jonglez <baptiste@bitsofnetworks.org> wrote:
Hi,
I would like to run some kind of continuous integration of my AUR packages. The goal is to know when a package I maintain fails to build because either:
1/ its dependencies have been updated (new API, new incompatible version of GCC, ...)
2/ for -git packages, changes made upstream broke something (new dependency needed, new build system...)
The scripts in devtools [1] look like they should work just fine to automate these kind of builds. After all, they are used to build the official Archlinux packages.
However, I found that the build scripts do not really handle dependencies. When building a given package, they just install deps and makedeps from the existing Archlinux repositories. This is an issue when AUR packages depend on each other, because then dependencies cannot be installed from the Archlinux repositories...
It *is* possible to manually pass packages to install in the chroot before building, but this is far from convenient. For instance, ring-daemon depends on opendht, so I would need to do this:
cd opendht extra-x86_64-build cd ../ring-daemon extra-x86_64-build -- -I ../opendht/opendht-0.5.1-1-x86_64.pkg.tar.xz
When there are multiple dependencies, it quickly becomes a nightmare to automate (especially because dependencies need to be passed in the right order).
Is there any script that automates dependency handling when building? Basically, it would probably need to perform a topological sort, build packages in this order, and push them to a local repository to be able to build later packages. Or did I take the wrong approach entirely?
Thanks, Baptiste
[1] https://projects.archlinux.org/devtools.git/
PS : Some existing efforts I found about CI with Arch:
- https://lists.archlinux.org/pipermail/arch-dev-public/2014-November/026757.h... https://jenkins.arch-ci.org/ No script provided, site appears to be down
- http://openbuildservice.org/2012/09/10/arch-linux-support/ Source code is unreadable (enormous Perl scripts, no modularity)
I was using the chroot tools for a year and a half or so before that, but had some issues occasionally with things like updating it, or using package caches inside the chroot, or other various things. I do still have the shell script I used at the time, it was called by jenkins and cloned a chroot based off a clean copy and numbered by the jenkins executor to support multiple builds at once. But I managed a few tens of thousands of docker containers for work, and figured I could use a similar build system for the arch packages, and it's been reliable for me since then. As Doug said, you need to add the packages you build to a repository for the next build to be able to install them, but that part is pretty simple and explained on the arch wiki already. So, in Jenkins I use the build generator plugin, and created a parameterized job (config.xml here: http://hastebin.com/oyuqiwokem.xml) that basically just calls `dmakepkg` on the git checkout for the aur package; you just give the build an aur package name and it will generate the job to track that package and any changes pushed to the AUR and build it. (it polls git once an hour). `dmakepkg` is in the AUR ( https://aur.archlinux.org/packages/dmakepkg-git/ ) and is what I ended up making to build the packages inside a docker container. It supports the same flags as makepkg (just passes them through) but defaults to '--force --syncdeps --noconfirm' unless other flags are specified (no point removing packages since the container is destroyed after use). Uses a slightly modified version of the official docker arch image, and is updated much more often (monthly at least instead of the 3-6 months they take). It auto-detects makepkg configs like SRCDEST/etc and copies it in to the instance and bind mounts the folders specified, as well as the pacman package cache, so all the builds follow the makepkg config on the host, but using everything else from a clean environment. If you make your own repo and add it to your host, you can use `-x` flag to use the host's pacman.conf inside the container.
On Thu, 18 Feb 2016 12:51:19 +0100 Baptiste Jonglez <baptiste@bitsofnetworks.org> wrote:
Hi,
The scripts in devtools [1] look like they should work just fine to automate these kind of builds. After all, they are used to build the official Archlinux packages.
However, I found that the build scripts do not really handle dependencies. When building a given package, they just install deps and makedeps from the existing Archlinux repositories. This is an issue when AUR packages depend on each other, because then dependencies cannot be installed from the Archlinux repositories... ... Thanks, Baptiste
So create you own local repository and add the binary packages to it after they're built.
participants (3)
-
Baptiste Jonglez
-
Doug Newgard
-
Justin Dray