[arch-general] Updating a (local) repository based on dependency changes

[Erich Eckner]

On Wed, 23 Feb 2022, Maarten de Vries via arch-general wrote:

> On Wed, 23 Feb 2022 at 11:40, Knut Ahlers via arch-general <
> arch-general at lists.archlinux.org> wrote:
>
>> Hey there,
>>
>> Through the latest update in glibc an issue / question emerged for me
>> and maybe someone on this list has some input for me how to solve
>> this:
>>
>> We do have a repository with custom built packages (for example a
>> custom version of ffmpeg we need for internal purposes)… These
>> packages obviously do have dependencies on libraries and break when
>> there is a change in a library. In this case: "version `GLIBC_2.34'
>> not found".
>>
>> Is there anyone who solved this already? How is it solved in the
>> Archlinux core / community repo?
>>
>> What is your trigger to rebuild packages and prevent running into some
>> command not finding its dependencies? So how do you know when to
>> rebuild [some binary package depending on glibc] without manually
>> keeping track of the glibc version you've built the package against
>> and then triggering the re-build when any of the underlying library
>> changes its version?
>>
>> How does the Archlinux community/core/extra build system keep track of
>> this and knows when to rebuild ffmpeg because glibc changed?
>>
>> Thanks in advance for any input!
>>
>> Cheers,
>>   Knut
>>
>> --
>> Knut Ahlers
>> Software & Infrastructure Developer
>>
>> Web & Blog: https://ahlers.me/
>> GPG-Key: 0xD215AD7D
>>
>
> Hey,
>
> I'm not related to the Arch Linux project, but I manage a few custom
> repositories with additional packages for internal use. I've solved the
> problem you describe by mirroring the Arch Linux repositories, and building
> the custom packages against that mirror.
>
> Then, about once a week, I run a script which updates the mirror in a
> staging location. The script checks the .BUILDINFO for all custom packages.
> That gives the version number of dependencies that were installed when the
> custom package was built. Those versions are checked against the versions
> in the synced Arch Linux repositories. If any of the runtime dependencies
> is updated, the package and all its reverse dependencies are flagged to be
> rebuild. Then, all flagged packages are rebuild in dependency order in a
> clean container (which is configured to use the staging repository).
>
> When the build is done, the staging repository becomes the live mirror.
> This is done by overwriting a symlink to make it atomic. I also use btrfs
> snapshots to create a staging repository to avoid copying it every time for
> a rebuild.
>
> Note that this approach rebuilds more often than necessary. You could
> further inspect library dependencies of ELF files if you want to reduce the
> amount of rebuilds. However, I prefer to err on the safe side in case ABI
> breakage happened without matching soname bump, at the cost of doing more
> rebuilds than necessary.
>
> Maybe you can take some inspiration from this :)
>
> Kind regards,
> Maarten de Vries

Hi Knut,

I have a similar problem, which I solved a little different: I put as many 
versions as possible on my dependencies (i.e. depend on "foo=1.2.3" 
instead of just "foo") and use "libfoo.so=..." dependencies instead of 
"foo=..." wherever possible. This has the same disadvantage, that you may 
recompile more often than necessary, but it safes you from already 
installing broken packages (i.e. at latest, you notice during "pacman 
-Syu", that you should recompile - and not when you try to run your 
software). To automate the recompile, I use a similar approach as Maarten: 
I scrape the sync database of pacman, extract the provided versions and 
update them at the PKGBUILDs in my repo as necessary and recompile the 
packages. This runs much more often than weekly on my machine, though (I 
think, I set the timer to once an hour or so).

regards,
Erich