[arch-dev-public] Library dependencies
Hi all, I submitted a patchset to pacman that I would like some packager feed-back on. [1] Essentially this replaces the old libdepends/libprovides system into something akin to that used by APK. In short, makepkg.conf will have a variable like: LIB_DIRS=('lib:usr/lib' 'lib32:usr/lib32') At the end of package building, makepkg will look in the library directories and add a provide. E.g. for pacman: provide = lib:libalpm.so.13 Note the prefix matches the prefix given to the relevant directory in LIB_DIRS. Similarly, makepkg can add dependencies on libraries. E.g. pacman may have: depends = lib:libgpgme.so.11 Note, to help with bootstrapping this system, or if packages just do not want to add libprovides, the depends entries are only added if a package actually provides them. This is different to the APK system for libraries which uses "so" as the prefix and is not configurable. But Alpine used musl, which has no concept of multilib, so we need to be a bit more flexible. Note the "lib" and "lib32" prefixes are just for discussion. Arch can configure how they want. The dependency/provides additions can all be disabled in pacman.conf with the '!autodeps' option. Note that APK has similar things for binaries and pkg-config files. e.g. provides = cmd:pacman provides = pc:libalpm These can now be readily be added as dropins to libmakepkg. Any opinions on this would be greatly appreciated. Is this a better system than the current one? Is adding automatic dependencies against the spirit of makepkg where everything is in the PKGBUILD? Thanks, Allan [1] https://gitlab.archlinux.org/pacman/pacman/-/commits/allan/autodeps
On 2021-12-13 18:35 +1000 Allan McRae via arch-dev-public wrote:
Hi all,
I submitted a patchset to pacman that I would like some packager feed-back on. [1]
Essentially this replaces the old libdepends/libprovides system into something akin to that used by APK. In short, makepkg.conf will have a variable like:
LIB_DIRS=('lib:usr/lib' 'lib32:usr/lib32')
At the end of package building, makepkg will look in the library directories and add a provide. E.g. for pacman:
provide = lib:libalpm.so.13
Note the prefix matches the prefix given to the relevant directory in LIB_DIRS. Similarly, makepkg can add dependencies on libraries. E.g. pacman may have:
depends = lib:libgpgme.so.11
Note, to help with bootstrapping this system, or if packages just do not want to add libprovides, the depends entries are only added if a package actually provides them.
This is different to the APK system for libraries which uses "so" as the prefix and is not configurable. But Alpine used musl, which has no concept of multilib, so we need to be a bit more flexible. Note the "lib" and "lib32" prefixes are just for discussion. Arch can configure how they want.
The dependency/provides additions can all be disabled in pacman.conf with the '!autodeps' option.
Note that APK has similar things for binaries and pkg-config files. e.g. provides = cmd:pacman provides = pc:libalpm
These can now be readily be added as dropins to libmakepkg.
Any opinions on this would be greatly appreciated. Is this a better system than the current one? Is adding automatic dependencies against the spirit of makepkg where everything is in the PKGBUILD?
Thanks, Allan
[1] https://gitlab.archlinux.org/pacman/pacman/-/commits/allan/autodeps
Is it necessary to hard-code LIB_DIRS with prefixes in each PKGBUILD that provides them? It seems simpler to me to use the standard library paths for the system with optional overrides in makepkg.conf. Makepkg could then scan those directories in the pkgdir, check the architecture of any shared objects and automatically add the prefixed provides (with the prefixes also configurable in makepkg.conf). Basically, any shared objects installed to system paths are effectively provided by the package anyway so they may as well all be included in the array so that packages can explicitly depend on them as packagers please. Any shared objects installed elsewhere are effectively invisible except for packages that specifically look for them in non-standard locations, but then it makes more sense to have a direct dependency on the package itself, with a version specification if necessary. On the depends side, the shared objects used by the package can also be scanned but it's obviously not as straight-forward to determine whether dependencies are general, versioned or even optional. A tool could be used to generate a list or prompt the package interactively to select relevant so deps to add to the package. Guidelines could be provided by the tool itself to avoid overspecification. It would be a shame if this led to a permanent dependency hell of packages depending explicitly on old versions even when not necessary. As for extending this to other dependency types such as commands, I wonder if cmd:name would be specific enough. It's rare but sometimes unrelated commands can have the same name. Some sort of unique identifier may be required. I only mention it in case it should be considered for generalizing the syntax now before settling on a final format. Possibly something like "prefix:identifier/object", where "identifer/" is optional. So you would have "cmd:unique_cmd" for something unique but "cmd:foo/common_cmd" for some generic fungible common_cmd provided by different packages when a conflicting common_cmd exists in another package. How would this syntax work for optional deps btw? Also, if this is added, it would be useful to have an option to display the provider package of such deps in the output of pacman -Qi (e.g. -Qii).
On 15/12/21 14:11, Xyne via arch-dev-public wrote:
On 2021-12-13 18:35 +1000 Allan McRae via arch-dev-public wrote:
Hi all,
I submitted a patchset to pacman that I would like some packager feed-back on. [1]
Essentially this replaces the old libdepends/libprovides system into something akin to that used by APK. In short, makepkg.conf will have a variable like:
LIB_DIRS=('lib:usr/lib' 'lib32:usr/lib32')
At the end of package building, makepkg will look in the library directories and add a provide. E.g. for pacman:
provide = lib:libalpm.so.13
Note the prefix matches the prefix given to the relevant directory in LIB_DIRS. Similarly, makepkg can add dependencies on libraries. E.g. pacman may have:
depends = lib:libgpgme.so.11
Note, to help with bootstrapping this system, or if packages just do not want to add libprovides, the depends entries are only added if a package actually provides them.
This is different to the APK system for libraries which uses "so" as the prefix and is not configurable. But Alpine used musl, which has no concept of multilib, so we need to be a bit more flexible. Note the "lib" and "lib32" prefixes are just for discussion. Arch can configure how they want.
The dependency/provides additions can all be disabled in pacman.conf with the '!autodeps' option.
Note that APK has similar things for binaries and pkg-config files. e.g. provides = cmd:pacman provides = pc:libalpm
These can now be readily be added as dropins to libmakepkg.
Any opinions on this would be greatly appreciated. Is this a better system than the current one? Is adding automatic dependencies against the spirit of makepkg where everything is in the PKGBUILD?
Thanks, Allan
[1] https://gitlab.archlinux.org/pacman/pacman/-/commits/allan/autodeps
Is it necessary to hard-code LIB_DIRS with prefixes in each PKGBUILD that provides them? It seems simpler to me to use the standard library paths for the system with optional overrides in makepkg.conf. Makepkg could then scan those directories in the pkgdir, check the architecture of any shared objects and automatically add the prefixed provides (with the prefixes also configurable in makepkg.conf).
LIB_DIRS is specified in makepkg.conf, not in PKGBUILDs. Given usr/lib/ is not even standard for 64bit libraries, I do not want to hard code anything.
Basically, any shared objects installed to system paths are effectively provided by the package anyway so they may as well all be included in the array so that packages can explicitly depend on them as packagers please. Any shared objects installed elsewhere are effectively invisible except for packages that specifically look for them in non-standard locations, but then it makes more sense to have a direct dependency on the package itself, with a version specification if necessary.
Non-standard paths for libraries was mentioned on pacman-dev. For these, LIB_DIRS can be added to in the PKGBUILD.
On the depends side, the shared objects used by the package can also be scanned but it's obviously not as straight-forward to determine whether dependencies are general, versioned or even optional. A tool could be used to generate a list or prompt the package interactively to select relevant so deps to add to the package. Guidelines could be provided by the tool itself to avoid overspecification. It would be a shame if this led to a permanent dependency hell of packages depending explicitly on old versions even when not necessary.
The dependencies added are purely sonames that the binary are explicitly linked to. So the binary will be non-function without libraries providing that exact soname. Thus all these dependencies are necessary. Of course it will be up to the distribution to decide how much they use this feature - should all libraries provide their lib:soname value or just some? Dependencies are only added if there is a relevant provide.
As for extending this to other dependency types such as commands, I wonder if cmd:name would be specific enough. It's rare but sometimes unrelated commands can have the same name. Some sort of unique identifier may be required. I only mention it in case it should be considered for generalizing the syntax now before settling on a final format. Possibly something like "prefix:identifier/object", where "identifer/" is optional. So you would have "cmd:unique_cmd" for something unique but "cmd:foo/common_cmd" for some generic fungible common_cmd provided by different packages when a conflicting common_cmd exists in another package.
I don't see why we can not have multiple packages provide the same command. We already have multiple packages with the same provides entry, just with a package name and not a command name.
How would this syntax work for optional deps btw? Also, if this is added, it would be useful to have an option to display the provider package of such deps in the output of pacman -Qi (e.g. -Qii).
People can manually add such things as optional dependencies. I will need to look and see if pacman recognises provides as being installed (I think it does...). Cheers, Allan
On 2021-12-15 16:40 +1000 Allan McRae via arch-dev-public wrote:
LIB_DIRS is specified in makepkg.conf, not in PKGBUILDs. Given usr/lib/ is not even standard for 64bit libraries, I do not want to hard code anything.
I somehow completely missed the part where you explicitly said it would be in makepkg.conf. That's exactly what I was suggesting precisely for the sake of having a centralized yet configurable setting. Sorry for the noise.
The dependencies added are purely sonames that the binary are explicitly linked to. So the binary will be non-function without libraries providing that exact soname. Thus all these dependencies are necessary.
Of course it will be up to the distribution to decide how much they use this feature - should all libraries provide their lib:soname value or just some? Dependencies are only added if there is a relevant provide.
What happens if a package includes "optional" binaries that depend on optdeps? Do those become hard deps?
As for extending this to other dependency types such as commands, I wonder if cmd:name would be specific enough. It's rare but sometimes unrelated commands can have the same name. Some sort of unique identifier may be required. I only mention it in case it should be considered for generalizing the syntax now before settling on a final format. Possibly something like "prefix:identifier/object", where "identifer/" is optional. So you would have "cmd:unique_cmd" for something unique but "cmd:foo/common_cmd" for some generic fungible common_cmd provided by different packages when a conflicting common_cmd exists in another package.
I don't see why we can not have multiple packages provide the same command. We already have multiple packages with the same provides entry, just with a package name and not a command name.
You can have multiple packages that provide the same command, but there may be rare cases where two conflicting packages provide unrelated commands with the same name, or a restricted version of a command that may not support the full argument set. It's worth considering how to handle such cases now before settling on a syntax.
How would this syntax work for optional deps btw? Also, if this is added, it would be useful to have an option to display the provider package of such deps in the output of pacman -Qi (e.g. -Qii).
For optdeps, what I mean is if the normal dependency would be "lib:libgpgme.so.11", how will you parse the normal optdep syntax of "pkgname: reason"? "lib:libfoo.so.13: required for the command foo". Won't using the same delimiter in two different contexts be problematic?
People can manually add such things as optional dependencies. I will need to look and see if pacman recognises provides as being installed (I think it does...).
I assumed that it did for dependency resolution. Regards, Xyne
On 16/12/21 13:24, Xyne via arch-dev-public wrote:
On 2021-12-15 16:40 +1000 Allan McRae via arch-dev-public wrote:
The dependencies added are purely sonames that the binary are explicitly linked to. So the binary will be non-function without libraries providing that exact soname. Thus all these dependencies are necessary.
Of course it will be up to the distribution to decide how much they use this feature - should all libraries provide their lib:soname value or just some? Dependencies are only added if there is a relevant provide.
What happens if a package includes "optional" binaries that depend on optdeps? Do those become hard deps?
Assuming that dependent library is not used elsewhere in the package, and the extra library had a provide of its library version, then this would add an extra dependency. There are several options: 1) disable autodeps - these really do not need used everywhere... 2) split the package 3) move the binary into /usr/lib/<pkgname> and add a symlink to /usr/bin. Then (assuming BIN_DIR=usr/bin is the usual search path), the dependency would not be added. Saying that, I am against optional dependencies that are genuinely needed for a binary to run. I think these should be used for features that could be dynamically loaded if the optional dependency is present. I prefer package splitting if that is not the case.
As for extending this to other dependency types such as commands, I wonder if cmd:name would be specific enough. It's rare but sometimes unrelated commands can have the same name. Some sort of unique identifier may be required. I only mention it in case it should be considered for generalizing the syntax now before settling on a final format. Possibly something like "prefix:identifier/object", where "identifer/" is optional. So you would have "cmd:unique_cmd" for something unique but "cmd:foo/common_cmd" for some generic fungible common_cmd provided by different packages when a conflicting common_cmd exists in another package.
I don't see why we can not have multiple packages provide the same command. We already have multiple packages with the same provides entry, just with a package name and not a command name.
You can have multiple packages that provide the same command, but there may be rare cases where two conflicting packages provide unrelated commands with the same name, or a restricted version of a command that may not support the full argument set. It's worth considering how to handle such cases now before settling on a syntax.
Do you have an example? I don't like adding complexity for "what if" cases that may never occur. For the case you described, cmd:foo is provided by two packages, foo1 and foo2. foo2 has a subset of the functionality. Then you could depend on cmd:foo if either works, or foo1 if you need the full functionality.
How would this syntax work for optional deps btw? Also, if this is added, it would be useful to have an option to display the provider package of such deps in the output of pacman -Qi (e.g. -Qii).
For optdeps, what I mean is if the normal dependency would be "lib:libgpgme.so.11", how will you parse the normal optdep syntax of "pkgname: reason"? "lib:libfoo.so.13: required for the command foo". Won't using the same delimiter in two different contexts be problematic?
From memory, the space in "<pkgname>: <reason>" is important for optdepends. I need to check, but I don't think the PKGBUILD linter will let PKGBUILDs with optdepends without the space build. And pacman will not split the string without it. So this should be fine. Allan
On 2021-12-16 19:53 +1000 Allan McRae via arch-dev-public wrote:
Assuming that dependent library is not used elsewhere in the package, and the extra library had a provide of its library version, then this would add an extra dependency.
There are several options: 1) disable autodeps - these really do not need used everywhere... 2) split the package 3) move the binary into /usr/lib/<pkgname> and add a symlink to /usr/bin. Then (assuming BIN_DIR=usr/bin is the usual search path), the dependency would not be added.
Saying that, I am against optional dependencies that are genuinely needed for a binary to run. I think these should be used for features that could be dynamically loaded if the optional dependency is present. I prefer package splitting if that is not the case.
I thought it was a supported use case but I agree with you that it's better to split.
You can have multiple packages that provide the same command, but there may be rare cases where two conflicting packages provide unrelated commands with the same name, or a restricted version of a command that may not support the full argument set. It's worth considering how to handle such cases now before settling on a syntax.
Do you have an example? I don't like adding complexity for "what if" cases that may never occur.
Nope, only a vague memory of some package conflict several years ago with two identically named commands that did completely different things. I think it was eventually solved by renaming one and telling upstream that the name was already used for a common executable. So yeah, it's not really an issue.
For the case you described, cmd:foo is provided by two packages, foo1 and foo2. foo2 has a subset of the functionality. Then you could depend on cmd:foo if either works, or foo1 if you need the full functionality.
Ok
For optdeps, what I mean is if the normal dependency would be "lib:libgpgme.so.11", how will you parse the normal optdep syntax of "pkgname: reason"? "lib:libfoo.so.13: required for the command foo". Won't using the same delimiter in two different contexts be problematic?
From memory, the space in "<pkgname>: <reason>" is important for optdepends. I need to check, but I don't think the PKGBUILD linter will let PKGBUILDs with optdepends without the space build. And pacman will not split the string without it. So this should be fine.
I wasn't sure that the space was enforced. If it is, then there's no issue. Coming back to your initial question:
Any opinions on this would be greatly appreciated. Is this a better system than the current one? Is adding automatic dependencies against the spirit of makepkg where everything is in the PKGBUILD?
It seems better to me. Less tedious and error-prone. Regards, Xyne
Hi, On 17/12/2021 04:42, Xyne via arch-dev-public wrote:
On 2021-12-16 19:53 +1000 Allan McRae via arch-dev-public wrote:
Assuming that dependent library is not used elsewhere in the package, and the extra library had a provide of its library version, then this would add an extra dependency.
There are several options: 1) disable autodeps - these really do not need used everywhere... 2) split the package 3) move the binary into /usr/lib/<pkgname> and add a symlink to /usr/bin. Then (assuming BIN_DIR=usr/bin is the usual search path), the dependency would not be added.
Saying that, I am against optional dependencies that are genuinely needed for a binary to run. I think these should be used for features that could be dynamically loaded if the optional dependency is present. I prefer package splitting if that is not the case.
I thought it was a supported use case but I agree with you that it's better to split.
You can have multiple packages that provide the same command, but there may be rare cases where two conflicting packages provide unrelated commands with the same name, or a restricted version of a command that may not support the full argument set. It's worth considering how to handle such cases now before settling on a syntax.
Do you have an example? I don't like adding complexity for "what if" cases that may never occur.
Nope, only a vague memory of some package conflict several years ago with two identically named commands that did completely different things. I think it was eventually solved by renaming one and telling upstream that the name was already used for a common executable. So yeah, it's not really an issue.
The example that comes to mind is docker[1]. This existed already (I also think we had it in the repos) as a system tray tool / dock app[2]. The issue then was resolved by changing the tray app's name and binary to docker-tray[3] and let the container tool docker have the docker name. There have been discussions[4] about the name then. But in general we can solve this kind of problems as was done then, by giving the package/binary a suffix in the name.
For the case you described, cmd:foo is provided by two packages, foo1 and foo2. foo2 has a subset of the functionality. Then you could depend on cmd:foo if either works, or foo1 if you need the full functionality.
Ok
For optdeps, what I mean is if the normal dependency would be "lib:libgpgme.so.11", how will you parse the normal optdep syntax of "pkgname: reason"? "lib:libfoo.so.13: required for the command foo". Won't using the same delimiter in two different contexts be problematic?
From memory, the space in "<pkgname>: <reason>" is important for optdepends. I need to check, but I don't think the PKGBUILD linter will let PKGBUILDs with optdepends without the space build. And pacman will not split the string without it. So this should be fine.
I wasn't sure that the space was enforced. If it is, then there's no issue.
Coming back to your initial question:
Any opinions on this would be greatly appreciated. Is this a better system than the current one? Is adding automatic dependencies against the spirit of makepkg where everything is in the PKGBUILD?
It seems better to me. Less tedious and error-prone.
Regards, Xyne
[1] https://archlinux.org/packages/community/x86_64/docker/ [2] https://aur.archlinux.org/packages/docker-tray/ [3] https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=docker-tray#n24 [4] https://lists.archlinux.org/pipermail/aur-general/2013-November/026213.html
participants (3)
-
Allan McRae
-
Ike Devolder
-
Xyne