[pacman-dev] Alternatives system brainstorm
Hi all, Some further brainstorming on alternatives system. Last time, I suggested something like: alternatives=(python->python3) But as was pointed out, there may be many other things you want to bring along with the python binary. Some directly related: /usr/share/man/man1/python3.1.gz -> python.1.gz Some less directly related: /usr/bin/idle /usr/bin/pydoc /usr/bin/python-config Another example would be lua, where you want to set the main interpreter and the header directory. In Debian & Fedora land, these additional files are called slaves. Note that both those distros use the "same" tool to manage these things. In addition, each alternative is given a priority. So if (e.g.) two python2 and python3 are installed at the same time, it would select the one with the higher priority. Fedora also has "ghost" files (not sure about Debian). I.e. the python3 package contains a ghost /usr/bin/python, so searching file databases gives a match. Now how does that inform our potential implementation? Mostly, I don't like that they set up all the alternatives in the post install with one long command passed to their tool. I'd prefer we had a file with a list of needed symlinks so this could happen automatically. So here is a starting point for discussion: In the PKGBUILD: alternatives=('python::python3') that would look for a file in our build root called "python3.alternatives" and include it in the package root. Note we could have more than one such entry. This file could look like: /usr/bin/python3 -> python /usr/bin/idle3 -> idle /usr/include/example3/ -> example/ Could also include an optional priority statement? Or should that be captured in the PKGBUILD? The file would be placed at /var/lib/pacman/alternatives/python/python3. Any comments on that as a starting point. Cheers, Allan
Em outubro 17, 2019 7:53 Allan McRae escreveu:
In Debian & Fedora land, these additional files are called slaves. Note that both those distros use the "same" tool to manage these things. In addition, each alternative is given a priority. So if (e.g.) two python2 and python3 are installed at the same time, it would select the one with the higher priority. Fedora also has "ghost" files (not sure about Debian). I.e. the python3 package contains a ghost /usr/bin/python, so searching file databases gives a match.
Priorities are a nice thing to have. I think I've reported this on the bugtracker (or there was a bug report I've commented in). The example is mkinitcpio/dracut. The only reason mkinitcpio is preferred over dracut right now is because it is on the core repo and dracut is in the extra repo. Being able to define priorities will benefit not just the alternatives system. Then again, how will repo priority work? Will it be completely removed?
Now how does that inform our potential implementation? Mostly, I don't like that they set up all the alternatives in the post install with one long command passed to their tool. I'd prefer we had a file with a list of needed symlinks so this could happen automatically.
I agree, we shouldn't do this on post install. Ideally pacman itself should handle all the filesystem operations.
Could also include an optional priority statement? Or should that be captured in the PKGBUILD?
I think the priorities should be captured at the PKGBUILD itself. It would also make it easier for pacman to know what they are. Regards, Giancarlo Razzolini
On 10/17/19 9:32 AM, Giancarlo Razzolini wrote:
Em outubro 17, 2019 7:53 Allan McRae escreveu:
In Debian & Fedora land, these additional files are called slaves. Note that both those distros use the "same" tool to manage these things. In addition, each alternative is given a priority. So if (e.g.) two python2 and python3 are installed at the same time, it would select the one with the higher priority. Fedora also has "ghost" files (not sure about Debian). I.e. the python3 package contains a ghost /usr/bin/python, so searching file databases gives a match.
Priorities are a nice thing to have. I think I've reported this on the bugtracker (or there was a bug report I've commented in). The example is mkinitcpio/dracut.
The only reason mkinitcpio is preferred over dracut right now is because it is on the core repo and dracut is in the extra repo. Being able to define priorities will benefit not just the alternatives system.
Then again, how will repo priority work? Will it be completely removed?
This is a completely different topic, you want "priorities" of one package over another as the default selector for interactive resolution of multiple dependency providers whereas this discussion is about whether two packages which are both installed in the same: pacman -S foo bar should prioritize one over the other when installing alternatives, and, optionally, whether installing bar *after* foo, should update the alternative that was already installed, if bar is "more priority" than foo. So the case of dracut/mkinitcpio is solved by simply declining to use --noconfirm (or alternatively, using pacstrap -i) and relegating non-preferred alternatives for core pacstrap software to [extra], which is the status quo. I don't think we've ever really made strong guarantees about how useful --noconfirm is, or blindly pressing "y" to prompts? There's a stronger argument to be made for alternatives, because ideally the "best" version of a command should always be preferred. e.g. if you install python2 as a dependency for some program, then by default /usr/bin/python -> python2 Once you install python3, one could argue it is preferred to automatically switch to the "best" version. (The counter-argument could be, maintain the status quo, first installed will win.) Now that I think about this more, we maybe don't need priorities at all, only a "default" key.
Now how does that inform our potential implementation? Mostly, I don't like that they set up all the alternatives in the post install with one long command passed to their tool. I'd prefer we had a file with a list of needed symlinks so this could happen automatically.
I agree, we shouldn't do this on post install. Ideally pacman itself should handle all the filesystem operations.
Could also include an optional priority statement? Or should that be captured in the PKGBUILD?
I think the priorities should be captured at the PKGBUILD itself. It would also make it easier for pacman to know what they are.
If priorities are implemented they should be wherever the actual alternatives are, IMO. Less fragmentation. If pacman already needs to read an "alternatives" database to know what the alternatives are to begin with, we don't need a priority without the thing it describes, encoded in the .PKGINFO on its own. ... But we should anyway focus on what the UI should be (priorities are UI) before focusing on how we implement the tracking database. :) -- Eli Schwartz Bug Wrangler and Trusted User
Em outubro 17, 2019 12:41 Eli Schwartz escreveu:
This is a completely different topic, you want "priorities" of one package over another as the default selector for interactive resolution of multiple dependency providers whereas this discussion is about whether two packages which are both installed in the same:
pacman -S foo bar
should prioritize one over the other when installing alternatives, and, optionally, whether installing bar *after* foo, should update the alternative that was already installed, if bar is "more priority" than foo.
I think priorities are complimentary to alternatives. Alternatives operate at the fs level, and we're talking about operating too at the package level. We can have alternatives without priorities, sure, and I also agree that they are not the same. You can have packages that don't have any fs level conflict at all, but they could provide the same thing and therefore you could at a meta level, prioritize them.
So the case of dracut/mkinitcpio is solved by simply declining to use --noconfirm (or alternatively, using pacstrap -i) and relegating non-preferred alternatives for core pacstrap software to [extra], which is the status quo. I don't think we've ever really made strong guarantees about how useful --noconfirm is, or blindly pressing "y" to prompts?
Repo precedence is ok for this, I agree.
If priorities are implemented they should be wherever the actual alternatives are, IMO. Less fragmentation. If pacman already needs to read an "alternatives" database to know what the alternatives are to begin with, we don't need a priority without the thing it describes, encoded in the .PKGINFO on its own.
You're mixing the priority with alternatives, and as I've stated above, they are complimentary, but could certainly be achieved each on its own.
But we should anyway focus on what the UI should be (priorities are UI) before focusing on how we implement the tracking database. :)
Agreed. But I want to make the point that if we ever implement priorities, they would better serve their purpose if implemented at the packaging level. Regards, Giancarlo Razzolini
On 18/10/19 5:04 am, Giancarlo Razzolini wrote:
Em outubro 17, 2019 12:41 Eli Schwartz escreveu:
If priorities are implemented they should be wherever the actual alternatives are, IMO. Less fragmentation. If pacman already needs to read an "alternatives" database to know what the alternatives are to begin with, we don't need a priority without the thing it describes, encoded in the .PKGINFO on its own.
You're mixing the priority with alternatives, and as I've stated above, they are complimentary, but could certainly be achieved each on its own.
No - this conversation is entire about the alternative system. We are talking about priorities within the alternative system (as implemented in other distros). E.g. when installing python3 and python2 in the same transaction, which one should be symlinked to /usr/bin/python? I'm thinking first one installed wins. It also avoids decisions about whether pacman should automatically update alternatives to a higher priority package if it gets installed later. Allan
Em outubro 17, 2019 22:06 Allan McRae escreveu:
No - this conversation is entire about the alternative system. We are talking about priorities within the alternative system (as implemented in other distros).
E.g. when installing python3 and python2 in the same transaction, which one should be symlinked to /usr/bin/python?
Sure, I understood that. But I'm talking about actually implementing a priority system in pacman.
I'm thinking first one installed wins. It also avoids decisions about whether pacman should automatically update alternatives to a higher priority package if it gets installed later.
As long pacman has an easy way to set the priorities, I'm fine with this behavior. Regards, Giancarlo Razzolini
On 10/17/19 10:10 PM, Giancarlo Razzolini wrote:
Em outubro 17, 2019 22:06 Allan McRae escreveu:
No - this conversation is entire about the alternative system. We are talking about priorities within the alternative system (as implemented in other distros).
E.g. when installing python3 and python2 in the same transaction, which one should be symlinked to /usr/bin/python?
Sure, I understood that. But I'm talking about actually implementing a priority system in pacman.
... but the thread topic is "alternatives system", so if you want to discuss "priority system" then for the sake of avoiding confusion you should probably start a new thread, in which case I'm sure we'd all be delighted to discuss said other topic.
I'm thinking first one installed wins. It also avoids decisions about whether pacman should automatically update alternatives to a higher priority package if it gets installed later.
As long pacman has an easy way to set the priorities, I'm fine with this behavior.
"This behavior" is "do not implement any sort of alternatives priorities at all, do not pass GO, do not collect $200". Naturally, pacman will have a way to manually select an alternatives provider, but there will be no form of "prioritizing" going on... I'm not sure whether by "easy way to set priorities" you actually mean "easy way to set the final resolved provider". -- Eli Schwartz Bug Wrangler and Trusted User
On 18.10.19 04:44, Eli Schwartz wrote:
I'm thinking first one installed wins. It also avoids decisions about whether pacman should automatically update alternatives to a higher priority package if it gets installed later.
As long pacman has an easy way to set the priorities, I'm fine with this behavior.
"This behavior" is "do not implement any sort of alternatives priorities at all, do not pass GO, do not collect $200".
Naturally, pacman will have a way to manually select an alternatives provider, but there will be no form of "prioritizing" going on... I'm not sure whether by "easy way to set priorities" you actually mean "easy way to set the final resolved provider".
Well, since the title says "brainstorm", I'll just throw in my thoughts about this: "first wins" feels more "KISS" to me, it's logic is less complex and I think it has the least "surprise potential". It would not break any assumptions about for example /usr/bin/python the user(s) of the system may have made at some point before someone installs an additional "provider" for /usr/bin/python on that system. (Though this kind of assuming is admittedly something one should probably not do? At least with python... other cases may be different.) If the auto-changing the alternative via priority is clearly communicated to the user, that "surprise" would be mitigated somewhat though. But that does not help when there's more than one user on a system and the change was not communicated to all users by the one who installs packages. On the other hand I'm not sure if that's something pacman should care about. I'm leaning towards "That's the sysadmin's job", but others may think otherwise. For me and my use cases it does not matter much if there's some automatic thing or "first wins". As long as pacman informs me about any change it makes and not silently changes an alternative, I'll be happy. The only thing I'd feel strongly about some automatic choosing is, that I think a manual selection should not be changed automagically. For example, if I have python2 and python3 installed, then manually set python to python2 and after that install python4 at some point, python should stay at python2 no matter how high python4's priority is because python2 was selected manually. -- regards, brainpower
* on Saturday, 2019-10-19 02:30 +0200, brainpower <brainpower@mailbox.org> wrote:
"first wins" feels more "KISS" to me, it's logic is less complex and I think it has the least "surprise potential". <snip>
The only thing I'd feel strongly about some automatic choosing is, that I think a manual selection should not be changed automagically. <snip>
It seems even more KISS to avoid managed alternatives altogether and let the user handle them. python2 and python3 are two different languages and any python script should be clear about what language they are written in. A shebang of /usr/bin/python is a bug, in my view, that the OS should not try to "fix". Currently, we offer a "python" package and a "python2" package. Similarly for lua, we offer packages "lua", "lua52", and "lua51". The way this behaves is clear and simple. I probably am overlooking a use case where managed alternatives are important. Can someone give a good example to help the brainstorm? Best, Daan van Rossum
On 19/10/19 5:56 pm, Daan van Rossum wrote:
* on Saturday, 2019-10-19 02:30 +0200, brainpower <brainpower@mailbox.org> wrote:
"first wins" feels more "KISS" to me, it's logic is less complex and I think it has the least "surprise potential". <snip>
The only thing I'd feel strongly about some automatic choosing is, that I think a manual selection should not be changed automagically. <snip>
It seems even more KISS to avoid managed alternatives altogether and let the user handle them.
python2 and python3 are two different languages and any python script should be clear about what language they are written in. A shebang of /usr/bin/python is a bug, in my view, that the OS should not try to "fix".
Currently, we offer a "python" package and a "python2" package. Similarly for lua, we offer packages "lua", "lua52", and "lua51". The way this behaves is clear and simple.
Yes - scripts that are python version dependent should specify what they need. The setting of /usr/bin/python is more for interactive use.
I probably am overlooking a use case where managed alternatives are important. Can someone give a good example to help the brainstorm?
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang ...
* on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang
Can we provide alternatives by means of sets of mutually exclusive link packages, using package properties that are already supported in pacman? sh-bash provides=('sh') depends=('bash') conflicts=('sh-dash') sh-dash provides=('sh') depends=('dash') conflicts=('sh-bash') bash depends('sh') dash depends('sh') The circular dependency can only be satisfied at install time if bash is installed together with sh-bash, or if another sh provider already exists in the system (e.g. dash and sh-dash). Daan
On 19/10/19 9:57 pm, Daan van Rossum wrote:
* on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang
Can we provide alternatives by means of sets of mutually exclusive link packages, using package properties that are already supported in pacman?
sh-bash provides=('sh') depends=('bash') conflicts=('sh-dash') sh-dash provides=('sh') depends=('dash') conflicts=('sh-bash') bash depends('sh') dash depends('sh')
Sure - or we could have a system to handle it in a less obtuse manner... What happens when a new package wants to provide "sh"? We need to rebuild sh-bash and sh-dash? Easy enough, except that assumes that all three packages are packaged by the same group. If bash and dash are packaged by a distro and the new package by a third party, then the conflicts provided by the distro are not enough. But an alternatives system would work.
On 10/19/19 at 10:15pm, Allan McRae wrote:
On 19/10/19 9:57 pm, Daan van Rossum wrote:
* on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang
Can we provide alternatives by means of sets of mutually exclusive link packages, using package properties that are already supported in pacman?
sh-bash provides=('sh') depends=('bash') conflicts=('sh-dash') sh-dash provides=('sh') depends=('dash') conflicts=('sh-bash') bash depends('sh') dash depends('sh')
Sure - or we could have a system to handle it in a less obtuse manner...
What happens when a new package wants to provide "sh"? We need to rebuild sh-bash and sh-dash? Easy enough, except that assumes that all three packages are packaged by the same group. If bash and dash are packaged by a distro and the new package by a third party, then the conflicts provided by the distro are not enough. But an alternatives system would work.
No rebuilding necessary if the conflicts are changed to just 'sh'.
* on Saturday, 2019-10-19 05:24 -0700, Andrew Gregory <andrew.gregory.8@gmail.com> wrote:
No rebuilding necessary if the conflicts are changed to just 'sh'.
A circular dependency may not even be necessary in many cases, or ever. All the circular dependency does is force the user to install as provider (by requiring a link package) the first time one of the alternatives is installed. But a user may not want or need to install a provider until other packages depend on it. Then the solution would simplify to just a set of mutually exclusive link packages: sh-bash provides=('sh') depends=('bash') conflicts=('sh') sh-dash provides=('sh') depends=('dash') conflicts=('sh') And pacman luckily finds alternative providers very easily: # pacman -Ss '\bawk\b' core/gawk 5.0.1-1 (base base-devel) [installed] GNU version of awk extra/nawk 20180827-1 The one, true implementation of AWK
On 19/10/19 10:24 pm, Andrew Gregory wrote:
On 10/19/19 at 10:15pm, Allan McRae wrote:
On 19/10/19 9:57 pm, Daan van Rossum wrote:
* on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang
Can we provide alternatives by means of sets of mutually exclusive link packages, using package properties that are already supported in pacman?
sh-bash provides=('sh') depends=('bash') conflicts=('sh-dash') sh-dash provides=('sh') depends=('dash') conflicts=('sh-bash') bash depends('sh') dash depends('sh')
Sure - or we could have a system to handle it in a less obtuse manner...
What happens when a new package wants to provide "sh"? We need to rebuild sh-bash and sh-dash? Easy enough, except that assumes that all three packages are packaged by the same group. If bash and dash are packaged by a distro and the new package by a third party, then the conflicts provided by the distro are not enough. But an alternatives system would work.
No rebuilding necessary if the conflicts are changed to just 'sh'. .
What will happen with "pacman -S dash"? My initial thought is dash will pull in the first provider of sh - sh-bash - which will then pull in bash. A
On 19/10/19 10:38 pm, Allan McRae wrote:
On 19/10/19 10:24 pm, Andrew Gregory wrote:
On 10/19/19 at 10:15pm, Allan McRae wrote:
On 19/10/19 9:57 pm, Daan van Rossum wrote:
* on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang
Can we provide alternatives by means of sets of mutually exclusive link packages, using package properties that are already supported in pacman?
sh-bash provides=('sh') depends=('bash') conflicts=('sh-dash') sh-dash provides=('sh') depends=('dash') conflicts=('sh-bash') bash depends('sh') dash depends('sh')
Sure - or we could have a system to handle it in a less obtuse manner...
What happens when a new package wants to provide "sh"? We need to rebuild sh-bash and sh-dash? Easy enough, except that assumes that all three packages are packaged by the same group. If bash and dash are packaged by a distro and the new package by a third party, then the conflicts provided by the distro are not enough. But an alternatives system would work.
No rebuilding necessary if the conflicts are changed to just 'sh'. .
What will happen with "pacman -S dash"? My initial thought is dash will pull in the first provider of sh - sh-bash - which will then pull in bash.
I forgot - pacman queries the user when multiple providers.
On 19/10/19 10:41 pm, Allan McRae wrote:
On 19/10/19 10:38 pm, Allan McRae wrote:
On 19/10/19 10:24 pm, Andrew Gregory wrote:
On 10/19/19 at 10:15pm, Allan McRae wrote:
On 19/10/19 9:57 pm, Daan van Rossum wrote:
* on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
/bin/sh -> bash/dash /usr/bin/awk -> gawk/nawk /usr/bin/cc -> gcc/clang
Can we provide alternatives by means of sets of mutually exclusive link packages, using package properties that are already supported in pacman?
sh-bash provides=('sh') depends=('bash') conflicts=('sh-dash') sh-dash provides=('sh') depends=('dash') conflicts=('sh-bash') bash depends('sh') dash depends('sh')
Sure - or we could have a system to handle it in a less obtuse manner...
What happens when a new package wants to provide "sh"? We need to rebuild sh-bash and sh-dash? Easy enough, except that assumes that all three packages are packaged by the same group. If bash and dash are packaged by a distro and the new package by a third party, then the conflicts provided by the distro are not enough. But an alternatives system would work.
No rebuilding necessary if the conflicts are changed to just 'sh'. .
What will happen with "pacman -S dash"? My initial thought is dash will pull in the first provider of sh - sh-bash - which will then pull in bash.
I forgot - pacman queries the user when multiple providers. .
$ pacman -S b resolving dependencies... :: There are 2 providers available for foo: :: Repository test 1) a-foo 2) b-foo Enter a number (default=1): It just picks the wrong one by default...
* on Saturday, 2019-10-19 22:43 +1000, Allan McRae <allan@archlinux.org> wrote:
$ pacman -S b resolving dependencies... :: There are 2 providers available for foo: :: Repository test 1) a-foo 2) b-foo
Enter a number (default=1):
It just picks the wrong one by default...
Maybe pacman can print additional information to help the user choose between the options, such as the number of additional dependencies each of the choices would pull in (recursively), or even press a certain key to list those dependencies. The options could then also be sorted by this number instead of alphabetically. This still leaves the issue that default providers are not specified for a base install and that the user gets asked about sh-bash, awk-gawk, cc-gcc. Even with sensible ordering, the user would have to press "enter" three times...
On 10/19/19 at 10:43pm, Allan McRae wrote:
On 19/10/19 10:41 pm, Allan McRae wrote:
On 19/10/19 10:38 pm, Allan McRae wrote:
On 19/10/19 10:24 pm, Andrew Gregory wrote:
No rebuilding necessary if the conflicts are changed to just 'sh'.
What will happen with "pacman -S dash"? My initial thought is dash will pull in the first provider of sh - sh-bash - which will then pull in bash.
I forgot - pacman queries the user when multiple providers.
$ pacman -S b resolving dependencies... :: There are 2 providers available for foo: :: Repository test 1) a-foo 2) b-foo
Enter a number (default=1):
It just picks the wrong one by default...
Implementing something like https://bugs.archlinux.org/task/62480 would take care of that. In order for an alternatives system to be worth it, I think it needs to be able to provide better flexibility than these link packages. For example, you mentioned using it for something like python. Imagine python4 ships IDLE separately; in order to use link packages you'd need something like python4-links-interpreter and python4-links-idle, and you'd probably want an all-encompassing python4-links for convenience too. That would get unwieldy pretty quickly. An alternatives system could allow the user to use python4 for everything else but leave idle pointing to an older version until idle4 is installed. So the features I think would be useful are: 1) the ability to select alternatives per-file 2) the ability to select alternatives per-package 3) the ability to group alternatives, potentially spanning multiple packages, and select them per-group As for conflicts, I think go with first-installed-wins and print a warning when a package gets a new alternative entry and the link is already set to something else.
On October 20, 2019 2:10:44 AM GMT+02:00, Andrew Gregory <andrew.gregory.8@gmail.com> wrote:
So the features I think would be useful are: 1) the ability to select alternatives per-file 2) the ability to select alternatives per-package 3) the ability to group alternatives, potentially spanning multiple packages, and select them per-group
Just a tiny side note, but it would be awesome to support paths instead of only pure files as providers. This would allow to make the Java system default jvm directory to be managed by pacman as well instead of relying on custom archlinux-java script. Cheers
On 20/10/19 10:15 am, Levente Polyak wrote:
On October 20, 2019 2:10:44 AM GMT+02:00, Andrew Gregory <andrew.gregory.8@gmail.com> wrote:
So the features I think would be useful are: 1) the ability to select alternatives per-file 2) the ability to select alternatives per-package 3) the ability to group alternatives, potentially spanning multiple packages, and select them per-group
Just a tiny side note, but it would be awesome to support paths instead of only pure files as providers. This would allow to make the Java system default jvm directory to be managed by pacman as well instead of relying on custom archlinux-java script.
That is on the cards. I had the example of the lua include directory in the first email, but did not explicitly specify that we could symlink directories as well as files. Allan
* on Sunday, 2019-10-20 14:24 +1000, Allan McRae <allan@archlinux.org> wrote:
That is on the cards. I had the example of the lua include directory in the first email, but did not explicitly specify that we could symlink directories as well as files.
Alternatives management via "dynamic modification of a user's environment" has been popular in HPC for decades, e.g. see [1]. In particular, being able to use different alternatives in separate environments is very useful. (This does not preclude setting user-wide or all-user-wide defaults.) It may be helpful to consider separately feature-rich, flexible, dynamic alternatives management for a user's convenience on one hand, versus functionality that is essential for the system to work properly on the other. Package dependencies probably fall under the latter category, as well as sh->bash/dash awk->gawk/nawk cc/clang, whereas lua->lua51/lua52/lua53 or java->java-8-openjdk/... fall under the second, I guess... Best, Daan P.S. I hope this is contribution is constructive, but please let me know if it isn't! 1. https://en.wikipedia.org/wiki/Environment_Modules_(software)
On 20/10/19 7:52 pm, Daan van Rossum wrote:
* on Sunday, 2019-10-20 14:24 +1000, Allan McRae <allan@archlinux.org> wrote:
That is on the cards. I had the example of the lua include directory in the first email, but did not explicitly specify that we could symlink directories as well as files.
Alternatives management via "dynamic modification of a user's environment" has been popular in HPC for decades, e.g. see [1]. In particular, being able to use different alternatives in separate environments is very useful. (This does not preclude setting user-wide or all-user-wide defaults.)
It may be helpful to consider separately feature-rich, flexible, dynamic alternatives management for a user's convenience on one hand, versus functionality that is essential for the system to work properly on the other.
Package dependencies probably fall under the latter category, as well as sh->bash/dash awk->gawk/nawk cc/clang, whereas lua->lua51/lua52/lua53 or java->java-8-openjdk/... fall under the second, I guess...
Best, Daan
P.S. I hope this is contribution is constructive, but please let me know if it isn't!
1. https://en.wikipedia.org/wiki/Environment_Modules_(software)
It is constructive, but I don't think is necessarily relevant. Pacman is a system package manager, not a userspace software manager. The HPC use case is quite different. It allows pieces of software to be made available to individual users, but not system-wide. Allan
* on Sunday, 2019-10-20 21:05 +1000, Allan McRae <allan@archlinux.org> wrote:
It is constructive, but I don't think is necessarily relevant. Pacman is a system package manager, not a userspace software manager.
The HPC use case is quite different. It allows pieces of software to be made available to individual users, but not system-wide.
I was under the impression that Arch Linux had gotten around well without both 1. alternatives 2. env-modules and that when talking about support for alternatives it may be helpful to clearly differentiate between these two[*], like Fedora does as well[1]. * on Saturday, 2019-10-19 18:15 +1000, Allan McRae <allan@archlinux.org> wrote:
The setting of /usr/bin/python is more for interactive use. The distinction appeared a bit vague to me here. Sorry if misinterpreted!
Back to brainstorming - continuing thoughts shared by Allan, Eli, Andrew, Levente, and others - let me share an idea for a not fully fledged alternatives system (such as Fedora has it) but something more simple that maybe fits nicely in pacman's approach and maybe a bit less obtuse than link packages: What if Arch offered alternatives through sets of 1. selector package 2. provider packages Selector package 'awk' owns links in e.g. /usr/bin/ and /usr/share/man that it copied from a fakeroot provided by a provider package under /var/lib/pacman/alternatives/awk/. In addition, it specifies: requires=('awk-provider') alternative_providers=('gawk' 'nawk') # for hinting only, may be incomplete! Provider package 'gawk' specifies selector=('awk') provides=('awk-provider') and a "provide()" function that links the files and directories that it provides to a fakeroot under /var/lib/pacman/alternatives/awk/. Installing or reinstalling a selector package will detect: 0 providers exist: abort suggesting providers from 'alternative_providers' 1 provider exists: continue silently >1 providers exist: ask the user to select a provider and continue Once a provider is selected, its "provide()" function is executed (setting up the fakeroot), and then the selector package is (re-)installed (copying the links from the fakeroot into the system). Providers that get newly installed on a system trigger reinstalling the selector package, offering the user to switch to the new provider. Alternatives can be reconfigured at any time by reinstalling the selector package. (But I think this action would be rare on most systems?) Here are two examples: # pacman -S awk -> failed: selector package 'awk' is missing a provider package. Potential providers are: gawk, nawk # pacman -S netcat gnu-netcat openbsd-netcat -> select a provider for netcat: [1] gnu-netcat, [2] openbsd-netcat. Best, Daan [*] IMHO it is one of Arch's strengths to follow upstream as much as possible and settle on a single current version of e.g. Python. Older versions can be installed but the '/usr/bin/python' is always what upstream deems 'current'. This implies that old versions are never an "alternative" for new versions in arch's sense. A user can always still depart from this KISS approach in his/her environment. [1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Alternatives/
Quoting Daan van Rossum (2019-10-23 07:35:45)
Installing or reinstalling a selector package will detect: 0 providers exist: abort suggesting providers from 'alternative_providers'
# pacman -S awk -> failed: selector package 'awk' is missing a provider package. Potential providers are: gawk, nawk
Seems to me it would be more streamlined to prompt to install a provider here. -- Best, Daniel <https://danielcapella.com>
On 23/10/19 9:35 pm, Daan van Rossum wrote:
* on Sunday, 2019-10-20 21:05 +1000, Allan McRae <allan@archlinux.org> wrote:
It is constructive, but I don't think is necessarily relevant. Pacman is a system package manager, not a userspace software manager.
The HPC use case is quite different. It allows pieces of software to be made available to individual users, but not system-wide.
I was under the impression that Arch Linux had gotten around well without both
<snip>
What if Arch offered alternatives through sets of
Without actually commenting on your idea (because bourbon), I will point out that pacman is distro independent. It just happens that Arch hosts our infrastructure. So our decisions are based around what is best for a package manager, but not Arch. Now, ignoring my comment about not commenting... My design principle for additions to makepkg is an addition should be mostly straight forward to a packager - i.e. if you can build the software manually, you can package it. Suggestions that look complex to package, are too complex. Looking at your suggestion (admittedly... bourbon), it did not pass my "that seems obvious" threshold.
[*] IMHO it is one of Arch's strengths to follow upstream as much as possible and settle on a single current version of e.g. Python. Older versions can be installed but the '/usr/bin/python' is always what upstream deems 'current'.
Upstream python specifically states that /usr/bin/python should always point to /usr/bin/python2. So Arch does not follow upstream in that regard. Just some idiot decided it was time to point to python3 about 10 years before upstream will... Allan
* on Wednesday, 2019-10-23 22:05 +1000, Allan McRae <allan@archlinux.org> wrote:
Now, ignoring my comment about not commenting... My design principle for additions to makepkg is an addition should be mostly straight forward to a packager - i.e. if you can build the software manually, you can package it. Suggestions that look complex to package, are too complex. Looking at your suggestion (admittedly... bourbon), it did not pass my "that seems obvious" threshold.
I think it looks less complex in a single-line summary: Replace "virtual packages" (those specified with "provides=()" statements in other packages) with actual packages that can make use of links prepared by providers. The added complexity for a packager should be small: 1. packager will only work on provider packages, selector packages typically don't change 2. his/her package being a provider is optional; it will still work the same way as it does now without a provider() function 3. the provider() function can almost be copy-pasted (only paths need to be adjusted) from other providers or from the selector PKGBUILD There ain't no such thing as a free lunch... Daan
On 24/10/19 12:31 am, Daan van Rossum wrote:
* on Wednesday, 2019-10-23 22:05 +1000, Allan McRae <allan@archlinux.org> wrote:
Now, ignoring my comment about not commenting... My design principle for additions to makepkg is an addition should be mostly straight forward to a packager - i.e. if you can build the software manually, you can package it. Suggestions that look complex to package, are too complex. Looking at your suggestion (admittedly... bourbon), it did not pass my "that seems obvious" threshold.
I think it looks less complex in a single-line summary:
Replace "virtual packages" (those specified with "provides=()" statements in other packages) with actual packages that can make use of links prepared by providers.
The added complexity for a packager should be small: 1. packager will only work on provider packages, selector packages typically don't change 2. his/her package being a provider is optional; it will still work the same way as it does now without a provider() function 3. the provider() function can almost be copy-pasted (only paths need to be adjusted) from other providers or from the selector PKGBUILD
Compare that to the complexity of the original proposal example for python2: In the PKGBUILD: alternatives=('python') And then provide a file python.alternative containing: /usr/bin/python -> python2 /usr/bin/idle -> idle2 Yes - this potentially results in more complexity in the backend (I'm not sure it will), but is dead simple for a packager. Allan
Am 24.10.19 um 02:10 schrieb Allan McRae:
On 24/10/19 12:31 am, Daan van Rossum wrote:
* on Wednesday, 2019-10-23 22:05 +1000, Allan McRae <allan@archlinux.org> wrote:
Now, ignoring my comment about not commenting... My design principle for additions to makepkg is an addition should be mostly straight forward to a packager - i.e. if you can build the software manually, you can package it. Suggestions that look complex to package, are too complex. Looking at your suggestion (admittedly... bourbon), it did not pass my "that seems obvious" threshold.
I think it looks less complex in a single-line summary:
Replace "virtual packages" (those specified with "provides=()" statements in other packages) with actual packages that can make use of links prepared by providers.
The added complexity for a packager should be small: 1. packager will only work on provider packages, selector packages typically don't change 2. his/her package being a provider is optional; it will still work the same way as it does now without a provider() function 3. the provider() function can almost be copy-pasted (only paths need to be adjusted) from other providers or from the selector PKGBUILD
Compare that to the complexity of the original proposal example for python2:
In the PKGBUILD:
alternatives=('python')
And then provide a file python.alternative containing:
/usr/bin/python -> python2 /usr/bin/idle -> idle2
Yes - this potentially results in more complexity in the backend (I'm not sure it will), but is dead simple for a packager.
I'll have to agree here. My thoughts when reading that last proposal from Daan were: Isn't that essentially an alternatives system now, just a lot more complex? And thinking back to Daan's first proposal, wasn't the goal at that point to implement alternatives with the existing tools and functionality? That might have had some merit. That last proposal needs a lot more than that, though. And to me it is not very clear what happens when you've installed both providers and then want to change say from bash to dash at some point later. Which command would I use? `pacman -S` e.g. reinstall? That's not really clear. Does the packager who writes the provider() function have to pay attention to that and write code that can handle all possible cases? I didn't think too much into it, but writing a function is certainly more complex than a dumb file of key-value pairs like "/bin/foo -> foo2". And even if it's copy-paste-able it's still code, a lot more cant go wrong than with a dump file with very simple syntax. (which could be a lot more easily checked for correctness by makepkg than arbitrary code) So: A dedicated `pacman -A` seems to be a lot more clear about what I'd have to do (to me). And no code written by the packager needed. An if you'd want hints for alternatives, maybe the alternatives name (in the original proposal the `alternatives=('python')`) could be added to the repo's DB, and then searched for? Of course then every package having alternatives for python would have to use the same name... But it would be a bit less data, then adding all the alternatives to the repo's db... though this could be done separately like the files.db? Then a `pacman -Ays python` could print all providers of python, like python3, python2, python4, python-from-third-party-repo So at this point I'd prefer the original proposal over Daan's latest, after we've bikeshedded the '->' of course. ;)
Allan
-- regards, brainpower
* on Thursday, 2019-10-24 10:10 +1000, Allan McRae <allan@archlinux.org> wrote:
Compare that to the complexity of the original proposal example for python2:
In this design, will /usr/bin/python be owned by a package like in the selector-provider design?
In the PKGBUILD:
alternatives=('python')
selector=('python') provides=('python-provider') #-- unnecessary if backend infers this
And then provide a file python.alternative containing:
/usr/bin/python -> python2 /usr/bin/idle -> idle2
provide() { mkdir -p usr/bin ln -s /usr/bin/python2 usr/bin/python ln -s /usr/bin/idle2 usr/bin/idle }
Yes - this potentially results in more complexity in the backend (I'm not sure it will), but is dead simple for a packager.
Complexity for developers vs complexity for users is a delicate tradeoff. The python.alternative design involves slightly less text to write for a maintainer, OK, but splits it over two files instead of one. A function is more powerful than a list of key-value pairs. Can you handle alternative lua include files with a list of key-value pairs? I imagine there may be exceptional situations where the extra power will be nice to have in the future. Best, Daan
On 28/10/19 8:30 pm, Daan van Rossum wrote:
* on Thursday, 2019-10-24 10:10 +1000, Allan McRae <allan@archlinux.org> wrote:
Compare that to the complexity of the original proposal example for python2:
In this design, will /usr/bin/python be owned by a package like in the selector-provider design?
Yes. For -Qo, it would be owned by the package currently providing the alternative (with a note showing it is not a real ownership). For -Fo, it would be own by all packages that provide that file.
In the PKGBUILD:
alternatives=('python')
selector=('python') provides=('python-provider') #-- unnecessary if backend infers this
And then provide a file python.alternative containing:
/usr/bin/python -> python2 /usr/bin/idle -> idle2
provide() { mkdir -p usr/bin ln -s /usr/bin/python2 usr/bin/python ln -s /usr/bin/idle2 usr/bin/idle }
That limits you to only having one alternative option per package.
Yes - this potentially results in more complexity in the backend (I'm not sure it will), but is dead simple for a packager.
Complexity for developers vs complexity for users is a delicate tradeoff.
The python.alternative design involves slightly less text to write for a maintainer, OK, but splits it over two files instead of one.
A function is more powerful than a list of key-value pairs. Can you handle alternative lua include files with a list of key-value pairs?
Yes - all alternatives are symlinks. Symlinks are key-value pairs. Include directories can be symlinked too.
I imagine there may be exceptional situations where the extra power will be nice to have in the future.
Do you have a plausible example where a list of symlinks is not enough? Allan
* on Monday, 2019-10-28 23:08 +1000, Allan McRae <allan@archlinux.org> wrote:
Yes - all alternatives are symlinks. Symlinks are key-value pairs. Include directories can be symlinked too.
I imagine there may be exceptional situations where the extra power will be nice to have in the future.
Do you have a plausible example where a list of symlinks is not enough?
# pacman -Fl lua lua52 | grep include lua usr/include/ lua usr/include/lauxlib.h lua usr/include/lua.h lua usr/include/lua.hpp lua usr/include/luaconf.h lua usr/include/lualib.h lua52 usr/include/ lua52 usr/include/lua5.2/ lua52 usr/include/lua5.2/lauxlib.h lua52 usr/include/lua5.2/lua.h lua52 usr/include/lua5.2/lua.hpp lua52 usr/include/lua5.2/luaconf.h lua52 usr/include/lua5.2/lualib.h Each of the lua header files lives in /usr/include. A function can symlink all files in a directory using wildcards. Would you not need at least some form of wildcard support in a list of key-value pairs? Daan
Do you have a plausible example where a list of symlinks is not enough?
# pacman -Fl lua lua52 | grep include lua usr/include/ lua usr/include/lauxlib.h lua usr/include/lua.h lua usr/include/lua.hpp lua usr/include/luaconf.h lua usr/include/lualib.h lua52 usr/include/ lua52 usr/include/lua5.2/ lua52 usr/include/lua5.2/lauxlib.h lua52 usr/include/lua5.2/lua.h lua52 usr/include/lua5.2/lua.hpp lua52 usr/include/lua5.2/luaconf.h lua52 usr/include/lua5.2/lualib.h
Sorry if this question is silly. For this case, wouldn't it be possible to just symlink the directory instead? Thanks, -Santiago.
On 10/28/19 9:29 AM, Daan van Rossum wrote:
* on Monday, 2019-10-28 23:08 +1000, Allan McRae <allan@archlinux.org> wrote:
I imagine there may be exceptional situations where the extra power will be nice to have in the future.
Do you have a plausible example where a list of symlinks is not enough?
# pacman -Fl lua lua52 | grep include lua usr/include/ lua usr/include/lauxlib.h lua usr/include/lua.h lua usr/include/lua.hpp lua usr/include/luaconf.h lua usr/include/lualib.h lua52 usr/include/ lua52 usr/include/lua5.2/ lua52 usr/include/lua5.2/lauxlib.h lua52 usr/include/lua5.2/lua.h lua52 usr/include/lua5.2/lua.hpp lua52 usr/include/lua5.2/luaconf.h lua52 usr/include/lua5.2/lualib.h
Each of the lua header files lives in /usr/include. A function can symlink all files in a directory using wildcards. Would you not need at least some form of wildcard support in a list of key-value pairs?
I'd argue it is incorrectly packaged, then. They should live in /usr/include/lua/ which would be a directory symlink. I'm admittedly not sure how to maintain the ability to include from /usr/include without additional symlinks that have no package to be packaged in... But if we needed wildcard support for only a subset of files in that directory, then we could use bash wildcards to generate a bunch of different alternatives, and link them together as a single group. This is already something we want, because we want to change the symlink for /usr/bin/lua at the same time anyway. -- Eli Schwartz Bug Wrangler and Trusted User
* on Wednesday, 2019-10-23 22:05 +1000, Allan McRae <allan@archlinux.org> wrote:
On 23/10/19 9:35 pm, Daan van Rossum wrote:
[*] IMHO it is one of Arch's strengths to follow upstream as much as possible and settle on a single current version of e.g. Python. Older versions can be installed but the '/usr/bin/python' is always what upstream deems 'current'.
Upstream python specifically states that /usr/bin/python should always point to /usr/bin/python2. So Arch does not follow upstream in that regard. Just some idiot decided it was time to point to python3 about 10 years before upstream will...
Given the alternatives system you, Allan, suggest. What if a user runs # pacman -S lua , will you print a list of alternatives for a user to choose from? Suppose you do that, and present the options 1. lua53 2. lua52 3. lua51 and the user choose lua53, now what happens when lua54 is released upstream? I was happy with pacman giving me the latest upstream and in peace with it following upstream. One concern I would like to share is that introducing alternatives for managing parallel "versions" of the same tool may impact how KISS Arch/pacman is going to be. Sure, the Arch devs that decided that python3 was more appropriate to follow for a bleeding edge distro like Arch than python2 might not have met everyone's choice, but it is something that users can easily work with (as evidenced by the fact that env-modules lives in AUR)! Supporting alternative upstreams such as nawk <-> gawk is all different in my view; easy to support and nice to have. IMHO it would stress the beauty and power of Arch/pacman if it is able to support such alternatives without the need of a fully-fledged alternatives system. Daan
On November 4, 2019 6:06:04 PM GMT+01:00, Daan van Rossum <d.r.vanrossum@gmx.de> wrote:
* on Wednesday, 2019-10-23 22:05 +1000, Allan McRae <allan@archlinux.org> wrote:
On 23/10/19 9:35 pm, Daan van Rossum wrote:
[*] IMHO it is one of Arch's strengths to follow upstream as much as possible and settle on a single current version of e.g. Python. Older versions can be installed but the '/usr/bin/python' is always what upstream deems 'current'.
Upstream python specifically states that /usr/bin/python should always point to /usr/bin/python2. So Arch does not follow upstream in that regard. Just some idiot decided it was time to point to python3 about 10 years before upstream will...
Given the alternatives system you, Allan, suggest. What if a user runs
# pacman -S lua
, will you print a list of alternatives for a user to choose from?
Suppose you do that, and present the options
1. lua53 2. lua52 3. lua51
and the user choose lua53, now what happens when lua54 is released upstream? I was happy with pacman giving me the latest upstream and in peace with it following upstream.
One concern I would like to share is that introducing alternatives for managing parallel "versions" of the same tool may impact how KISS Arch/pacman is going to be.
What exactly forbids people to still name the non pin-versioned variant lua as it today is the case in Arch?
On 11/4/19 12:06 PM, Daan van Rossum wrote:
Given the alternatives system you, Allan, suggest. What if a user runs
# pacman -S lua
, will you print a list of alternatives for a user to choose from?
Suppose you do that, and present the options
1. lua53 2. lua52 3. lua51
and the user choose lua53, now what happens when lua54 is released upstream? I was happy with pacman giving me the latest upstream and in peace with it following upstream.
My expectation is it installs what it does today, the "lua" package which provides /usr/bin/lua5.3 and now, instead of installing /usr/bin/lua, installs an alternative that is fulfilled by default for /usr/bin/lua5.3 I suppose I see your point w.r.t. updating the package, does updating lua 5.3.5-2 to lua 5.4.0-1 keep the symlink, updating it to point to /usr/bin/lua5.4, but I don't see why it needs to change how we name packages and we're under no obligation to rename 'extra/python' to 'extra/python3' and have it merely provide 'python'. We can and should continue to make `pacman -S python lua` install the preferred forms of the package.
One concern I would like to share is that introducing alternatives for managing parallel "versions" of the same tool may impact how KISS Arch/pacman is going to be.
Sure, the Arch devs that decided that python3 was more appropriate to follow for a bleeding edge distro like Arch than python2 might not have met everyone's choice, but it is something that users can easily work with (as evidenced by the fact that env-modules lives in AUR)!
I think it's fine to have pacman -S python install python3, and have there be a very high likelihood that /usr/bin/python points to python3. This doesn't mean we need to stop people from switching it if they really want to. python 3.x will be installed first in most cases, because much more common software depends on python3 these days than on python2, including many core desktop environment components.
Supporting alternative upstreams such as nawk <-> gawk is all different in my view; easy to support and nice to have. IMHO it would stress the beauty and power of Arch/pacman if it is able to support such alternatives without the need of a fully-fledged alternatives system.
Do you really think gawk/nawk is all as simple as that -- consider makepkg, which depends on awk, but "Results may be inaccurate when using a broken implementation of awk, such as mawk or busybox awk." Thankfully we don't package mawk, we do package busybox awk though. Either gawk or nawk would work, but do we force people to use gawk? There's no such thing as /usr/bin/nontrashy-awk. Opinions, opinions, opinions everywhere. You'll always be able to find something objectionable about making any tool selectable, so let's not take sides and say python is less deserving than awk. :) -- Eli Schwartz Bug Wrangler and Trusted User
* on Monday, 2019-11-04 12:39 -0500, Eli Schwartz <eschwartz@archlinux.org> wrote:
My expectation is it installs what it does today, the "lua" package which provides /usr/bin/lua5.3 and now, instead of installing /usr/bin/lua, installs an alternative that is fulfilled by default for /usr/bin/lua5.3
Then you end up with package lua52 that provides lua, and a package lua providing lua53. Ugly!
<snip> I think it's fine to have pacman -S python install python3, and have there be a very high likelihood that /usr/bin/python points to python3. This doesn't mean we need to stop people from switching it if they really want to. python 3.x will be installed first in most cases, because much more common software depends on python3 these days than on python2, including many core desktop environment components.
Pacman already doesn't stop a user from having python point to whatever the user wants. Arch users know that but does pacman also need to support OSes where users don't know that? That brings up another idea: why does pacman not provide alternatives via managing symlinks in /usr/local (or even /usr/alternatives)? That would be entirely independent of the current package structure. If a user doesn't set alternatives then /usr/local remains empty. This would also support the ability to select alternatives per-file etc. as was pointed out by Andrew. I guess a simple version of this could as well be a separate tool instead of part of pacman, except that no comprehensive alternatives info would be available to the user. But do users really need that? I think a user that wants to set an alternative knows what he/she wants to do. The simplest version of this is to let the user manage these links manually :) Daan
On 5/11/19 7:40 am, Daan van Rossum wrote:
* on Monday, 2019-11-04 12:39 -0500, Eli Schwartz <eschwartz@archlinux.org> wrote:
My expectation is it installs what it does today, the "lua" package which provides /usr/bin/lua5.3 and now, instead of installing /usr/bin/lua, installs an alternative that is fulfilled by default for /usr/bin/lua5.3
Then you end up with package lua52 that provides lua, and a package lua providing lua53. Ugly!
<snip> I think it's fine to have pacman -S python install python3, and have there be a very high likelihood that /usr/bin/python points to python3. This doesn't mean we need to stop people from switching it if they really want to. python 3.x will be installed first in most cases, because much more common software depends on python3 these days than on python2, including many core desktop environment components.
Pacman already doesn't stop a user from having python point to whatever the user wants. Arch users know that but does pacman also need to support OSes where users don't know that?
Yes - pacman is used many places that is not Arch. e.g. Msys2.
That brings up another idea: why does pacman not provide alternatives via managing symlinks in /usr/local (or even /usr/alternatives)? That would be entirely independent of the current package structure. If a user doesn't set alternatives then /usr/local remains empty. This would also support the ability to select alternatives per-file etc. as was pointed out by Andrew.
I guess a simple version of this could as well be a separate tool instead of part of pacman, except that no comprehensive alternatives info would be available to the user. But do users really need that? I think a user that wants to set an alternative knows what he/she wants to do.
The simplest version of this is to let the user manage these links manually :)
Sure, you can let a user symlink /usr/local/bin/python -> /usr/bin/python2 and put /usr/local/bin first in the path. But not "man python" does not provide information on what the user sees as the "python" command, and python-config does not point at the python2 version, and ... Simple does not mean half-arsed. Allan
participants (9)
-
Allan McRae
-
Andrew Gregory
-
brainpower
-
Daan van Rossum
-
Daniel M. Capella
-
Eli Schwartz
-
Giancarlo Razzolini
-
Levente Polyak
-
Santiago Torres Arias