[arch-general] depends vs. optdepends
So, I found 4 executable/binary files installed (out of thousands - good job guys) that fail a 'ldd' check (missing shared libraries). The packages in question mark the missing libraries as 'optdepends' so they never get installed - I filed 4 bug reports, all 4 got closed as "not a bug" by the same person. I disagree with this choice as I believe it's ignoring system integrity needlessly - 4 files are installed which knowingly do not *function* (cannot launch) at all. The person who closed it even wrote "So what? Non-functional executables are not an issue here, this is the point of optional dependencies." The wiki states "optdepends: An array of package names that are not needed for the software to function but provides additional features." OK, so per the wiki it's pretty clear - missing libs fail the function test. The man PKGBUILD states "optdepends (array) An array of packages (and accompanying reasons) that are not essential for base functionality, but may be necessary to make full use of the contents of this package." Per this, there is a huge grey area open to interpretation. It is my opinion this is a dangerous precedent to install binaries from an official package that are linked to shared libraries which are not required to be installed. I'm wondering why Arch considers it acceptable to just ignore the problem (which could be solved in all 4 cases by simply splitting the package for that binary and adding a proper depends). Thoughts? -te [1] https://bugs.archlinux.org/task/43260 [2] https://bugs.archlinux.org/task/43261 [3[ https://bugs.archlinux.org/task/43262 [4] https://bugs.archlinux.org/task/43263
On Mon, 29 Dec 2014 17:58:49 -0600 Troy Engel <troyengel+arch@gmail.com> wrote:
So, I found 4 executable/binary files installed (out of thousands - good job guys) that fail a 'ldd' check (missing shared libraries). The packages in question mark the missing libraries as 'optdepends' so they never get installed - I filed 4 bug reports, all 4 got closed as "not a bug" by the same person. I disagree with this choice as I believe it's ignoring system integrity needlessly - 4 files are installed which knowingly do not *function* (cannot launch) at all. The person who closed it even wrote "So what? Non-functional executables are not an issue here, this is the point of optional dependencies."
The wiki states "optdepends: An array of package names that are not needed for the software to function but provides additional features." OK, so per the wiki it's pretty clear - missing libs fail the function test.
The man PKGBUILD states "optdepends (array) An array of packages (and accompanying reasons) that are not essential for base functionality, but may be necessary to make full use of the contents of this package." Per this, there is a huge grey area open to interpretation.
It is my opinion this is a dangerous precedent to install binaries from an official package that are linked to shared libraries which are not required to be installed. I'm wondering why Arch considers it acceptable to just ignore the problem (which could be solved in all 4 cases by simply splitting the package for that binary and adding a proper depends).
Thoughts?
-te
[1] https://bugs.archlinux.org/task/43260 [2] https://bugs.archlinux.org/task/43261 [3[ https://bugs.archlinux.org/task/43262 [4] https://bugs.archlinux.org/task/43263
You found 4 binaries, how many more would you have found if you didn't coincidently happen to have the optional deps for them already installed? Then add in all of the number of python and perl scripts that are broken/would be broken without the optional deps. It's actually very, very ubiquitous in Arch and is the entire point of optional dependencies. If it's not required for the major functionality of the package and the software functions without it, it's optional. This isn't setting a new precedent, this has been SOP for a long time. Arch isn't Debian, every binary isn't given it's own package. Doug
On Mon, Dec 29, 2014 at 6:13 PM, Doug Newgard <scimmia@archlinux.info> wrote:
You found 4 binaries, how many more would you have found if you didn't coincidently happen to have the optional deps for them already installed?
If I found more, I would report them - just as I did these four. Using a logical fallacy of false equivalence won't win this particular point, just because I didn't find every last one does not denote it isn't an issue to be discussed. By finding the four I did, I contribute towards chipping away at problems and try making Arch better tomorrow that it is today.
Then add in all of the number of python and perl scripts that are broken/would be broken without the optional deps. It's actually very, very ubiquitous in Arch and is the entire point of optional dependencies.
I agree on anything scripted - again you're trying to use false equivalence logic as I *specifically* called out that they were executable/binary files, not interpreted scripts of some sort. I presented two conflicting sources of what 'optdepends' means, there was no comment about why they're different descriptions. I have packaged python/perl/bash/ruby/etc. scripts and understand the pitfalls of these, this isn't what was presented for discussion.
If it's not required for the major functionality of the package and the software functions without it, it's optional.
The lm_sensors package provides the daemon sensorsd (complete with systemd unit). To use this very specific example, are you declaring that this included binary is not "major" and you *cannot* launch this daemon as-is but that's not considered non-functional. What is your definition of "the software functions" if every binary included in the package does not at least launch?
This isn't setting a new precedent, this has been SOP for a long time.
So that makes it correct? "This is the way we've always done it" -- we'd never have systemd as an example if things were not open to discussion and change.
Arch isn't Debian, every binary isn't given it's own package.
Nor did I ever claim it was (and what are you even arguing here?) -- I was careful to mention the packaging tools (RPM/DEB) as both of these infrastructures underpin many distros. In both of these systems there is automatic library dependency tracking/inclusion as part of the process; if a binary is in a package and has a shared library dependency, it's implicitly pulled in as a need to install (if not explicitly defined). As far as I am able to tell, Pacman has no such capability so everything must be done manually. -te
Arch currently uses optional dependencies even when it means that executables provided by the package aren't going to work with the minimal set of dependencies. The packages could be split up more to avoid this without pulling in more stuff, but it's not what packagers usually choose to it. It's a design choice left up to the packager, not a bug. The namcap tool reports missing libraries when packages are built with devtools so you're not going to be finding stuff that hasn't already been reported automatically.
On Mon, Dec 29, 2014 at 6:50 PM, Daniel Micay <danielmicay@gmail.com> wrote:
Arch currently uses optional dependencies even when it means that executables provided by the package aren't going to work with the minimal set of dependencies. The packages could be split up more to avoid this without pulling in more stuff, but it's not what packagers usually choose to it. It's a design choice left up to the packager, not a bug.
Thank you - tribal knowledge that wasn't written down as an acceptable standard to the Arch packaging. I have added your first sentence to a note (and credited you in the comment) so that it's not tribal anymore: https://wiki.archlinux.org/index.php/PKGBUILD#optdepends The reason this is complicated is anyone who comes into Arch from (most) other distros where this is considered a bug needs to understand Arch doesn't work by the same rules as all the others for binary shared library resolution. I disagree with the SOP, but can leave my opinion at the door. -te
Hi, On Mon, Dec 29, 2014 at 07:20:42PM -0600, Troy Engel wrote:
On Mon, Dec 29, 2014 at 6:50 PM, Daniel Micay <danielmicay@gmail.com> wrote:
Arch currently uses optional dependencies even when it means that executables provided by the package aren't going to work with the minimal set of dependencies. The packages could be split up more to avoid this without pulling in more stuff, but it's not what packagers usually choose to it. It's a design choice left up to the packager, not a bug.
Thank you - tribal knowledge that wasn't written down as an acceptable standard to the Arch packaging. I have added your first sentence to a note (and credited you in the comment) so that it's not tribal anymore:
Just because something is written in a wiki doesn't make it rational....
The reason this is complicated is anyone who comes into Arch from (most) other distros where this is considered a bug needs to understand Arch doesn't work by the same rules as all the others for binary shared library resolution. I disagree with the SOP, but can leave my opinion at the door.
Well, other distros following certain packaging guidelines doesn't imply that Arch should follow them as well. If you subscribe e.g. to fedora-devel you'll see how much more complicated compared to Arch the packaging process in Fedora is (and this process is faaaar from perfect). And I am not even talking about Debian... There is a tradeoff of complexity vs some "breakage". So it's not about you agreeing or not with the status quo, it's about contributing a more rational alternative than the existing PKGBUILD. In your jasper bugreport, it would have helped if you provided a PKGBUILD with split-packages, like jasper-base and jasper-jiv with proper "conflicts" and "provides". At least that's what I would do... Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
To use the sensord example: in lm_sensors/PKGBUILD: optdepends=('rrdtool: for logging with sensord') I am unsure if there is any way to be more blatantly clear than it is already. :p This is a phenomenal non-issue, and the perfect example of why optdepends exists in the first place... as has already been pointed out in every response on the matter... -- Eli Schwartz
On Mon, 29 Dec 2014 18:39:55 -0600 Troy Engel <troyengel+arch@gmail.com> wrote:
On Mon, Dec 29, 2014 at 6:13 PM, Doug Newgard <scimmia@archlinux.info> wrote:
You found 4 binaries, how many more would you have found if you didn't coincidently happen to have the optional deps for them already installed?
If I found more, I would report them - just as I did these four. Using a logical fallacy of false equivalence won't win this particular point, just because I didn't find every last one does not denote it isn't an issue to be discussed. By finding the four I did, I contribute towards chipping away at problems and try making Arch better tomorrow that it is today.
No false equivalence here. My point is there are likely many, many more binaries on your system that have the same issue, you just happen to already have the optional deps installed. That equivalence is not false.
Then add in all of the number of python and perl scripts that are broken/would be broken without the optional deps. It's actually very, very ubiquitous in Arch and is the entire point of optional dependencies.
I agree on anything scripted - again you're trying to use false equivalence logic as I *specifically* called out that they were executable/binary files, not interpreted scripts of some sort. I presented two conflicting sources of what 'optdepends' means, there was no comment about why they're different descriptions. I have packaged python/perl/bash/ruby/etc. scripts and understand the pitfalls of these, this isn't what was presented for discussion.
Why are you trying to separate compiled binaries and interpreted scripts? There's no functional difference in respect to this discussion.
If it's not required for the major functionality of the package and the software functions without it, it's optional.
The lm_sensors package provides the daemon sensorsd (complete with systemd unit). To use this very specific example, are you declaring that this included binary is not "major" and you *cannot* launch this daemon as-is but that's not considered non-functional. What is your definition of "the software functions" if every binary included in the package does not at least launch?
This is up to the judgment of the individual packager.
This isn't setting a new precedent, this has been SOP for a long time.
So that makes it correct? "This is the way we've always done it" -- we'd never have systemd as an example if things were not open to discussion and change.
I was responding to your claim of setting "a dangerous precedent", where that precedent was set long ago. You need to realize that you aren't talking about a bug in one or two packagers' packages, you're talking about the packaging consensus that Arch has used for a long time.
Arch isn't Debian, every binary isn't given it's own package.
Nor did I ever claim it was (and what are you even arguing here?) -- I was careful to mention the packaging tools (RPM/DEB) as both of these infrastructures underpin many distros. In both of these systems there is automatic library dependency tracking/inclusion as part of the process; if a binary is in a package and has a shared library dependency, it's implicitly pulled in as a need to install (if not explicitly defined). As far as I am able to tell, Pacman has no such capability so everything must be done manually.
-te
Yes, pacman is a much simpler package manager, but that has nothing to do with Arch's packaging practices in this case. Arch will generally keep a package together as shipped by upstream unless there's a very compelling reason to split it up. Debian is the other extreme where they split just about everything into it's own package. Simply different philosophies. Doug
On Mon, 29 Dec 2014 19:01:16 -0600, Doug Newgard wrote:
On Mon, 29 Dec 2014 18:39:55 -0600 Troy Engel <troyengel+arch@gmail.com> wrote:
Nor did I ever claim it was (and what are you even arguing here?) -- I was careful to mention the packaging tools (RPM/DEB) as both of these infrastructures underpin many distros. In both of these systems there is automatic library dependency tracking/inclusion as part of the process; if a binary is in a package and has a shared library dependency, it's implicitly pulled in as a need to install (if not explicitly defined). As far as I am able to tell, Pacman has no such capability so everything must be done manually.
-te
Yes, pacman is a much simpler package manager, but that has nothing to do with Arch's packaging practices in this case. Arch will generally keep a package together as shipped by upstream unless there's a very compelling reason to split it up. Debian is the other extreme where they split just about everything into it's own package. Simply different philosophies.
Regarding optional dependencies DEB is insane complicated. Optional dependencies are split into suggested and recommended dependencies. The way to not install or to install those different kinds of optional dependencies is handled in opposing work-flows. While I don't maintain my Debian and *buntu installs, I still help users on DEB based distros lists. I very often need to explain how to not install or to get rid of suggested and or recommended hard dependencies and how to install suggested and/or recommended dependencies. Hopefully the "tribal knowledge" how maintainers decide what becomes a suggested and what becomes a recommended dependency will be add by Troy to the Debian and Ubuntu Wikis. The editing of the Arch Wiki IMO is nonsense, just spreading FUD.
On Tue, Dec 30, 2014 at 5:39 AM, Ralf Mardorf <ralf.mardorf@rocketmail.com> wrote:
Hopefully the "tribal knowledge" how maintainers decide what becomes a suggested and what becomes a recommended dependency will be add by Troy to the Debian and Ubuntu Wikis. The editing of the Arch Wiki IMO is nonsense, just spreading FUD.
Now you're just attacking me with no basis, so while I was done I feel the need to rebut your whole diatribe - you're out in left field with all this talk about Debian. I didn't bring it up, I don't even *like* DPKG/APT -- I am a classic Red Hat (family) guy and base most of my knowledge on RPM techniques. The whole Suggests/Recommends stuff the DEB packaging world does annoy me personally, but that's just another red herring. For your paragraph, you might want to understand Recommends and Suggests in DEB[1] have nothing to do with shared library deps. (RPM has no such concept) That said, you're not even talking about the right things here - Arch is using optdepends() for which the other two players have automatic resolution. In RPM[2] there's a macro "_use_internal_dependency_generator" that can be defined to use internal or external shared library resolution (and then it gets rather complicated from there in code). In DEB[3] it's a feature ${shlibs:Depends} that calls dh_shlibdeps for the shared library resolution (and both also have hooks for perl, e.g. - much more complicated). Both are supplemental to any manually declared requirements and actually inspect the binaries for shared libs, then add to the package metadata that this shared lib must be present before it will install. So no, Ralf, it is not "editing of the Arch Wiki IMO is nonsense, just spreading FUD" at all -- it's clarifying a situation that is unique for Arch (and realistically, Pacman) that packages can have "dangling binaries" which may not function unless you manually install the optdepends() items. To quote Rashif - "it's a hack"; documented and we move on. Let the horse die. -te [1] https://www.debian.org/doc/manuals/maint-guide/dreq.en.html [2] http://rpm.org/gitweb?p=rpm.git;a=blob;f=build/rpmfc.c;hb=HEAD#l1235 [3] http://anonscm.debian.org/cgit/debhelper/debhelper.git/tree/dh_shlibdeps
On Tue, Dec 30, 2014 at 4:54 AM, Leonid Isaev <lisaev@umail.iu.edu> wrote:
Well, other distros following certain packaging guidelines doesn't imply that Arch should follow them as well. If you subscribe e.g. to fedora-devel you'll see how much more complicated compared to Arch the packaging process in Fedora is (and this process is faaaar from perfect). And I am not even talking about Debian... There is a tradeoff of complexity vs some "breakage".
+1 On Tue, Dec 30, 2014 at 3:15 PM, Troy Engel <troyengel+arch@gmail.com> wrote:
So no, Ralf, it is not "editing of the Arch Wiki IMO is nonsense, just spreading FUD" at all -- it's clarifying a situation that is unique for Arch (and realistically, Pacman) that packages can have "dangling binaries" which may not function unless you manually install the optdepends() items. To quote Rashif - "it's a hack"; documented and we move on. Let the horse die.
As you have correctly deduced that we are using something called "pacman", on this "Arch Linux" distro, let me elucidate to you the prudent course of action: Different distros will have different policies, and different package managers, different ways of doing things. When you see a *major bug* in the foundational design strategy of a distro, don't automatically assume it IS a bug, it may be by design. Don't assume arch works like debian or redhat, and don't assume pacman works like yum or apt. I won't assume debian works like redhat or arch, and I won't assume apt works like pacman or yum. I won't assume redhat works like debian or arch, and I won't assume yum works like pacman or apt. If I had to hazard a guess: Since Arch is a do-it-yourself system, which expects its users to scrutinize the update process and do configuration themselves, it seems they assumed that anyone who does NOT know when they install a program that certain binaries require optdepends to run (even though they are on the system)... should not be using arch. [2] Either way, though, it would seem logical to first look into what arch users say about the matter, maybe ask a question on the mailing list "I am not overly familiar with arch packaging standards, but is this behavior normal?", rather than first reporting it as a bug. And for gosh sakes, when the person who closes the bug as "not a bug" because *already an optdep* (clue) is one of the Arch Linux Support Staff[1] that would imply he knows what he is talking about. [1] https://www.archlinux.org/people/support-staff/ [2] https://wiki.archlinux.org/index.php/The_Arch_Way#User-centric
On Mon, 29 Dec 2014 18:13:52 -0600, Doug Newgard wrote:
Arch isn't Debian, every binary isn't given it's own package.
When I read the OPs mail, I wanted to write the same. Now I just need to ad my +1. I very seldom want split packages, I prefer that Arch Linux very seldom does split packages and usually keep it as done by upstream. We perhaps still remember how many times jackd for Debian was broken, because Debian split the jackd package. The libjack package alone was completely useless, but often contributed a broke link. jackd upstream always was pissed off, when they got tons of requests, just because Debian package maintainers did a bad job. The bad job was enforced by Debian's odd policy. The dependency discussion could become the only reason for me, on day to drop Linux. Fortunately there are still many applications that don't force to install unneeded hard dependencies. Many hard dependencies are completely useless for my needs and at least one hard dependency even damages my hardware. Some packages make gvfs a hard dependency. This does damage my external hard drive. Other packages make it an optional dependencies. Making it optional IMO is the only sane way to satisfy all users. When I can install an empty dummy package as replacement for a dependency, without breaking the main functionality off the app, then such a dependency should become an optional dependency. For good reasons I stopped maintaining my Debian and Ubuntu installs and only maintain my Arch Linux install. Regards, Ralf
On 30 December 2014 at 05:58, Troy Engel <troyengel+arch@gmail.com> wrote:
It is my opinion this is a dangerous precedent to install binaries from an official package that are linked to shared libraries which are not required to be installed. I'm wondering why Arch considers it acceptable to just ignore the problem (which could be solved in all 4 cases by simply splitting the package for that binary and adding a proper depends).
Thoughts?
Troy, This is a hack, I agree. However, this hack has no significant side-effects, and yet accommodates a minimalist use case. A dependency which has significant installation/resource footprint can be made optional if its executables or libraries are not required for the basic functionality of the program. As we do not subscribe to bureaucracy we see it as a pragmatic means to an end. If it works without side-effects, and serves a purpose, then we have no problem with it. If the users of a package think something should not be optional, they can file a bug and vote on it. The packager should normally honour users' wishes. -- GPG/PGP ID: C0711BF1
participants (7)
-
Daniel Micay
-
Doug Newgard
-
Eli Schwartz
-
Leonid Isaev
-
Ralf Mardorf
-
Rashif Ray Rahman
-
Troy Engel