On 19.01.2011 19:29, Dan McGee wrote:

On Wed, Jan 19, 2011 at 11:13 AM, Florian Pritz wrote:

...

-dd ignores only the version of a dependency being checked, but not the package itself.

I don't mind this, but it just seems...backwards. Specifying more flags should make it less-restrictive, not more restrictive, but I understand the desire to keep backward compatibility. With that said, does it matter? People that use -d very often are usually screwing their system or know what they are doing- should we make -d just skip versions and -dd skip everything?

...

...

static int parsearg_trans(int opt) { + static int nodeps = 0; switch(opt) { - case 'd': config->flags |= PM_TRANS_FLAG_NODEPS; break; + case 'd': + nodeps++; + if(nodeps == 1) { + config->flags |= PM_TRANS_FLAG_NODEPS; + } else if(nodeps == 2) { + config->flags ^= PM_TRANS_FLAG_NODEPS; + config->flags |= PM_TRANS_FLAG_NODEPVERSION; + } + break; You can do this without a static local; look at "case 's'" in parsearg_remove().

I'll wait for a decision about -d and -dd before fixing that. -- Florian Pritz -- {flo,bluewind}@server-speed.net

On 25/01/11 23:03, Dan McGee wrote:

On Thu, Jan 20, 2011 at 12:20 PM, Florian Pritz wrote:

...

On 19.01.2011 19:29, Dan McGee wrote:

...

On Wed, Jan 19, 2011 at 11:13 AM, Florian Pritz wrote:

...

-dd ignores only the version of a dependency being checked, but not the package itself.

I don't mind this, but it just seems...backwards. Specifying more flags should make it less-restrictive, not more restrictive, but I understand the desire to keep backward compatibility. With that said, does it matter? People that use -d very often are usually screwing their system or know what they are doing- should we make -d just skip versions and -dd skip everything?

...

...

...

static int parsearg_trans(int opt) { + static int nodeps = 0; switch(opt) { - case 'd': config->flags |= PM_TRANS_FLAG_NODEPS; break; + case 'd': + nodeps++; + if(nodeps == 1) { + config->flags |= PM_TRANS_FLAG_NODEPS; + } else if(nodeps == 2) { + config->flags ^= PM_TRANS_FLAG_NODEPS; + config->flags |= PM_TRANS_FLAG_NODEPVERSION; + } + break; You can do this without a static local; look at "case 's'" in parsearg_remove().

I'll wait for a decision about -d and -dd before fixing that.

Anyone else want to offer an opinion on this? I'd like to reverse the two options to match what we do elsewhere- -d will now just ignore versions, -dd everything.

That seems reasonable. With -d getting stricter in terms of dependency checks, there should not be any issues caused by this change going unnoticed by people who currently using -d. Allan

On Fri, Jan 21, 2011 at 8:25 AM, Florian Pritz wrote:

On 19.01.2011 19:54, Dan McGee wrote:

...

It is most definitely not a valid pkgver (dash) or pkgrel (not a number).

The dash here just seperates pkgver from pkgrel.

Did a quick test with libc.so=6-x86_64_Linux as dependency and a package called libc.so with that pkgver and pkgrel and it worked just fine.

"Works" was not the standard to meet here. The standard is what the prevailing values tend to be (which is integers or decimals), what we may enforce in the future. Perhaps more importantly, this is still wrong (I can't run your i686 binary on my i386 system as it seems to indicate) and if we do keep it, it has *nothing* to do with a version in the normal ordering sense- it would belong as part of the provision name. -Dan

On 22/01/11 00:57, Florian Pritz wrote:

On 21.01.2011 15:49, Dan McGee wrote:

...

On Fri, Jan 21, 2011 at 8:25 AM, Florian Pritz wrote:

...

On 19.01.2011 19:54, Dan McGee wrote:

...

It is most definitely not a valid pkgver (dash) or pkgrel (not a number).

The dash here just seperates pkgver from pkgrel.

Did a quick test with libc.so=6-x86_64_Linux as dependency and a package called libc.so with that pkgver and pkgrel and it worked just fine.

...

Perhaps more importantly, this is still wrong (I can't run your i686 binary on my i386 system as it seems to indicate)

http://mailman.archlinux.org/pipermail/pacman-dev/2010-February/010410.html

That reply is just wrong... i686 is not a restricted flavour of i386. It is the other way around. I can not run i686 optimised software on an i386 system. Just ask all the Via C3 owners who do not have that "nopl" instruction and the joys they had with glibc-2.12. So there is a real problem that you can not get the correct value out of the library on i686 systems. We could use CARCH, but that does not work for multilib stuff, which was the entire point of including it in the first place... This needs left out unless the correct value can be given.

...

and if we do keep it, it has *nothing* to do with a version in the normal ordering sense- it would belong as part of the provision name.

I had that before and Allan didn't like it.

http://mailman.archlinux.org/pipermail/pacman-dev/2010-February/010420.html

I did not like provides=(libfoo.so) magically turning into "libfoo.so-i686" in the .PKGINFO file. The details of the package should reflect what is in the PKGBUILD. Allan

Wow, this degenerated into a flamewar quickly. Can we all calm down just a tad? If we were truly at an impasse, Allan and I wouldn't even be looking at these patches. The biggest reason I'm putting up a stop sign and red flags is the lack of explanation at the questions I've posed, and stating I've only offered destructive criticism is not the truth- I just don't understand this whole thing. On Fri, Jan 21, 2011 at 10:54 AM, Thomas Bächler wrote:

Am 21.01.2011 17:01, schrieb Allan McRae:

...

That reply is just wrong...  i686 is not a restricted flavour of i386. It is the other way around.  I can not run i686 optimised software on an i386 system.  Just ask all the Via C3 owners who do not have that "nopl" instruction and the joys they had with glibc-2.12.

That is not the point. We should reflect what the dynamic linker tries to do whenever we want to load a binary:

1) If we install a i686 version of libfoo.so.2 onto a i386 system, the linker will still try to load that version (and fail with SIGILL). 2) If we install a x86_64 version and a i686 version of libfoo.so.2 into the system, and launch a 32 bit binary, the linker will determine that it cannot load the the x86_64 version, but the i686 version.

So, in order to "do it right", we only need to reflect the distinction the linker does in 2). Not even the linker makes the distinction in 1), so why should we?

And now think about use cases: The package architecture is already in PKGINFO. So the only distinction we need to make is between 32 bit and 64 bit binaries - the only case where you are able to run more than one architecture in the same system is when executing 32 bit binaries on a 64 bit platform with a backward compatibility mode for 32 bit. (AFAIK, this is not only the case for x86, but for some other platforms, too).

The only info we should include into the soprovides/depends is whether we have a 32 bit or a 64 bit binary. There shouldn't be an explicit statement about architecture.

Taking a step back here, if we want to not lock ourself into ELF-only (OSX does not use it), it sounds like using objdump due to it's use of the BFD backend which understands multiple file formats. From objdump -a, although the output is rather arcane, we should always be able to get the file format of the library, which I believe is the most important to matching what the linker does. Sidenote: I saw this when running objdump --info on my x86_64 box: plugin (header little endian, data little endian) BFD: BFD (GNU Binutils) 2.21.0.20101217 assertion fail /build/src/binutils/bfd/plugin.c:453 objdump: plugin: Bad value

...

So there is a real problem that you can not get the correct value out of the library on i686 systems.  We could use CARCH, but that does not work for multilib stuff, which was the entire point of including it in the first place...

I think my previous comments should make this irrelevant, unless I am completely wrong.

...

...

...

and if we do keep it, it has *nothing* to do with a version in the normal ordering sense- it would belong as part of the provision name.

I had that before and Allan didn't like it.

http://mailman.archlinux.org/pipermail/pacman-dev/2010-February/010420.html

I did not like provides=(libfoo.so) magically turning into "libfoo.so-i686" in the .PKGINFO file.  The details of the package should reflect what is in the PKGBUILD.

See, this is what I still don't understand. Is this libprovides, or is this "well we'll hijack your provides array sometimes depending on what you put there". It *still* hasn't been explained how this thing works in either a commit message or docs. Do I include soprovides=() in my PKGBUILD? Do I need to insert something in provides=() to get a library to show up in the final package? Is it all completely automatic- all libraries in the package get an automatic provide entry generated? This is why I am so damn frustrated with these patches.

So, to summarize:

1) Dan does not want a new PKGINFO field to be introduced for this.

Where did I say that- if I did, I'm not remembering.

2) Allan does not want any auto-generated meta-info in the provides and depends in PKGINFO. No, he stated he doesn't want *magic changes* to data the user specified in the PKGBUILD. I think he and I would be more comfortable if this was less magic.

3) Dan does not want non-versioning information in the version field in provides/depends in PKGINFO. Sure.

Where does that leave us? Essentially, we cannot get this feature implemented, as all those restrictions combined leave us with no options. Clearly, you don't see any options either, as you only provide destructive criticism, no suggestions on how you think it should be done.

Am 21.01.2011 18:35, schrieb Dan McGee:

Wow, this degenerated into a flamewar quickly.

I deleted all flames before submitting this post. That said, I was a bit pissed about that last part of Allan's post.

and stating I've only offered destructive criticism is not the truth- I just don't understand this whole thing.

That statement was directed only at the part of Allan's reply that I quoted right above the statement. It was not in any way meant for you, or to any other part of Allan's post.

On Fri, Jan 21, 2011 at 10:54 AM, Thomas Bächler wrote:

...

Am 21.01.2011 17:01, schrieb Allan McRae:

...

That reply is just wrong... i686 is not a restricted flavour of i386. It is the other way around. I can not run i686 optimised software on an i386 system. Just ask all the Via C3 owners who do not have that "nopl" instruction and the joys they had with glibc-2.12.

That is not the point. We should reflect what the dynamic linker tries to do whenever we want to load a binary:

1) If we install a i686 version of libfoo.so.2 onto a i386 system, the linker will still try to load that version (and fail with SIGILL). 2) If we install a x86_64 version and a i686 version of libfoo.so.2 into the system, and launch a 32 bit binary, the linker will determine that it cannot load the the x86_64 version, but the i686 version.

So, in order to "do it right", we only need to reflect the distinction the linker does in 2). Not even the linker makes the distinction in 1), so why should we?

And now think about use cases: The package architecture is already in PKGINFO. So the only distinction we need to make is between 32 bit and 64 bit binaries - the only case where you are able to run more than one architecture in the same system is when executing 32 bit binaries on a 64 bit platform with a backward compatibility mode for 32 bit. (AFAIK, this is not only the case for x86, but for some other platforms, too).

The only info we should include into the soprovides/depends is whether we have a 32 bit or a 64 bit binary. There shouldn't be an explicit statement about architecture.

Taking a step back here, if we want to not lock ourself into ELF-only (OSX does not use it), it sounds like using objdump due to it's use of the BFD backend which understands multiple file formats. From objdump -a, although the output is rather arcane, we should always be able to get the file format of the library, which I believe is the most important to matching what the linker does.

I do not want to lock anything explicitly to ELF. Considering that OSX also supports 32 and 64 bit libraries on the same system (and ones that contain both), it's a bad idea. The question remains, how portable can we even be here? Do we maybe need support for different binary formats, implemented in different ways?

...

...

I did not like provides=(libfoo.so) magically turning into "libfoo.so-i686" in the .PKGINFO file. The details of the package should reflect what is in the PKGBUILD.

See, this is what I still don't understand. Is this libprovides, or is this "well we'll hijack your provides array sometimes depending on what you put there".

It *still* hasn't been explained how this thing works in either a commit message or docs. Do I include soprovides=() in my PKGBUILD? Do I need to insert something in provides=() to get a library to show up in the final package? Is it all completely automatic- all libraries in the package get an automatic provide entry generated? This is why I am so damn frustrated with these patches.

The original idea was this: makepkg automatically adds provides for all libraries with a proper SONAME. These are added as provides to PKGINFO in addition to the user-specified provides. Alternatively, the PKGBUILD has to specify which libraries to add. So we would write soprovides=(libfoo.so) and this would turn into a provider for PKGINFO that would contain some information about architecture and SONAME. I don't like this option because the list of libraries is maintained by a human and not the computer. TBH, reading the current patch again, I am unsure what it actually does now. I think it is using the first option.

...

So, to summarize:

1) Dan does not want a new PKGINFO field to be introduced for this. Where did I say that- if I did, I'm not remembering.

Now that you ask, I am not 100% sure you even did. I thought I remembered something from a few months back. I apologize for not confirming this. But back to the point: Would you want this information separate? Adding a new field to PKGINFO here would essentially duplicate the whole provides/depends handling, and add new pacman code. When adding this auto-generated info to PKGINFO's provides and depends, there are no changes in pacman.

...

2) Allan does not want any auto-generated meta-info in the provides and depends in PKGINFO. No, he stated he doesn't want *magic changes* to data the user specified in the PKGBUILD. I think he and I would be more comfortable if this was less magic.

And what is "magic"? It is simply automatic. Furthermore, as far as I can see, there are no changes to the user data, but only additions to it.

...

3) Dan does not want non-versioning information in the version field in provides/depends in PKGINFO. Sure.

So, let's summarize again what we want (because it seems you don't understand that): 1) We want to have information in the PKGINFO that tells the user and pacman which libraries are included in the package, and what their SONAME versions are. 2) We also want (optional?) information in the PKGINFO about which libraries in which versions are needed to run this package. 3) We want this information automatically generated, not manually maintained. The least invasive approach in our opinion was to add this information to PKGINFO's provides and depends, as it would work without any changes to pacman. This means that the provides and depends are extended by auto-generated ones. Nothing is magically modified or removed. Now, as everything we suggested is so weird to you: How would you want such a feature implemented?

On 22/01/11 02:54, Thomas Bächler wrote:

Where does that leave us? Essentially, we cannot get this feature implemented, as all those restrictions combined leave us with no options. Clearly, you don't see any options either, as you only provide destructive criticism, no suggestions on how you think it should be done.

I only pointed out the things that I see are wrong because they are what needed fixed. Also it was 2am when I replied in between dealing with a sick child so my time for pleasantries was limited. I did not think everybodies self esteem was so low that I needed to praise the correct parts too. And how can we do this portably and subject to the restrictions given... Well, it has been established that using objdump or elfutils has issues so lets just exclude them from the start. How else can we get information about a file? file... That tells you whether a binary is 32 or 64 bit, and is already required by makepkg. So: provides=(libfoo.so) Results in %PROVIDES% libfoo.so=6-32 where pkgver is the library soname and pkgrel indicates 32/64 bit. If the "file" output does not include 32-bit or 64-bit, then just do not set the pkgrel value. That solves all comments made so far apart from the documentation needed on what this does... Allan

On 22/01/11 19:31, Florian Pritz wrote:

On 22.01.2011 01:20, Allan McRae wrote:

...

And how can we do this portably and subject to the restrictions given... Well, it has been established that using objdump or elfutils has issues so lets just exclude them from the start. How else can we get information about a file? file... That tells you whether a binary is 32 or 64 bit, and is already required by makepkg.

file doesn't tell you the library soname so I don't know what ld will use for linking. So far readelf seems to be the best choice because I can also use it to generate the dependencies (right now it seems like we are too focused on the provides). Your example below can be achieved using readelf. Also file's output doesn't have to be the same on different platforms.

What are we going to do about OSX and its lack of elf binaries?

On 22.01.2011 11:54, Xavier Chantry wrote:

On Sat, Jan 22, 2011 at 11:25 AM, Allan McRae wrote:

...

What are we going to do about OSX and its lack of elf binaries?

Who cares about soprovides/sodepends besides ArchLinux people ?

After a quick google search it looks like OS X uses .dylib so I won't even look at those files anyway since my find command only search *.so* -- Florian Pritz -- {flo,bluewind}@server-speed.net

On 22/01/11 21:25, Thomas Bächler wrote:

Am 22.01.2011 11:25, schrieb Allan McRae:

...

What are we going to do about OSX and its lack of elf binaries?

A solution that fits all platforms probably doesn't exist here. We can implement this for ELF now, and maybe someone can figure out the OS X version at some point. (.dylib is very weird IIRC, I had to look into that once).

That sounds fine. I did not realise that a different extension is used for shared libraries on OSX (although I should have thinking about it...). So just working with ELF for now seems fine. Some other things to think about. I just built a bunch of packages using soprovides/sodepends with readline providing libreadline.so. With this, on an upgrade with a soname bump we will get a whole heap of messages like: warning: provider package was selected (readline provides libreadline.so) So we need to consider whether we want to keep that message displayed, or if there is some way to only display it once (see next). I also noticed the performance hit in satisfying dependencies using provides. I wonder if we can optimise dependency checking such that we check for a given dependency only once, even if it is needed across multiple packages? That would also solve the output issue above. I'm not familiar enough with the dependency checking code to know if that is viable. Allan

On 22/01/11 22:43, Florian Pritz wrote:

On 22.01.2011 13:15, Allan McRae wrote:

...

With this, on an upgrade with a soname bump we will get a whole heap of messages like:

warning: provider package was selected (readline provides libreadline.so)

...

I also noticed the performance hit in satisfying dependencies using provides.

When I tried that in August 2009 it only displayed that message once (at least I think it did, but I'm pretty sure)

Ugh... my bad... you have to weird things to your pacman databases get this message to appear multiple times. Moving along...

and the time to resolve the dependencies for ~900 packages with (using -U; all sodeps/provides and not just readline iirc) went from 25 secs to 27 secs.

Could you stop the time so we can compare the results?

When installing two packages, changing the depend to a provide takes 8.7sec compared to 7.9sec. So about 10%, which is consistent with what you got. I'm a bit surprised it scales so well to a much large number of packages and provides. Allan

On 22/01/11 00:25, Florian Pritz wrote:

On 19.01.2011 19:54, Dan McGee wrote: <snip>

...

* introducing dependencies on readelf and objdump. At least elsewhere in the strip code, we use `file` to locate shared libraries. And you can get the SONAME bit out of objdump -p, which would at least limit this to one tool.

readelf and objdump are both in binutils, but I've noticed that objdump doesn't support the file format generated by tcc although readelf does and it's a valid ELF file afaict.

eu-objdump (elfutils) can't output that much, but it understands tcc's format and outputs pretty nice architecture information. Sadly I have no idea if that's portable.

Now I'm using readelf (binutils) and eu-objdump (elfutils). I think it's fine since both are in core and you only need them for makepkg anyway.

We have pacman/makepkg users on Linux, OSX, BSD, Hurd, Cygwin... Whatever is used here needs to be portable to all those. I do not believe elfutils is particularly portable... Allan

On 22/01/11 01:02, Florian Pritz wrote:

On 21.01.2011 15:49, Allan McRae wrote:

...

On 22/01/11 00:25, Florian Pritz wrote:

...

On 19.01.2011 19:54, Dan McGee wrote: <snip>

...

* introducing dependencies on readelf and objdump. At least elsewhere in the strip code, we use `file` to locate shared libraries. And you can get the SONAME bit out of objdump -p, which would at least limit this to one tool.

readelf and objdump are both in binutils, but I've noticed that objdump doesn't support the file format generated by tcc although readelf does and it's a valid ELF file afaict.

eu-objdump (elfutils) can't output that much, but it understands tcc's format and outputs pretty nice architecture information. Sadly I have no idea if that's portable.

Now I'm using readelf (binutils) and eu-objdump (elfutils). I think it's fine since both are in core and you only need them for makepkg anyway.

We have pacman/makepkg users on Linux, OSX, BSD, Hurd, Cygwin... Whatever is used here needs to be portable to all those. I do not believe elfutils is particularly portable...

I'd use objdump (binutils), but I don't know if it's compatible with binaries not generated by gcc (at least tcc doesn't work) and I can't really assume all users use gcc either.

There is a "portability" patch [1] for elfutils. Not sure if that is what we need though.

That portability patch seems to be mainly to add compatibility to older Linux systems, not for non-Linux systems. Allan

On 22/01/11 02:05, Yaro Kasear wrote:

On Friday, January 21, 2011 08:49:48 am Allan McRae wrote:

...

On 22/01/11 00:25, Florian Pritz wrote:

...

On 19.01.2011 19:54, Dan McGee wrote: <snip>

...

...

* introducing dependencies on readelf and objdump. At least elsewhere in the strip code, we use `file` to locate shared libraries. And you can get the SONAME bit out of objdump -p, which would at least limit this to one tool.

readelf and objdump are both in binutils, but I've noticed that objdump doesn't support the file format generated by tcc although readelf does and it's a valid ELF file afaict.

eu-objdump (elfutils) can't output that much, but it understands tcc's format and outputs pretty nice architecture information. Sadly I have no idea if that's portable.

Now I'm using readelf (binutils) and eu-objdump (elfutils). I think it's fine since both are in core and you only need them for makepkg anyway.

We have pacman/makepkg users on Linux, OSX, BSD, Hurd, Cygwin... Whatever is used here needs to be portable to all those. I do not believe elfutils is particularly portable...

Allan

Stop me if I am wrong, but don't all those platforms use ELF by default?

Yes, and... That does not make elfutils portable or fix the issue with objdump and tcc generated files. Allan