On Tue, Feb 2, 2010 at 3:48 PM, Florian Pritz wrote:

...

On 01/22/2010 09:19 PM, Florian Pritz wrote:

...

Hi,

Regarding the current mass rebuilds because of libpng I'd like to suggest implementing so dependencies again.

Bump.

I'd really like to get some comments regarding this patch.

Nagy, you complained about libpng/libjpeg rebuilds yesterday. Are you interested in this ?

Well, I am not good at script programming, so I assume that this patch is OK. :-) In my opinion, even if ArchLinux doesn't switch to sodepends/soprovides, it would be useful, if makepkg supported this feature. (I like this, because pacman doesn't need to be tweaked. ;-) This is not the competent ML to decide if ArchLinux's repos should use these flags, personally I have some concerns about db.tar.gz size. (That needs testing, which requires patched makepkg, too.) Bye

On 02/02/2010 09:13 PM, Nagy Gabor wrote:

...

This is not the competent ML to decide if ArchLinux's repos should use these flags, personally I have some concerns about db.tar.gz size. (That needs testing, which requires patched makepkg, too.)

I've repacked core and the results are quite good:

core.db.tar.gz from 35KB to 40KB unpacked db from 2.1MB to 2.2MB

Here is a little explanation for this nice result: Florian's patch puts a library into sodepends iff the .so file is owned by one of its _direct_ dependencies (optdepends are omitted). That is why the "sodepends" array is shorter than expected (by me). You may say this method is not as safe as it should be, but imho it is safe enough (if the user doesn't use the -d switch, of course): For example, yesterday I broke all my gtk2 packages due to my custom cairo build (I should have rebuild it with recent libpng). With soprovides/sodepends enabled, pacman would catch the broken cairo-cleartype->libpng dependency in the firefox->gtk2->cairo->libpng dependency chain. So in sum, sodepends/soprovides trick is just ensures that all "normal" dependencies are not broken in .so level neither. Moreover, due to this feature, distros can introduce sodepends/soprovides step-by-step (without rebuilding all packages in the repo), or they can limit this feature to some "critical" packages. Bye

On 03/02/10 09:33, Nagy Gabor wrote:

...

...

On 02/02/2010 09:13 PM, Nagy Gabor wrote:

...

This is not the competent ML to decide if ArchLinux's repos should use these flags, personally I have some concerns about db.tar.gz size. (That needs testing, which requires patched makepkg, too.)

I've repacked core and the results are quite good:

core.db.tar.gz from 35KB to 40KB unpacked db from 2.1MB to 2.2MB

Here is a little explanation for this nice result:

Florian's patch puts a library into sodepends iff the .so file is owned by one of its _direct_ dependencies (optdepends are omitted). That is why the "sodepends" array is shorter than expected (by me).

You may say this method is not as safe as it should be, but imho it is safe enough (if the user doesn't use the -d switch, of course): For example, yesterday I broke all my gtk2 packages due to my custom cairo build (I should have rebuild it with recent libpng). With soprovides/sodepends enabled, pacman would catch the broken cairo-cleartype->libpng dependency in the firefox->gtk2->cairo->libpng dependency chain. So in sum, sodepends/soprovides trick is just ensures that all "normal" dependencies are not broken in .so level neither.

Moreover, due to this feature, distros can introduce sodepends/soprovides step-by-step (without rebuilding all packages in the repo), or they can limit this feature to some "critical" packages.

Basically, +1 for implementing this to makepkg and use this in ArchLinux's repos. (Although this ML is not competent ;-)

I can see only one edge case that should be addressed (that's why I'm writing this e-mail): Dependencies can be satisfied via provision. What to do then? (That is not handled at all atm.)

I will have a decent look into that patch later, but I still think it is a bit of a false sense of security. e.g readline-6.0 and readline-6.1 both provide libreadline.so.6. However, building bash against readline-6.1 and running against readline-6.0 results in a non-booting system... (Trust me, I got the bug reports). If this approach does not work for even one of the most critical pair of packages on a system, then I do not see what we are gaining. Finally, this would prevent proper upgrades to partially complete rebuilds. That is fine for some users who would like such things, but the people doing and testing a rebuild would have issues. I upgraded the libpng/libjpeg rebuild packages while there were still several packages needing to be rebuilt for my system. With this patch, I would be required to do an incredibly stupid "pacman -Syud". Allan

Am Mittwoch, 3. Februar 2010 10:27:43 schrieb Thomas Bächler:

I get the feeling that you are desperately looking for reason to not implement a feature that is obviously a necesity.

I like the idea of adding sodeps and soprovides. Ideally they should be even added to the db files so tools could use this information to create a rebuild list or a global dependency check. What I really don't like is magically adding them to existing depends and provide arrays. They should stay in their own arrays. The point where you added the soname-prefix should have told you that you are probably doing some kind of evil hack here. Here are my arguments why adding those sodeps to existing depends is bad; the idea in general is not: In addition to Allan's arguments one has to see that theses sodeps just cannot be 100% safe. And even if they would work for 90% of our packages the left 10% would possibly break the other due the nature of a dependency tree. E.g. you are just providing the soname of any file that is included in a package. But you have no idea if other apps will load this library at runtime or if the linker does pick it up. For example there are file with the same so names in different directories (I think I have seen same names in kdelibs3 and kdelibs). Some apps bring their own version of a lib like jre has its own libjpeg. Sure, you could disable soprovides for known "broken" packages. But let's face it: we will have such packages in our repos that break everything else due to such false provides and deps; especially if the maintainer does not see what happens here and has no real control over it. My suggestion is: implement this as separete sodeps and soprovides arrays without the soname-prefix and let's write tools to make use of it. And maybe we could teach pacman using those information, too. E.g. it could issue a warning if it found a broken sodep. Pierre -- Pierre Schmitz, https://users.archlinux.de/~pierre

On 04/02/10 22:54, Thomas Bächler wrote:

However, it's worthless to discuss implementation and concepts if the idea is entirely rejected by our main makepkg developer.

It is not entirely rejected. I just have very limited time at the moment so end up just pointing out what I do not like about ideas. I actually do like some of the idea, but there are parts I obviously dislike. In future I'll put some effort into being clearer about that... Anyway, here are some of my opinions on this idea: 1) I'd prefer just using a single depends array rather than an additional sodepends array. A version of pacman's "-d" ignoring dependency versioning would not require a separate array and would be far more useful than one that ignored sodepends. I think that should be implemented in pacman no matter what. 2) I do not like "magically" adding libraries to the depends/provides array. I really want them to be manually specified (versions can be automatic - see #3). For example, the readline package would want to provide libreadline.so as that is important but libhistory.so is really not... What is important or not would be up to the distro packagers. 3) I would like the sodeps to be listed like (e.g) "libreadline.so". This makes the dependency named closer to what is actually is. Makepkg could recognize the ".so" at the end and use readelf on the binaries and automatically add the relevant version. The "soname-arch" type prefix is ugly. "soname" is covered by the ".so" and multi-lib stuff is not really as critical so "arch" is not really needed either. These three points would go a long way to alleviate my concerns about this. From my suggestions it should be clear that I really do not want every package to have all possible so-provides and -depends added automatically. That seems too much like an even poorer implementation of rpm to me and would be very, very difficult to get around all the edge cases. Allan

On 02/04/2010 04:38 PM, Thomas Bächler wrote:

I disagree in the "arch" part. It might not be as important now, but in the future it could cause problems if we end up havnig multiple architecture libraries in one system (and ultimately I want to make that possible, if I ever get to it). If we could agree on this scheme:

soprovides=(libreadline.so) would generate provides=("${provides[@]}" libreadline.so-x86_64=6)

Then the "weak -d" flag could ignore the =6 version above and we still have everything we would need for matching so-names.

...

From my suggestions it should be clear that I really do not want every package to have all possible so-provides and -depends added automatically. That seems too much like an even poorer implementation of rpm to me and would be very, very difficult to get around all the edge cases. Currently both sodepends and soprovides have to be specified in the PKGBUILD: sodepends=(libreadline.so libc.so) soprovides=(libreadline.so)

Versions and arch will be added afterwards and the final line in .PKGINFO looks like this: provides = libreadline.so-elf64_x86_64=6 For the arch part a better way to find the arch of the lib would be nice. Neither readelf nor objdump seem to fit. Readelf only shows 32/64 and objdump shows i686 libs as i368. If anyone has an idea please share. -- Florian Pritz -- {flo,bluewind}@server-speed.net

On 06/02/10 10:00, Florian Pritz wrote:

+#-- soprovides:add .so files to provides array # -OPTIONS=(strip docs libtool emptydirs zipman purge) +OPTIONS=(strip docs libtool emptydirs zipman purge soprovides)

Just a small comment on this. Do we really need the new option here? I think that someone specifying a library so name in the depends/provides is enough to indicate that they want to do this. Allan

...

I disagree in the "arch" part. It might not be as important now, but in the future it could cause problems if we end up havnig multiple architecture libraries in one system (and ultimately I want to make that possible, if I ever get to it). If we could agree on this scheme:

soprovides=(libreadline.so) would generate provides=("${provides[@]}" libreadline.so-x86_64=6)

Then the "weak -d" flag could ignore the =6 version above and we still have everything we would need for matching so-names.

OK. I will concede the need for the arch part then. But it needs some changing. As I said before, I would prefer soprovides detected by ending in ".so" in the provides array and I do not really like entries in the provides array automatically changing apart from versioning. So...

provides=(foobar libfoo.so)

would result in

provides = foobar provides = libfoo.so=6-x86_64 (does that order look right...?)

in the .PKGINFO file.

I am fairly sure that pacman can handle two packages providing different versions of libfoo.so but that needs checked.

A patch that did this would be very acceptable to me.

Allan

What arch are we talking about here. I ran into this inconsistency when I used pacman/makepkg in FreeBSD. makepkg uses CARCH , pacman.conf uses "uname -m" which returns amd64 not x86_64 on FreeBSD , and If I'm looking at the right patch "readelf -h" is used here which will return ELF32 in 32bin-* and 32lib-* packages and that is technically correct and an intended behaviour (I think).

On 11/02/10 04:06, Florian Pritz wrote:

On 02/10/2010 06:01 PM, Nezmer wrote:

...

What arch are we talking about here.

I ran into this inconsistency when I used pacman/makepkg in FreeBSD. makepkg uses CARCH , pacman.conf uses "uname -m" which returns amd64 not x86_64 on FreeBSD , and If I'm looking at the right patch "readelf -h" is used here which will return ELF32 in 32bin-* and 32lib-* packages and that is technically correct and an intended behaviour (I think).

I'm now using objdump to be able to tell apart ARM and x86 because readelf returns ELF32 for both, but I can use sort of everything. uname doesn't help because you could crosscompile 32bit libraries on a 64bit host.

Who is going to be mixing ARM and x86 binaries on the same system... cross-compiler libraries maybe? I do not think that is a situation we really need to worry about. Allan

On 02/10/2010 07:02 PM, Florian Pritz wrote:

On 02/10/2010 07:08 AM, Allan McRae wrote:

...

provides=(foobar libfoo.so)

would result in

provides = foobar provides = libfoo.so=6-x86_64 (does that order look right...?)

in the .PKGINFO file.

I am fairly sure that pacman can handle two packages providing different versions of libfoo.so but that needs checked. I did a quick test and it seems to work.

The attached two patches should fit. -- Florian Pritz -- {flo,bluewind}@server-speed.net

On Thursday 25 February 2010 04:34:06 Allan McRae wrote:

I have had a quick look at these today and there are still a couple of things I would like to address. but they are getting much closer to what I think they need to be.

I have put them on a "sodeps" branch of my pacman repo so they do not get lost and they will consistently remind me to take a decent look at them...

Any news on this? As you already knows, there was a breakage of KDEmod because of the openssl update lately, that could have been prevented with soname dependencies. ;) Bye...Frank

1) I'd prefer just using a single depends array rather than an additional sodepends array. A version of pacman's "-d" ignoring dependency versioning would not require a separate array and would be far more useful than one that ignored sodepends. I think that should be implemented in pacman no matter what.

I don't really like introducing sodepends/soprovides array neither, but I am not 100% against it neither. I like versioned soprovides/sodepends, but it looks very hackish. :-)

2) I do not like "magically" adding libraries to the depends/provides array. I really want them to be manually specified (versions can be automatic - see #3). For example, the readline package would want to provide libreadline.so as that is important but libhistory.so is really not... What is important or not would be up to the distro packagers.

In order to get this work without pain, the patch requires a slight modification. If you want to limit soprovides to important .so files, then the requiredby packages' (so)depends array should be also filled in manually or with care. It would be packager-friendly to leave this work to makepkg, and then it should check the (so)provides array of .so-provider dependency. This modification is also needed in order to make my earlier statement true (sorry for my mistake): "Moreover, due to this feature, distros can introduce sodepends/soprovides step-by-step (without rebuilding all packages in the repo), or they can limit this feature to some "critical" packages." Bye

On Wed, Feb 3, 2010 at 5:50 PM, Nagy Gabor wrote:

Packagers should know what they are doing... Moreover, I can also support adding some more sophisticated dependency switches (maybe hidden from end-users), if needed. (Personally, I don't like that I can't turn on resolvedeps with -d.

I was wondering how bad -Sud was and this is exactly what came to mind. -d is too extreme as it doesn't resolve dependencies. If it did resolve dependencies but didnt check for breakages, it could be a better option.