[pacman-dev] sodeps: makepkg patches
The -d/-dd patches are in master now so I'll start a new thread for the makepkg patches only. I've tested them with readline and they work fine. The final PKGINFO contains entries like these: depend = libc.so=6-64 provides = libreadline.so=6-64 Git repo is here http://git.server-speed.net/users/flo/pacman/?h=sodeps
From: Florian Pritz <bluewind@xssn.at> The user adds libaries to the provides array without a version. These must end with .so. Example: provides=(readline libreadline.so) find_soprovides() looks for .so files (not symlinks because these could point outside of pkgdir) in $pkgdir, extracts the library soname (ld links the binary to this name) and outputs provides seperated by spaces. This list contains all libraries provided by the package. Example: libfoo.so=3-64 write_pkginfo() only keeps .so provides with version information and warns the user about unneded ones. Support-by: Thomas Bächler <thomas@archlinux.org> Support-by: Christoph Schied <Christoph.Schied@uni-ulm.de> Signed-off-by: Florian Pritz <bluewind@server-speed.net> --- scripts/makepkg.sh.in | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 45 insertions(+), 1 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index b1b1b75..0d811b6 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -933,6 +933,33 @@ tidy_install() { fi } +find_soprovides() { + local soprovides + find "$pkgdir" -type f -name \*.so\* | while read filename + do + # check if we really have a shared object + if LC_ALL=C readelf -h "$filename" 2>/dev/null | grep -q '.*Type:.*DYN (Shared object file).*'; then + # 64 + soarch=$(LC_ALL=C readelf -h "$filename" | sed -n 's/.*Class.*ELF\(32\|64\)/\1/p') + # get the string binaries link to: libfoo.so.1.2 -> libfoo.so.1 + sofile=$(LC_ALL=C readelf -d "$filename" 2>/dev/null | sed -n 's/.*Library soname: \[\(.*\)\].*/\1/p') + [ -z "$sofile" ] && sofile="${$filename##*/}" + + # extract the library name: libfoo.so + soname="${sofile%%\.so\.*}.so" + # extract the major version: 1 + soversion="${sofile##*\.so\.}" + if in_array "${soname}" ${provides[@]}; then + if ! in_array "${soname}=${soversion}-${soarch}" ${soprovides[@]}; then + # libfoo.so=1-64 + echo "${soname}=${soversion}-${soarch}" + soprovides=(${soprovides[@]} "${soname}=${soversion}-${soarch}") + fi + fi + fi + done +} + write_pkginfo() { local builddate=$(date -u "+%s") if [[ -n $PACKAGER ]]; then @@ -965,10 +992,27 @@ write_pkginfo() { [[ $depends ]] && printf "depend = %s\n" "${depends[@]}" [[ $optdepends ]] && printf "optdepend = %s\n" "${optdepends[@]}" [[ $conflicts ]] && printf "conflict = %s\n" "${conflicts[@]}" - [[ $provides ]] && printf "provides = %s\n" "${provides[@]}" [[ $backup ]] && printf "backup = %s\n" "${backup[@]}" local it + + soprovides=$(find_soprovides) + provides=("${provides[@]}" ${soprovides}) + + for it in "${provides[@]}"; do + # ignore versionless entires (those come from the PKGBUILD) + if [[ $it = *.so ]]; then + # check if the entry has been found by find_soprovides + # if not, it's unneeded; tell the user so he can remove it + if [[ ! $soprovides =~ (^|\s)${it}=.* ]]; then + error "$(gettext "Can't find library listed in \$provides: %s")" "$it" + return 1 + fi + else + echo "provides = $it" + fi + done + for it in "${packaging_options[@]}"; do local ret="$(check_option $it)" if [[ $ret != "?" ]]; then -- 1.7.3.5
From: Florian Pritz <bluewind@xssn.at> The user adds libaries to the depends array without a version. These must end with .so. Example: depends=(glibc libc.so) find_sodepends() looks for ELF files (not symlinks because these could point outside of pkgdir) in $pkgdir, extracts the library sonames the binary liks to and outputs depends seperated by spaces. This list contains all libraries needed by the package. Example: libfoo.so=3-64 write_pkginfo() only keeps .so depends with version information and warns the user about unneded ones. Support-by: Thomas Bächler <thomas@archlinux.org> Support-by: Christoph Schied <Christoph.Schied@uni-ulm.de> Signed-off-by: Florian Pritz <bluewind@server-speed.net> --- scripts/makepkg.sh.in | 41 ++++++++++++++++++++++++++++++++++++++++- 1 files changed, 40 insertions(+), 1 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 0d811b6..422b4c0 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -933,6 +933,31 @@ tidy_install() { fi } +find_sodepends() { + local sodepends + find $pkgdir -type f | while read filename + do + # get architecture of the file; if soarch is empty it's not an ELF binary + soarch=$(LC_ALL=C readelf -h "$filename" 2>/dev/null | sed -n 's/.*Class.*ELF\(32\|64\)/\1/p') + [ -n "$soarch" ] || continue + # process all libraries needed by the binary + for sofile in $(LC_ALL=C readelf -d "$filename" 2>/dev/null | sed -nr 's/.*Shared library: \[(.*)\].*/\1/p') + do + # extract the library name: libfoo.so + soname="${sofile%%\.so\.*}.so" + # extract the major version: 1 + soversion="${sofile##*\.so\.}" + if in_array "${soname}" ${depends[@]}; then + if ! in_array "${soname}=${soversion}-${soarch}" ${sodepends[@]}; then + # libfoo.so=1-64 + echo "${soname}=${soversion}-${soarch}" + sodepends=(${sodepends[@]} "${soname}=${soversion}-${soarch}") + fi + fi + done + done +} + find_soprovides() { local soprovides find "$pkgdir" -type f -name \*.so\* | while read filename @@ -989,7 +1014,6 @@ write_pkginfo() { [[ $license ]] && printf "license = %s\n" "${license[@]}" [[ $replaces ]] && printf "replaces = %s\n" "${replaces[@]}" [[ $groups ]] && printf "group = %s\n" "${groups[@]}" - [[ $depends ]] && printf "depend = %s\n" "${depends[@]}" [[ $optdepends ]] && printf "optdepend = %s\n" "${optdepends[@]}" [[ $conflicts ]] && printf "conflict = %s\n" "${conflicts[@]}" [[ $backup ]] && printf "backup = %s\n" "${backup[@]}" @@ -997,7 +1021,22 @@ write_pkginfo() { local it soprovides=$(find_soprovides) + sodepends=$(find_sodepends) provides=("${provides[@]}" ${soprovides}) + depends=("${depends[@]}" ${sodepends}) + + for it in "${depends[@]}"; do + if [[ $it = *.so ]]; then + # check if the entry has been found by find_sodepends + # if not, it's unneeded; tell the user so he can remove it + if [[ ! $sodepends =~ (^|\s)${it}=.* ]]; then + error "$(gettext "Can't find library listed in \$depends: %s")" "$it" + return 1 + fi + else + echo "depend = $it" + fi + done for it in "${provides[@]}"; do # ignore versionless entires (those come from the PKGBUILD) -- 1.7.3.5
On Mon, Jan 31, 2011 at 2:15 PM, Florian Pritz <bluewind@server-speed.net>wrote:
From: Florian Pritz <bluewind@xssn.at>
The user adds libaries to the depends array without a version. These must end with .so. Example: depends=(glibc libc.so)
find_sodepends() looks for ELF files (not symlinks because these could point outside of pkgdir) in $pkgdir, extracts the library sonames the binary liks to and outputs depends seperated by spaces. This list contains all libraries needed by the package. Example: libfoo.so=3-64
write_pkginfo() only keeps .so depends with version information and warns the user about unneded ones.
Support-by: Thomas Bächler <thomas@archlinux.org> Support-by: Christoph Schied <Christoph.Schied@uni-ulm.de> Signed-off-by: Florian Pritz <bluewind@server-speed.net> --- scripts/makepkg.sh.in | 41 ++++++++++++++++++++++++++++++++++++++++- 1 files changed, 40 insertions(+), 1 deletions(-)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 0d811b6..422b4c0 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -933,6 +933,31 @@ tidy_install() { fi }
+find_sodepends() { + local sodepends + find $pkgdir -type f | while read filename + do + # get architecture of the file; if soarch is empty it's not an ELF binary + soarch=$(LC_ALL=C readelf -h "$filename" 2>/dev/null | sed -n 's/.*Class.*ELF\(32\|64\)/\1/p')
Can I suggest using a lower level approach less prone to breakage because of a change in output format? It's a little more verbose, but it directly follows the ELF standard: magic=$(dd if="$filename" bs=1 skip=1 count=3 2>/dev/null) [[ "$magic" = ELF ]] || continue read _ elfclass < <(hexdump -s5 -n1 "$filename") case $elfclass in *(0)1) soarch=32 ;; *(0)2) soarch=64 ;; esac [[ $soarch ]] || continue Note: this requires extglob, which we don't currently have enabled in makepkg.
+ [ -n "$soarch" ] || continue + # process all libraries needed by the binary + for sofile in $(LC_ALL=C readelf -d "$filename" 2>/dev/null | sed -nr 's/.*Shared library: \[(.*)\].*/\1/p')
This can be read directly from the binary with hexdump/dd as well, but requires a bit more voodoo, as you have to locate the dynamic linking section via the header and then read each null delimited linkage. 2 or more successive null bytes indicates the end of the section. If this is something we want, I can put in the legwork for it.
+ do + # extract the library name: libfoo.so + soname="${sofile%%\.so\.*}.so" + # extract the major version: 1 + soversion="${sofile##*\.so\.}" + if in_array "${soname}" ${depends[@]}; then + if ! in_array "${soname}=${soversion}-${soarch}" ${sodepends[@]}; then + # libfoo.so=1-64 + echo "${soname}=${soversion}-${soarch}" + sodepends=(${sodepends[@]} "${soname}=${soversion}-${soarch}") + fi + fi + done + done +} + find_soprovides() { local soprovides find "$pkgdir" -type f -name \*.so\* | while read filename @@ -989,7 +1014,6 @@ write_pkginfo() { [[ $license ]] && printf "license = %s\n" "${license[@]}" [[ $replaces ]] && printf "replaces = %s\n" "${replaces[@]}" [[ $groups ]] && printf "group = %s\n" "${groups[@]}" - [[ $depends ]] && printf "depend = %s\n" "${depends[@]}" [[ $optdepends ]] && printf "optdepend = %s\n" "${optdepends[@]}" [[ $conflicts ]] && printf "conflict = %s\n" "${conflicts[@]}" [[ $backup ]] && printf "backup = %s\n" "${backup[@]}" @@ -997,7 +1021,22 @@ write_pkginfo() { local it
soprovides=$(find_soprovides) + sodepends=$(find_sodepends) provides=("${provides[@]}" ${soprovides}) + depends=("${depends[@]}" ${sodepends}) + + for it in "${depends[@]}"; do + if [[ $it = *.so ]]; then + # check if the entry has been found by find_sodepends + # if not, it's unneeded; tell the user so he can remove it + if [[ ! $sodepends =~ (^|\s)${it}=.* ]]; then + error "$(gettext "Can't find library listed in \$depends: %s")" "$it" + return 1 + fi + else + echo "depend = $it" + fi + done
for it in "${provides[@]}"; do # ignore versionless entires (those come from the PKGBUILD) -- 1.7.3.5
On 31.01.2011 22:35, dave reisner wrote:
+ soarch=$(LC_ALL=C readelf -h "$filename" 2>/dev/null | sed -n 's/.*Class.*ELF\(32\|64\)/\1/p')
Can I suggest using a lower level approach less prone to breakage because of a change in output format? It's a little more verbose, but it directly follows the ELF standard:
magic=$(dd if="$filename" bs=1 skip=1 count=3 2>/dev/null) [[ "$magic" = ELF ]] || continue read _ elfclass < <(hexdump -s5 -n1 "$filename") case $elfclass in *(0)1) soarch=32 ;; *(0)2) soarch=64 ;; esac [[ $soarch ]] || continue
And I already though the current implementation is too obfuscated... If you really want to do that, please move it to a script/binary with a few options so we can just get exactly the strings we need, but keep makepkg readable. -- Florian Pritz -- {flo,bluewind}@server-speed.net
On Mon, Jan 31, 2011 at 1:15 PM, Florian Pritz <bluewind@server-speed.net> wrote:
The -d/-dd patches are in master now so I'll start a new thread for the makepkg patches only.
I've tested them with readline and they work fine.
The final PKGINFO contains entries like these: depend = libc.so=6-64 provides = libreadline.so=6-64
Git repo is here http://git.server-speed.net/users/flo/pacman/?h=sodeps
So... we still haven't looked at any cross-compatibility with this. Things that are tied heavily to specific implementation details have bitten us before; see makepkg VCS integration. Here is a non-so case that we need to at least think about how we might handle: http://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptu... "so" is nowhere to be found in there, so my first thought on this whole thing is to at least name this generically so we can extend it now and/or in the future. Can we start calling it libdepends/libprovides? I have the feeling this has been hashed/rehashed/blended before, but why aren't we just doing something like: provides=('myenv') libprovides=('libx.so' 'liby.so') To me, this seems a lot clearer and alleviates the black magic concerns of changing depends/provides arrays on the fly. -Dan
On Tue, Feb 1, 2011 at 3:53 AM, Dan McGee <dpmcgee@gmail.com> wrote:
I have the feeling this has been hashed/rehashed/blended before, but why aren't we just doing something like:
provides=('myenv') libprovides=('libx.so' 'liby.so')
To me, this seems a lot clearer and alleviates the black magic concerns of changing depends/provides arrays on the fly.
I got confused by 'myenv', its supposed to be a normal provision like 'cron' ? Anyway libprovides sounds fine to me, but the question is where do we propagate this distinction, because there is no difference for pacman. Should there also be a libprovides in PKGINFO and %LIBPROVIDES% in db, and pacman will just read these two and merge them with provides ? Or can they both be merged at the PKGINFO level ?
On 01.02.2011 03:53, Dan McGee wrote:
On Mon, Jan 31, 2011 at 1:15 PM, Florian Pritz <bluewind@server-speed.net> wrote:
The -d/-dd patches are in master now so I'll start a new thread for the makepkg patches only.
I've tested them with readline and they work fine.
The final PKGINFO contains entries like these: depend = libc.so=6-64 provides = libreadline.so=6-64
Git repo is here http://git.server-speed.net/users/flo/pacman/?h=sodeps
So... we still haven't looked at any cross-compatibility with this. Things that are tied heavily to specific implementation details have bitten us before; see makepkg VCS integration.
I don't know how to that properly. Could someone give me a few hints?
Here is a non-so case that we need to at least think about how we might handle: http://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptu...
.dylib is a different format and I have no idea what tools are available on OS X nor do I really care.
"so" is nowhere to be found in there, so my first thought on this whole thing is to at least name this generically so we can extend it now and/or in the future. Can we start calling it libdepends/libprovides?
Since they are connected we should probably just call it libdepends.
I have the feeling this has been hashed/rehashed/blended before, but why aren't we just doing something like:
provides=('myenv') libprovides=('libx.so' 'liby.so')
To me, this seems a lot clearer and alleviates the black magic concerns of changing depends/provides arrays on the fly.
Allan said he preferred a single array. I think that was the only reason. Also using the depends array allows makepkg to check if the needed libraries are installed. -- Florian Pritz -- {flo,bluewind}@server-speed.net
participants (4)
-
Dan McGee
-
dave reisner
-
Florian Pritz
-
Xavier Chantry