[pacman-dev] [RFC] makepkg: introduce .SRCINFO files for source packages
Similar to .PKGINFO, .SRCINFO provides structured metadata from the PKGBUILD to be included with source packages. The format is structured such that it contains a "pkgbase" and one to many "pkgname" sections. Each "pkgname" section represents an "output package", and inherits all of the attributes of the "pkgbase" section, and then can define their own additive fields. For example, a simple PKGBUILD: pkgbase=ponies pkgname=('applejack' 'pinkiepie') pkgver=1.2.3 pkgrel=1 arch=('x86_64' 'i686') depends=('friendship' 'magic') build() { ...; } package_applejack() { provides=('courage') ...; } package_pinkiepie() { provides=('laughter') ...; } Would yield the following .SRCINFO file: pkgbase = ponies pkgdesc = friendship is magic pkgver = 1.2.3 pkgrel = 1 arch = x86_64 arch = i686 depends = friendship depends = magic pkgname = applejack provides = courage pkgname = pinkiepie provides = laughter The code to generate this new file is taken a project which I've been incubating[0] under the guise of 'mkaurball', which creates .AURINFO files for the AUR. AURINFO is the exactly same file as .SRCINFO, but named as such to make it clear that this is specific to the AUR. Because we're parsing shell in the packaging functions rather than executing it, there *are* some limitations, but these only really crop up in more "exotic" PKGBUILDs. Smoketesting[1] for accuracy in the Arch repos yields 100% accuracy for [core] and [extra]. [community] clocks in at ~98% accuracy (.3% difference per PKGBUILD), largely due to silly haskell packages calling pacman from inside the PKGBUILD to determine dependencies. [multilib] currently shows about 92% accuracy -- a statistic which can be largely improved by utilizing the recently merged arch-specific attribute work. This is also a smaller repo so the numbers are somewhat inflated. In reality, this is only a .8% variance per PKGBUILD. Together, we can make PKGBUILD better. [0] https://github.com/falconindy/pkgbuild-introspection [1] https://github.com/falconindy/pkgbuild-introspection/blob/master/test/smoket... --- Based on Allan's recent email about the feature freeze being pushed back, I'm offering this up for discussion. scripts/makepkg.sh.in | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 8a67d94..0349e05 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -2031,6 +2031,99 @@ find_libprovides() { printf '%s\n' "${libprovides[@]}" } +srcinfo_open_section() { + printf '%s = %s\n' "$1" "$2" +} + +srcinfo_close_section() { + echo +} + +srcinfo_write_attr() { + # $1: attr name + # $2: attr values + + local attrname=$1 attrvalues=("${@:2}") + + # normalize whitespace, strip leading and trailing + attrvalues=("${attrvalues[@]//+([[:space:]])/ }") + attrvalues=("${attrvalues[@]#[[:space:]]}") + attrvalues=("${attrvalues[@]%[[:space:]]}") + + printf "\t$attrname = %s\n" "${attrvalues[@]}" +} + +pkgbuild_extract_to_srcinfo() { + # $1: pkgname + # $2: attr name + # $3: multivalued + + local pkgname=$1 attrname=$2 isarray=$3 outvalue= + + if pkgbuild_get_attribute "$pkgname" "$attrname" "$isarray" 'outvalue'; then + srcinfo_write_attr "$attrname" "${outvalue[@]}" + fi +} + +srcinfo_write_section_details() { + local attr package_arch a + local multivalued_arch_attrs=(source provides conflicts depends replaces + optdepends makedepends checkdepends) + + for attr in "${singlevalued[@]}"; do + pkgbuild_extract_to_srcinfo "$1" "$attr" 0 + done + + for attr in "${multivalued[@]}"; do + pkgbuild_extract_to_srcinfo "$1" "$attr" 1 + done + + pkgbuild_get_attribute "$1" 'arch' 1 'package_arch' + for a in "${package_arch[@]}"; do + # 'any' is special. there's no support for, e.g. depends_any. + [[ $a = any ]] && continue + + for attr in "${multivalued_arch_attrs[@]}"; do + pkgbuild_extract_to_srcinfo "$1" "${attr}_$a" 1 + done + done +} + +srcinfo_write_global() { + local singlevalued=(pkgdesc pkgver pkgrel epoch url install changelog) + local multivalued=(arch groups license checkdepends makedepends + depends optdepends provides conflicts replaces + noextract options backup + source {md5,sha{1,224,256,384,512}}sums) + + srcinfo_open_section 'pkgbase' "${pkgbase:-$pkgname}" + srcinfo_write_section_details '' + srcinfo_close_section +} + +srcinfo_write_package() { + local singlevalued=(pkgdesc url install changelog) + local multivalued=(arch groups license checkdepends depends optdepends + provides conflicts replaces options backup) + + srcinfo_open_section 'pkgname' "$1" + srcinfo_write_section_details "$1" + srcinfo_close_section +} + +write_srcinfo() { + local pkg + + printf "# Generated by makepkg %s\n" "$makepkg_version" + printf "# %s\n" "$(LC_ALL=C date -u)" + + srcinfo_write_global + + for pkg in "${pkgname[@]}"; do + srcinfo_write_package "$pkg" + done +} + write_pkginfo() { local builddate=$(date -u "+%s") if [[ -n $PACKAGER ]]; then @@ -2243,6 +2336,9 @@ create_srcpackage() { msg2 "$(gettext "Adding %s...")" "$BUILDSCRIPT" ln -s "${BUILDFILE}" "${srclinks}/${pkgbase}/${BUILDSCRIPT}" + msg2 "$(gettext "Generating %s file...")" .SRCINFO + write_srcinfo > "$srclinks/$pkgbase"/.SRCINFO + local file all_sources get_all_sources 'all_sources' -- 2.1.2
On 18/10/14 11:59, Dave Reisner wrote:
Similar to .PKGINFO, .SRCINFO provides structured metadata from the PKGBUILD to be included with source packages.
The format is structured such that it contains a "pkgbase" and one to many "pkgname" sections. Each "pkgname" section represents an "output package", and inherits all of the attributes of the "pkgbase" section, and then can define their own additive fields.
For example, a simple PKGBUILD:
pkgbase=ponies pkgname=('applejack' 'pinkiepie') pkgver=1.2.3 pkgrel=1 arch=('x86_64' 'i686') depends=('friendship' 'magic')
build() { ...; }
package_applejack() { provides=('courage')
...; }
package_pinkiepie() { provides=('laughter')
...; }
Would yield the following .SRCINFO file:
pkgbase = ponies pkgdesc = friendship is magic pkgver = 1.2.3 pkgrel = 1 arch = x86_64 arch = i686 depends = friendship depends = magic
pkgname = applejack provides = courage
pkgname = pinkiepie provides = laughter
Having been forced to watch this by my daughter, I still do not understand the appeal of ponies...
The code to generate this new file is taken a project which I've been incubating[0] under the guise of 'mkaurball', which creates .AURINFO files for the AUR. AURINFO is the exactly same file as .SRCINFO, but named as such to make it clear that this is specific to the AUR.
Because we're parsing shell in the packaging functions rather than executing it, there *are* some limitations, but these only really crop up in more "exotic" PKGBUILDs. Smoketesting[1] for accuracy in the Arch repos yields 100% accuracy for [core] and [extra]. [community] clocks in at ~98% accuracy (.3% difference per PKGBUILD), largely due to silly haskell packages calling pacman from inside the PKGBUILD to determine dependencies.
WTF!
[multilib] currently shows about 92% accuracy -- a statistic which can be largely improved by utilizing the recently merged arch-specific attribute work. This is also a smaller repo so the numbers are somewhat inflated. In reality, this is only a .8% variance per PKGBUILD.
Having not even looked at your patch at this stage... Do we detect when this fails? Could we detect it at the .PKGINFO construction stage and print a warning?
Together, we can make PKGBUILD better.
[0] https://github.com/falconindy/pkgbuild-introspection [1] https://github.com/falconindy/pkgbuild-introspection/blob/master/test/smoket... --- Based on Allan's recent email about the feature freeze being pushed back, I'm offering this up for discussion.
This was something that had been flagged as potential for inclusion anyway. And I want this in for what I have planned post release... My usual test PKGBUILD: PKGBUILD: pkgname='foobar' pkgver=1 pkgrel=1 arch=('i686' 'x86_64') source=('tmp.txt') source_x86_64+=('foo.bar') md5sums=('d41d8cd98f00b204e9800998ecf8427e') md5sums_x86_64=('d41d8cd98f00b204e9800998ecf8427e') .SRCINFO # Generated by makepkg 4.1.2-468-gf74e # Sun Oct 19 11:22:00 UTC 2014 pkgbase = foobar pkgver = 1 pkgrel = 1 epoch = 0 arch = i686 arch = x86_64 checkdepends = makedepends = depends = optdepends = provides = conflicts = replaces = source = tmp.txt md5sums = d41d8cd98f00b204e9800998ecf8427e source_x86_64 = foo.bar pkgname = foobar So there are some bugs: - epoch = 0 (should be not printed) - Lots of empty values - No md5sum_x86_64 Otherwise, all good. Small query:
+ +srcinfo_write_attr() { + # $1: attr name + # $2: attr values + + local attrname=$1 attrvalues=("${@:2}") + + # normalize whitespace, strip leading and trailing + attrvalues=("${attrvalues[@]//+([[:space:]])/ }") + attrvalues=("${attrvalues[@]#[[:space:]]}") + attrvalues=("${attrvalues[@]%[[:space:]]}")
Make all whitespace a single space then strip whitespace from start and finish? What is the first step for? Allan
On Sun, Oct 19, 2014 at 09:28:12PM +1000, Allan McRae wrote:
Having been forced to watch this by my daughter, I still do not understand the appeal of ponies...
I don't even care about the show anymore, I just like using ponies for test data (*especially* at work).
Having not even looked at your patch at this stage... Do we detect when this fails? Could we detect it at the .PKGINFO construction stage and print a warning?
Not sure if this is a typo. There won't be a .PKGINFO built at the same time as a .SRCINFO so we can't diff them in any way. We could possibly detect it at .SRCINFO construction time for some subset of the failures. My take on parsing problems: There's a bunch of unit tests which highlight known features and deficiencies (see the expectedFailure decorated methods): https://github.com/falconindy/pkgbuild-introspection/blob/master/test/testca... There's probably others. I think only HandlesShellVarInPackageAttr is worth putting effort into fixing/detecting. We can detect *some* cases of it by calling out empty attributes. As for the rest -- we can largely "fix" these by shaming people and asking for patches and/or more tests. Speaking of tests, I have to remind you about this patch (or at least the idea of this patch): https://lists.archlinux.org/pipermail/pacman-dev/2014-August/019261.html Merging something like that would allow me to at least move the existing tests that I've written against this code into the pacman repo. We'd then be in a much better position to accept patches against the SRCINFO generation code and be less afraid of regressions.
This was something that had been flagged as potential for inclusion anyway. And I want this in for what I have planned post release...
My usual test PKGBUILD:
PKGBUILD: pkgname='foobar' pkgver=1 pkgrel=1 arch=('i686' 'x86_64') source=('tmp.txt') source_x86_64+=('foo.bar') md5sums=('d41d8cd98f00b204e9800998ecf8427e') md5sums_x86_64=('d41d8cd98f00b204e9800998ecf8427e')
.SRCINFO # Generated by makepkg 4.1.2-468-gf74e # Sun Oct 19 11:22:00 UTC 2014 pkgbase = foobar pkgver = 1 pkgrel = 1 epoch = 0 arch = i686 arch = x86_64 checkdepends = makedepends = depends = optdepends = provides = conflicts = replaces = source = tmp.txt md5sums = d41d8cd98f00b204e9800998ecf8427e source_x86_64 = foo.bar
pkgname = foobar
So there are some bugs: - epoch = 0 (should be not printed)
I didn't see this in mkaurball because it operates in a much cleaner environment. makepkg explicitly does this: # set defaults if they weren't specified in buildfile pkgbase=${pkgbase:-${pkgname[0]}} epoch=${epoch:-0} basever=$(get_full_version) ...which is where an epoch of 0 comes from. I'm sure I can fix this, but I'm punting it to another patch.
- Lots of empty values
Porting error -- just needed to localize a var in extract_global_var.
- No md5sum_x86_64
Missing from the arch-specific list. An oversight that exists in mkaurball, too!
Otherwise, all good. Small query:
+ +srcinfo_write_attr() { + # $1: attr name + # $2: attr values + + local attrname=$1 attrvalues=("${@:2}") + + # normalize whitespace, strip leading and trailing + attrvalues=("${attrvalues[@]//+([[:space:]])/ }") + attrvalues=("${attrvalues[@]#[[:space:]]}") + attrvalues=("${attrvalues[@]%[[:space:]]}")
Make all whitespace a single space then strip whitespace from start and finish? What is the first step for?
We do something similar for .PKGINFO (but without steps 2 and 3 which we should probably do there, too). Consider some busted ass shit like this: pkgdesc=" ponies of the world unite!!! " Step 1 normalizes the whitespace, yielding this: pkgdesc=" ponies of the world unite!!! " Steps 2/3 get rid of the remaining leading/trailing. I've thought about making this a lint error, but I think it's a bit too esoteric. Let's just clean it up for the user. d
On 19/10/14 23:33, Dave Reisner wrote:
Speaking of tests, I have to remind you about this patch (or at least the idea of this patch):
https://lists.archlinux.org/pipermail/pacman-dev/2014-August/019261.html
I'm sure I said this was fine as an interim solution (with the final goal to have functions split out into logical groups). And then I have proceeded to ignore it! I will try and remember to look at this in the next few days. A
On 19/10/14 23:33, Dave Reisner wrote:
On Sun, Oct 19, 2014 at 09:28:12PM +1000, Allan McRae wrote:
Having been forced to watch this by my daughter, I still do not understand the appeal of ponies...
I don't even care about the show anymore, I just like using ponies for test data (*especially* at work).
Having not even looked at your patch at this stage... Do we detect when this fails? Could we detect it at the .PKGINFO construction stage and print a warning?
Not sure if this is a typo. There won't be a .PKGINFO built at the same time as a .SRCINFO so we can't diff them in any way. We could possibly detect it at .SRCINFO construction time for some subset of the failures.
I was thinking aloud. The .SRCINFO could be readily built when the .PKGINFO is build and a comparison made. As you point out, not the other way. This somewhat fits into the shaming people who have bad PKGBUILDs approach...
Similar to .PKGINFO, .SRCINFO provides structured metadata from the PKGBUILD to be included with source packages. The format is structured such that it contains a "pkgbase" and one to many "pkgname" sections. Each "pkgname" section represents an "output package", and inherits all of the attributes of the "pkgbase" section, and then can define their own additive fields. For example, a simple PKGBUILD: pkgbase=ponies pkgname=('applejack' 'pinkiepie') pkgver=1.2.3 pkgrel=1 arch=('x86_64' 'i686') depends=('friendship' 'magic') build() { ...; } package_applejack() { provides=('courage') ...; } package_pinkiepie() { provides=('laughter') ...; } Would yield the following .SRCINFO file: pkgbase = ponies pkgdesc = friendship is magic pkgver = 1.2.3 pkgrel = 1 arch = x86_64 arch = i686 depends = friendship depends = magic pkgname = applejack provides = courage pkgname = pinkiepie provides = laughter The code to generate this new file is taken a project which I've been incubating[0] under the guise of 'mkaurball', which creates .AURINFO files for the AUR. AURINFO is the exactly same file as .SRCINFO, but named as such to make it clear that this is specific to the AUR. Because we're parsing shell in the packaging functions rather than executing it, there *are* some limitations, but these only really crop up in more "exotic" PKGBUILDs. Smoketesting[1] for accuracy in the Arch repos yields 100% accuracy for [core] and [extra]. [community] clocks in at ~98% accuracy (.3% difference per PKGBUILD), largely due to silly haskell packages calling pacman from inside the PKGBUILD to determine dependencies. [multilib] currently shows about 92% accuracy -- a statistic which can be largely improved by utilizing the recently merged arch-specific attribute work. This is also a smaller repo so the numbers are somewhat inflated. In reality, this is only a .8% variance per PKGBUILD. Together, we can make PKGBUILD better. [0] https://github.com/falconindy/pkgbuild-introspection [1] https://github.com/falconindy/pkgbuild-introspection/blob/master/test/smoket... --- v2: - Avoid emitting empty list values - Treat md5sum as architecture-specific scripts/makepkg.sh.in | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 98 insertions(+), 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 86f8a77..9614069 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -2031,6 +2031,100 @@ find_libprovides() { printf '%s\n' "${libprovides[@]}" } +srcinfo_open_section() { + printf '%s = %s\n' "$1" "$2" +} + +srcinfo_close_section() { + echo +} + +srcinfo_write_attr() { + # $1: attr name + # $2: attr values + + local attrname=$1 attrvalues=("${@:2}") + + # normalize whitespace, strip leading and trailing + attrvalues=("${attrvalues[@]//+([[:space:]])/ }") + attrvalues=("${attrvalues[@]#[[:space:]]}") + attrvalues=("${attrvalues[@]%[[:space:]]}") + + printf "\t$attrname = %s\n" "${attrvalues[@]}" +} + +pkgbuild_extract_to_srcinfo() { + # $1: pkgname + # $2: attr name + # $3: multivalued + + local pkgname=$1 attrname=$2 isarray=$3 outvalue= + + if pkgbuild_get_attribute "$pkgname" "$attrname" "$isarray" 'outvalue'; then + srcinfo_write_attr "$attrname" "${outvalue[@]}" + fi +} + +srcinfo_write_section_details() { + local attr package_arch a + local multivalued_arch_attrs=(source provides conflicts depends replaces + optdepends makedepends checkdepends + {md5,sha{1,224,256,384,512}}sums) + + for attr in "${singlevalued[@]}"; do + pkgbuild_extract_to_srcinfo "$1" "$attr" 0 + done + + for attr in "${multivalued[@]}"; do + pkgbuild_extract_to_srcinfo "$1" "$attr" 1 + done + + pkgbuild_get_attribute "$1" 'arch' 1 'package_arch' + for a in "${package_arch[@]}"; do + # 'any' is special. there's no support for, e.g. depends_any. + [[ $a = any ]] && continue + + for attr in "${multivalued_arch_attrs[@]}"; do + pkgbuild_extract_to_srcinfo "$1" "${attr}_$a" 1 + done + done +} + +srcinfo_write_global() { + local singlevalued=(pkgdesc pkgver pkgrel epoch url install changelog) + local multivalued=(arch groups license checkdepends makedepends + depends optdepends provides conflicts replaces + noextract options backup + source {md5,sha{1,224,256,384,512}}sums) + + srcinfo_open_section 'pkgbase' "${pkgbase:-$pkgname}" + srcinfo_write_section_details '' + srcinfo_close_section +} + +srcinfo_write_package() { + local singlevalued=(pkgdesc url install changelog) + local multivalued=(arch groups license checkdepends depends optdepends + provides conflicts replaces options backup) + + srcinfo_open_section 'pkgname' "$1" + srcinfo_write_section_details "$1" + srcinfo_close_section +} + +write_srcinfo() { + local pkg + + printf "# Generated by makepkg %s\n" "$makepkg_version" + printf "# %s\n" "$(LC_ALL=C date -u)" + + srcinfo_write_global + + for pkg in "${pkgname[@]}"; do + srcinfo_write_package "$pkg" + done +} + write_pkginfo() { local builddate=$(date -u "+%s") if [[ -n $PACKAGER ]]; then @@ -2243,6 +2337,9 @@ create_srcpackage() { msg2 "$(gettext "Adding %s...")" "$BUILDSCRIPT" ln -s "${BUILDFILE}" "${srclinks}/${pkgbase}/${BUILDSCRIPT}" + msg2 "$(gettext "Generating %s file...")" .SRCINFO + write_srcinfo > "$srclinks/$pkgbase"/.SRCINFO + local file all_sources get_all_sources 'all_sources' @@ -2379,7 +2476,7 @@ extract_global_var() { # $2: multivalued # $3: name of output var - local attr=$1 isarray=$2 outputvar=$3 + local attr=$1 isarray=$2 outputvar=$3 ref if (( isarray )); then array_build ref "$attr" -- 2.1.3
participants (3)
-
Allan McRae
-
Dave Reisner
-
Dave Reisner