On Sun, 12 Jan 2014 at 15:20:03, Sébastien Luttringer wrote:

On 12/01/2014 14:35, Lukas Fleischer wrote:

...

Hi,

I think we should promote the use of .AURINFO files. Currently, only a very small fraction of packages use it. A basic description of its format can be found in the commit message of AUR commit 5a11373 [1]. Regardless of whether we keep that format or use something entirely different for metadata later, it is good to have this information stored somewhere and the format is simple enough to migrate to another format later.

Hi,

I read the commit info but why do we need another file? What's the purpose of overriding PKGBUILD variables in AUR?

* It is impossible to extract information from a PKGBUILD properly in general (for example if there are conflicting package descriptions for different architectures, different dependencies depending on machine-specific stuff), the user is the only one to decide what to display on the AUR in these cases. * It is hard to extract information from a PKGBUILD when common bash features (variable substitutions etc.) are used -- basically requires "execution" of the PKGBUILD -- this is better done on the client side. * The current PKGBUILD parser in the AUR is extremely hackish and should be removed.

Cheers,

-- Sébastien "Seblu" Luttringer https://seblu.net GPG: 0x2072D77A

On Sun, 12 Jan 2014 at 15:51:48, Anton Larionov wrote:

Hello,

I was under the impression that .AURINFO was introduced to override some fields in PKGBUILD when they are written in format which can't be properly displayed by AUR (or maybe I've missed something). But why do you want to force it's usage for all packages? In most cases AURINFO will just duplicate same fields from PKGBUILD.

The long-term plan is to use it for all AUR packages, improve the format and finally make it an official feature of makepkg(8) (the file will probably be called .SRCINFO then but we're far from there). See my other reply to Sebastien for some reasons on why it should be used.

Also I have some questions about it's format:

1) If package has different dependencies for 86_64 and 686, what should I put in depend array?

Good question. This cannot be answered properly, though, since dependencies actually are a property of the binary package and not a property of the source package. Maybe we should loosen the format for dependencies of source packages and allow optdep-like comments? Something like: depends = foo depends = bar depends = foobar: x86_64 only Just an idea. Comments welcome.

2) Which 'pkgname' will be unique - from PKGBUILD or AURINFO? E.g if I upload package with name 'foo' and overriden name 'bar' will someone be able to upload new package with name 'foo'? Or 'bar'?

Only the information from .AURINFO will be used. You can already trick the AUR into reading a completely different name from the PKGBUILD than it actually produces (and that problem is unavoidable), so that isn't a (new) issue.

--

Regards, Anton Larionov

On Sun, Jan 12, 2014 at 3:11 PM, Lukas Fleischer wrote:

On Sun, 12 Jan 2014 at 15:51:48, Anton Larionov wrote:

...

Hello,

I was under the impression that .AURINFO was introduced to override some fields in PKGBUILD when they are written in format which can't be properly displayed by AUR (or maybe I've missed something). But why do you want to force it's usage for all packages? In most cases AURINFO will just duplicate same fields from PKGBUILD.

The long-term plan is to use it for all AUR packages, improve the format and finally make it an official feature of makepkg(8) (the file will probably be called .SRCINFO then but we're far from there). See my other reply to Sebastien for some reasons on why it should be used.

So the official goal is to have it generated as part of makepkg -S? Because I see that as the only way the format will get popular: if it's nobody's problem. J. Leclanche

...

Also I have some questions about it's format:

1) If package has different dependencies for 86_64 and 686, what should I put in depend array?

Good question. This cannot be answered properly, though, since dependencies actually are a property of the binary package and not a property of the source package. Maybe we should loosen the format for dependencies of source packages and allow optdep-like comments? Something like:

depends = foo depends = bar depends = foobar: x86_64 only

Just an idea. Comments welcome.

...

2) Which 'pkgname' will be unique - from PKGBUILD or AURINFO? E.g if I upload package with name 'foo' and overriden name 'bar' will someone be able to upload new package with name 'foo'? Or 'bar'?

Only the information from .AURINFO will be used. You can already trick the AUR into reading a completely different name from the PKGBUILD than it actually produces (and that problem is unavoidable), so that isn't a (new) issue.

...

--

Regards, Anton Larionov

On Sun, 12 Jan 2014 at 23:49:39, Justin Dray wrote:

[...] It might also be a good idea to write out what fields are available and their purpose on the wiki similar to the PKGBUILD page ( https://wiki.archlinux.org/index.php/PKGBUILD) and perhaps link to it from the AUR user guidelines page? It will be forgotten by most packagers if the only information about it is a commit message and a mailing list thread.

That is what I suggested in the initial mail.

Regards, Justin Dray E: justin@dray.be M: 0433348284

On Mon, 13 Jan 2014 at 02:36:03, Jerome Leclanche wrote:

I don't fully understand the logic in currently having an .AURINFO instead of defining the .SRCINFO format right now and have a tool generating those for the time being (since it'll likely be the same format as .PKGINFO). It can go in makepkg later.

Because the requirements for such a .SRCINFO are much higher than the requirements for what we need in the AUR. For example, it needs to support split packages. Feel free to come up with a format that is general enough to be accepted as official .SRCINFO and send patches to pacman-dev, though.

J. Leclanche

On Sun, Jan 12, 2014 at 11:12 PM, Lukas Fleischer wrote:

...

On Sun, 12 Jan 2014 at 23:49:39, Justin Dray wrote:

...

[...] It might also be a good idea to write out what fields are available and their purpose on the wiki similar to the PKGBUILD page ( https://wiki.archlinux.org/index.php/PKGBUILD) and perhaps link to it from the AUR user guidelines page? It will be forgotten by most packagers if the only information about it is a commit message and a mailing list thread.

That is what I suggested in the initial mail.

...

Regards, Justin Dray E: justin@dray.be M: 0433348284

On Mon, 13 Jan 2014 at 14:24:58, Martti Kühne wrote:

It may sound like two dumb or unrelated ideas, but I figure they would become relevant to the discussion at one point anyway. So, let's look at the question we're avoiding all the time: What is the problem with PKGBUILDs? That it's not limited to a subset of sh?

It cannot be parsed without full execution. Theoretically, a PKGBUILD can contain something like pkgname="$(echo foo | sed 's/o/a/' | awk '{ print "b", $0, "r" }' | tr -d ' fo')" and something even fancier.

There seem to be regularly efforts appearing and disappearing to clean that mess up on the initial side of the problem, which could render this .AURINFO / .SRCINFO ballet unnecessary.

How about creating a parseable and functionally equivalent "PKGBUILD.gen" from PKGBUILD?

So, you're basically evaluating all PKGBUILD variables on the client side, then print them into a file called "PKGBUILD.gen"? That's exactly what .AURINFO is, except it has another file name.

cheers! mar77i

On Mon, Jan 13, 2014 at 6:03 PM, Martti Kühne wrote:

[...] You have a full egg with .AURINFO

at least, in an ideal world, you *could*, and probably *should* have an AUR-normalized PKGBUILD instead of plethoras of more files that all do all-too similar things for all-too similar tools. cheers! mar77i

On Sun, 12 Jan 2014 at 21:17:29, Jerome Leclanche wrote:

[...] So the official goal is to have it generated as part of makepkg -S?

Yes, but the requirements for makepkg(8) inclusion are much higher than what we need for the AUR and it doesn't look like there is someone who wants to deal with them right now.

Because I see that as the only way the format will get popular: if it's nobody's problem.

So having to invoke `makepkg --source && addaurinfo` instead of the usual `makepkg --source` is a problem? And it still is a problem if you add a line like alias mkaurpkg='makepkg --source && addaurinfo' to your shell rc file and use `mkaurpkg` to build AUR packages (the names are all made up)? Sorry, but that's nonsense. The current AUR PKGBUILD parser is a dead-end. We have several bug reports and adding more workarounds makes no sense at this point. Also, having some kind of metadata makes some of our future plans easier to implement. Why should we refuse to switch to a system that is much cleaner and (almost) just as convenient as the former method?

J. Leclanche [...]

On Tue, 14 Jan 2014 at 19:59:28, Dave Reisner wrote:

[...] The library includes an output format which I've created based on the last discussion from pacman-dev; in particular a post from Allan [1]. This can easily be changed if we forsee undesirable shortcomings. I should note that the format emphasizes split packages as a first class citizen. My hope is that this can be leveraged to introduce proper support for split packages in the AUR eventually. I realize that this probably means work on the AUR side (which I likely won't contribute to) in order to integrate my solution, but I firmly believe it's worthwhile if support for split packages is a desirable goal for the AUR (please tell me it is).

Along with this code, I've created two other utilities:

- parse_aurinfo.py: A parser implementation for the proposed .AURINFO format written in python. - mkaurball: A shell script which creates a source tarball and appends the generated .AURINFO file to the tarball.

There's also a debugging utility which simply imports the lib and dumps an .AURINFO file from a PKGBUILD you point it at.

Awesome! I will give it a try soon. Split packages are definitely a desirable goal for the AUR but that feature requires a lot of work on the AUR side indeed. I won't be able to do much AUR coding until April or May, so what I suggest is the following strategy: * Test your utility. Do some manual tests and automated tests you described below. Fix common use cases. * Create a package that contains mkaurball and put that in [extra] or [community]. Update all Wiki articles etc., replacing `makepkg --source` with `mkaurball`. * Add a deprecation warning to the AUR in the upcoming release that is displayed whenever a source tarball without meta data is submitted. * Fix any remaining bugs that are revealed in production use. * Add support for split packages in the next major release. At the same time, try to integrate .SRCINFO support into makepkg (and support both .AURINFO and .SRCINFO in the AUR, deprecating .AURINFO). That way, the format and the meta data generator gets a lot of testing. The only downside of this approach is that users temporarily need to switch to `mkaurball` and (probably) switch back to `makepkg --source` later.

A nice thing to do next might be to create a test harness which compares my extracted data to the pacman DBs and look at the precise failure modes. I know some of the failure modes, but I won't claim to know them all.

Cheers, Dave

[1] https://www.github.com/falconindy/pkgbuild_reflection [2] https://mailman.archlinux.org/pipermail/pacman-dev/2012-August/015910.html

On Tue, 14 Jan 2014 at 23:54:28, Dave Reisner wrote:

On Tue, Jan 14, 2014 at 11:12:17PM +0100, Lukas Fleischer wrote: [...]

...

* Test your utility. Do some manual tests and automated tests you described below. Fix common use cases.

* Create a package that contains mkaurball and put that in [extra] or [community]. Update all Wiki articles etc., replacing `makepkg --source` with `mkaurball`.

I think you're missing a few steps here. For starters, I don't believe the current .AURINFO parser is capable of consuming the format I'm advertising. Including it without changing the AUR's parser means... Refused uploads? Bad data displayed?

I didn't read code or test your tool before I wrote the mail, so I didn't know that you 1) always add a "pkgbase" section (even to non-split packages), 2) indent some lines using tabs and 3) duplicate a lot of stuff in the pkgname section, even if it's identical to what is listed in the pkgbase section. I assumed that the .AURINFO looks as usual for non-split packages and the pkgbase stuff is just added for split packages. Is there any reason for having this pkgbase overhead in non-split packages?

It's been suggested a few times in a cousin thread that currently .AURINFO is not widely used. I cloned the AUR to find out how much truth there was to that: 75 out of 56k packages have .AURINFO files (.0001%). So, I think any changes in the AUR to consume a new format should be bothering an inconsequential number of people.

Existing packages won't be affected by any .AURINFO change anyway (at least not on the AUR side). That file is only parsed once when uploading.

...

* Add a deprecation warning to the AUR in the upcoming release that is displayed whenever a source tarball without meta data is submitted.

* Fix any remaining bugs that are revealed in production use.

With the understanding that not *all* bugs will be fixed in the parser itself. Some PKGBUILDs will simply not be supported.

Of course. The good thing is that people can still adjust the meta data manually if they want, without adding hacks to the PKGBUILD.

[...]

I just did some tests and the results look pretty good indeed. Do you plan on creating an Arch package for that (as soon as the controversial points are addressed)?

On Thu, Jan 16, 2014 at 11:31 PM, Dave Reisner wrote:

On Thu, Jan 16, 2014 at 08:52:46PM +0100, Lukas Fleischer wrote:

...

On Tue, 14 Jan 2014 at 23:54:28, Dave Reisner wrote:

...

On Tue, Jan 14, 2014 at 11:12:17PM +0100, Lukas Fleischer wrote: [...]

...

* Test your utility. Do some manual tests and automated tests you described below. Fix common use cases.

* Create a package that contains mkaurball and put that in [extra] or [community]. Update all Wiki articles etc., replacing `makepkg --source` with `mkaurball`.

I think you're missing a few steps here. For starters, I don't believe the current .AURINFO parser is capable of consuming the format I'm advertising. Including it without changing the AUR's parser means... Refused uploads? Bad data displayed?

I didn't read code or test your tool before I wrote the mail, so I didn't know that you

Fair enough... I assumed when I mentioned split packages and referenced Allan's post that it was understood I was going outside of the current .AURINFO format (which is far too simplistic to be of value in the long term).

Do you have some example .AURINFO files you can post to the list? I've been dealing with domain-specific packaging a lot the past month and I'm very interested in a potential solution to all this. J. Leclanche

...

1) always add a "pkgbase" section (even to non-split packages),

Intentional.

...

2) indent some lines using tabs and

3) duplicate a lot of stuff in the pkgname section, even if it's identical to what is listed in the pkgbase section.

That shouldn't be the case. What package were you looking at that shows this in the .AURINFO? The goal is that pkgbase section provides the bulk of the metadata -- the individual pkgname sections are only overrides and supplements. The GetMergedPackage def in the python parser illustrates how the base and "overlay" create each output package.

...

I assumed that the .AURINFO looks as usual for non-split packages and the pkgbase stuff is just added for split packages. Is there any reason for having this pkgbase overhead in non-split packages?

Because, IMNSHO, this is old fashioned thinking. Everything should be treated as a split package these days, even if the PKGBUILD only produces a single package as output. makepkg's code is moving in this direction as well. It's totally valid to write a PKGBUILD that produces one package, but has a package_$pkgname() function instead of package().

Since we're going to allow human-massaged .AURINFO files, the AUR's parser should probably allow pkgname without pkgbase. This should be simple to handle if you can handle pkgbase + pkgname, as you're essentially just merging your overrides into the empty set.

...

...

It's been suggested a few times in a cousin thread that currently .AURINFO is not widely used. I cloned the AUR to find out how much truth there was to that: 75 out of 56k packages have .AURINFO files (.0001%). So, I think any changes in the AUR to consume a new format should be bothering an inconsequential number of people.

Existing packages won't be affected by any .AURINFO change anyway (at least not on the AUR side). That file is only parsed once when uploading.

...

...

* Add a deprecation warning to the AUR in the upcoming release that is displayed whenever a source tarball without meta data is submitted.

* Fix any remaining bugs that are revealed in production use.

With the understanding that not *all* bugs will be fixed in the parser itself. Some PKGBUILDs will simply not be supported.

Of course. The good thing is that people can still adjust the meta data manually if they want, without adding hacks to the PKGBUILD.

...

[...]

I just did some tests and the results look pretty good indeed. Do you plan on creating an Arch package for that (as soon as the controversial points are addressed)?

Yep -- sounds good to me.

Cheers, Dave

On Fri, Jan 17, 2014 at 1:52 AM, Dave Reisner wrote:

On Thu, Jan 16, 2014 at 11:46:12PM +0000, Jerome Leclanche wrote:

...

On Thu, Jan 16, 2014 at 11:31 PM, Dave Reisner wrote:

...

On Thu, Jan 16, 2014 at 08:52:46PM +0100, Lukas Fleischer wrote:

...

On Tue, 14 Jan 2014 at 23:54:28, Dave Reisner wrote:

...

On Tue, Jan 14, 2014 at 11:12:17PM +0100, Lukas Fleischer wrote: [...]

...

* Test your utility. Do some manual tests and automated tests you described below. Fix common use cases.

* Create a package that contains mkaurball and put that in [extra] or [community]. Update all Wiki articles etc., replacing `makepkg --source` with `mkaurball`.

I think you're missing a few steps here. For starters, I don't believe the current .AURINFO parser is capable of consuming the format I'm advertising. Including it without changing the AUR's parser means... Refused uploads? Bad data displayed?

I didn't read code or test your tool before I wrote the mail, so I didn't know that you

Fair enough... I assumed when I mentioned split packages and referenced Allan's post that it was understood I was going outside of the current .AURINFO format (which is far too simplistic to be of value in the long term).

Do you have some example .AURINFO files you can post to the list? I've been dealing with domain-specific packaging a lot the past month and I'm very interested in a potential solution to all this.

Clone my repo and run './reflect /path/to/PKGBUILD'. If you have an ABS tree cloned into /var/abs, you can just run './reflect $repo/$pkg'.

I should have said: do you have some exotic examples of packages that showcase the full functionality :) If you're going to change the format, may I advise going with the ini/conf format? It's very simple to implement, and in python it's already handled by the configparser module. It would require one change: duplicate keys result in undefined behaviour. I would recommend going with the same "list" syntax that xdg defines for its ini files: list values are separated by a semicolon. So it could look something like this: [pkgbase python34] pkgdesc = Next generation of the python high-level scription language pkgver = 3.4.0b2 pkgrel = 1 url = http://www.python.org/ makedepends = tk;sqlite;valgrind depends = expat;bzip2;gdbm;openssl;libffi;zlib optdepends = "tk: for tkinter";sqlite [pkgname python34] pkgdesc = Next generation of the python high-level scription language url = http://www.python.org/ license = custom depends = expat;bzip2;gdbm;openssl;libffi;zlib optdepends = "tk: for tkinter";sqlite J. Leclanche

On Thu, Jan 16, 2014 at 06:31:40PM -0500, Dave Reisner wrote:

On Thu, Jan 16, 2014 at 08:52:46PM +0100, Lukas Fleischer wrote:

...

3) duplicate a lot of stuff in the pkgname section, even if it's identical to what is listed in the pkgbase section.

That shouldn't be the case. What package were you looking at that shows this in the .AURINFO? The goal is that pkgbase section provides the bulk of the metadata -- the individual pkgname sections are only overrides and supplements. The GetMergedPackage def in the python parser illustrates how the base and "overlay" create each output package.

Nevermind this -- I found cases where this happens. Fixed locally, just need to write some regression tests.

...

On Thu, Jan 16, 2014 at 06:31:40PM -0500, Dave Reisner wrote:

...

...

3) duplicate a lot of stuff in the pkgname section, even if it's identical to what is listed in the pkgbase section. That shouldn't be the case. What package were you looking at that shows

On Thu, Jan 16, 2014 at 08:52:46PM +0100, Lukas Fleischer wrote: this in the .AURINFO? The goal is that pkgbase section provides the bulk of the metadata -- the individual pkgname sections are only overrides and supplements. The GetMergedPackage def in the python parser illustrates how the base and "overlay" create each output package. Nevermind this -- I found cases where this happens. Fixed locally, just need to write some regression tests. Great! I just submitted some patches to the AUR to add support for the new format and add a deprecation warning for packages not containing any

On Fri, 17 Jan 2014 at 03:11:54, Dave Reisner wrote: meta data [1].

[1] https://mailman.archlinux.org/pipermail/aur-dev/2014-January/002616.html Could you please send more infos about mkaurball, the new format and if

Am 17.01.2014 11:53, schrieb Lukas Fleischer: this page in the wiki is still true: https://wiki.archlinux.org/index.php/AUR_Metadata What should be changed? BTW: How to handle split packages?

On Fri, Jan 17, 2014 at 8:08 AM, Lukas Fleischer wrote:

On Fri, 17 Jan 2014 at 13:56:44, Peter Littmann wrote:

...

[...] Could you please send more infos about mkaurball, the new format and if this page in the wiki is still true: https://wiki.archlinux.org/index.php/AUR_Metadata

Wow, didn't know about that page. Thanks for the work.

mkaurball is a tool written by Dave that should be used to build packages for the AUR in the future. It is included in the pkgbuild_reflection [1] repository (which might change some day, and there will be a package in the official Arch repositories). Basically, that script just calls `makepkg --source`, then generates .AURINFO and includes it in the source tarball.

...

What should be changed? BTW: How to handle split packages?

* The "depend" field has been renamed to "depends".

* Split packages are supported by mkaurball and by .AURINFO but not by the AUR yet. That means that you will be able to create a valid source tarball with valid meta data using mkaurball but the AUR will simply reject such packages. I am working on a solution. There might be support for split packages in the AUR soon.

...

[1] https://github.com/falconindy/pkgbuild_reflection

I just created an AUR package for pkgbuild_reflection. Dave or Lukas, feel free to orphan and adopt this package, if you want. https://aur.archlinux.org/packages/pkgbuild-reflection-git/ Jason

On Tue, Jan 14, 2014 at 11:12:17PM +0100, Lukas Fleischer wrote:

On Tue, 14 Jan 2014 at 19:59:28, Dave Reisner wrote:

...

[...] The library includes an output format which I've created based on the last discussion from pacman-dev; in particular a post from Allan [1]. This can easily be changed if we forsee undesirable shortcomings. I should note that the format emphasizes split packages as a first class citizen. My hope is that this can be leveraged to introduce proper support for split packages in the AUR eventually. I realize that this probably means work on the AUR side (which I likely won't contribute to) in order to integrate my solution, but I firmly believe it's worthwhile if support for split packages is a desirable goal for the AUR (please tell me it is).

Along with this code, I've created two other utilities:

- parse_aurinfo.py: A parser implementation for the proposed .AURINFO format written in python. - mkaurball: A shell script which creates a source tarball and appends the generated .AURINFO file to the tarball.

There's also a debugging utility which simply imports the lib and dumps an .AURINFO file from a PKGBUILD you point it at.

Awesome! I will give it a try soon. Split packages are definitely a desirable goal for the AUR but that feature requires a lot of work on the AUR side indeed. I won't be able to do much AUR coding until April or May, so what I suggest is the following strategy:

* Test your utility. Do some manual tests and automated tests you described below. Fix common use cases.

So this went pretty well. I chose the automated route since I'm a little short on time (traveling tomorrow) and wrote some more python. Basically, it uses my tools to convert PKGBUILD -> .AURINFO -> python, and compares that against the parsed output of 'pacman -Si $repo/$pkg'. PKGBUILDs all come from ABS. As is the nature of ABS, there's bound to be differences between the archived PKGBUILD and the actual PKGBUILD that produced the package currently in the repos. There's bound to be false positives which may vary from day to day. Still, this gives a test bed of ~5000 PKGBUILDs to play with. Out of those, I'm able to fully match the repo 99.1% of the time (including false positives). I can actually boast a 100% strike rate on [extra], as the only differences were caused by out of sync PKGBUILDs which were otherwise correctly parsed. Legitimate problems fell into the expected categories: 1) architecture specifics (examples: core/grub, multilib/dev86) This is obvious, and I knew it would be here. Fixing this requires changes in makepkg. (something like depends_x86_64=(..) or what have you). This cannot reasonably be fixed in any parser. 2) external commands (examples: community/haskell-*, core/perl) I'd suggest that we consider some of these to be false positives or unfixable. The haskell packages all rely on the output of 'pacman -Q ghc' to lock the package to a specific GHC version (I'm not a haskell person). perl just jumps the shark and requires you to be on one of a few Arch servers (it unzips a tarball in a very specific location). 3) core/linux This is an interesting case and gets special mention. The goal of this hackery is to make it easier for folks maintaining custom versions of this PKGBUILD to track and merge changes. I suspect that a legitimate solution to this problem could handled in the PKGBUILD by introducing a new variable, pkgsuffix=, similar to the --with-program-suffix flag that automake offers. The code that I used for all of the above has been pushed, and I'm attaching a tarball of my results from the 4 repos I parsed in case anyone is curious in the gory details. Cheers, Dave