2013/12/10 Dave Reisner :

On Tue, Dec 10, 2013 at 10:45:58PM +0800, 郑文辉(Techlive Zheng) wrote:

...

2013/12/10 Karol Blazewicz :

...

On Tue, Dec 10, 2013 at 2:18 PM, 郑文辉(Techlive Zheng) wrote:

...

Hey, I wonder if it is possible to query the `provides` of a package in AUR?

Is e.g. 'cower -ii wine-git' good enough?

Acctually, I want to know if it is possible to query the AUR through API to know what packages provide the desired package. e.g. I query for packages provived 'wine', and get I get 'wine-git' and others. Does cower do that?

No, cower doesn't do this because the AUR's RPC interface doesn't do this.

Okay, understand, so that's a todo, I guess.

On Tue, Dec 10, 2013 at 10:54:42PM +0800, 郑文辉(Techlive Zheng) wrote:

...

2013/12/10 Dave Reisner :

...

On Tue, Dec 10, 2013 at 10:45:58PM +0800, 郑文辉(Techlive Zheng) wrote:

...

2013/12/10 Karol Blazewicz :

...

On Tue, Dec 10, 2013 at 2:18 PM, 郑文辉(Techlive Zheng) wrote:

...

Hey, I wonder if it is possible to query the `provides` of a package in AUR?

Is e.g. 'cower -ii wine-git' good enough?

Acctually, I want to know if it is possible to query the AUR through API to know what packages provide the desired package. e.g. I query for packages provived 'wine', and get I get 'wine-git' and others. Does cower do that?

No, cower doesn't do this because the AUR's RPC interface doesn't do this.

Okay, understand, so that's a todo, I guess.

Well, sure... but I suspect this will never make it into the AUR, given the current implementation of a lot of things. For a universal solution, you'd need to:

1) Extend the PKGBUILD parser to parse provides. This alone is problematic since people insist on dumping split packages on the AUR. There's also plenty of reasons not to extend the PKGBUILD parser in its current form. 2) Extend the DB schema to store and relate the newly parsed provides. 3) Extend RPC responses to include the provides in info/search responses. 4) Add a flag to the search API to allow searching by providers (because this shouldn't be the default behavior, lest you break existing tools). 5) Reparse every single package in the AUR so that steps 1-4 are actually meaningful.

The half-assed solution would be to draw provides data from .AURINFO, but then you rely on submitters to do the right thing. Inevitably, this How about change `makepkg --source` to let it generate a PHP-parsing friendly

2013/12/10 Dave Reisner : format like the `ini` format in the final src tarball, doss this solves everything?

would leave you with a useless "feature" as only a fraction of applicable packages would have the data.

d

2013/12/10 Jerome Leclanche :

On Tue, Dec 10, 2013 at 3:16 PM, 郑文辉(Techlive Zheng) wrote:

...

...

On Tue, Dec 10, 2013 at 10:54:42PM +0800, 郑文辉(Techlive Zheng) wrote:

...

2013/12/10 Dave Reisner :

...

On Tue, Dec 10, 2013 at 10:45:58PM +0800, 郑文辉(Techlive Zheng) wrote:

...

2013/12/10 Karol Blazewicz : > On Tue, Dec 10, 2013 at 2:18 PM, 郑文辉(Techlive Zheng) > wrote: >> Hey, I wonder if it is possible to query the `provides` of a package in AUR? > > Is e.g. 'cower -ii wine-git' good enough?

Acctually, I want to know if it is possible to query the AUR through API to know what packages provide the desired package. e.g. I query for packages provived 'wine', and get I get 'wine-git' and others. Does cower do that?

No, cower doesn't do this because the AUR's RPC interface doesn't do this.

Okay, understand, so that's a todo, I guess.

Well, sure... but I suspect this will never make it into the AUR, given the current implementation of a lot of things. For a universal solution, you'd need to:

1) Extend the PKGBUILD parser to parse provides. This alone is problematic since people insist on dumping split packages on the AUR. There's also plenty of reasons not to extend the PKGBUILD parser in its current form. 2) Extend the DB schema to store and relate the newly parsed provides. 3) Extend RPC responses to include the provides in info/search responses. 4) Add a flag to the search API to allow searching by providers (because this shouldn't be the default behavior, lest you break existing tools). 5) Reparse every single package in the AUR so that steps 1-4 are actually meaningful.

The half-assed solution would be to draw provides data from .AURINFO, but then you rely on submitters to do the right thing. Inevitably, this How about change `makepkg --source` to let it generate a PHP-parsing friendly

2013/12/10 Dave Reisner : format like the `ini` format in the final src tarball, doss this solves everything?

That's exactly what .PKGINFO files are and it's imho the only sane way to do this. Problem are, they're not generated by -S.

...

...

would leave you with a useless "feature" as only a fraction of applicable packages would have the data.

That shold be an easy fix, I guess, someone just need to acctually submit the patch and discuss it with Allan.

On Tue, Dec 10, 2013 at 4:21 PM, Jerome Leclanche wrote:

I don't understand why makepkg -S doesn't include the .PKGINFO file from makepkg (and subsequently the AUR would use that if present instead of the grep system which fails as soon as variables/expansions are involved which is every other package). All the implementation is there.

It would probably be better then what we have now, but a perfect solution would also account for PKGBUILDs that use Bash conditionals to set different variables on i386 systems than on x86_64 systems (which is pretty common among AUR packages that package upstream binaries rather than compiling from source). Reading values from .PKGINFO would populate the AUR with the values for whichever architecture the package uploader happened to use. (So if the maintainer changes, or the same maintainer works on different computers, a simple re-upload of an AUR package could suddenly change the package's meta-info, i.e. the AUR would become more "fragile".) Of course, the "perfect solution" would be pretty difficult to implement. Gentoo had a GSoC project last year [1], to implement an efficient and safe (side-effect free) limited Bash parser / pseudo-interpreter in C++, sufficient to extract all necessary values from Genoo's equivalent of PKGBUILDs. Surely, this could have been useful for the AUR as well. But I can find no evidence of continued project activity after the GSoC period ended, so it appears they have given up... :( --- [1] http://dev.gentoo.org/~qiaomuf/libbash.html

On Wed, Dec 11, 2013 at 06:12:59PM +0000, Jerome Leclanche wrote:

On Wed, Dec 11, 2013 at 5:52 PM, Sam S. wrote:

...

On Tue, Dec 10, 2013 at 4:21 PM, Jerome Leclanche wrote:

...

I don't understand why makepkg -S doesn't include the .PKGINFO file from makepkg (and subsequently the AUR would use that if present instead of the grep system which fails as soon as variables/expansions are involved which is every other package). All the implementation is there.

It would probably be better then what we have now, but a perfect solution would also account for PKGBUILDs that use Bash conditionals to set different variables on i386 systems than on x86_64 systems (which is pretty common among AUR packages that package upstream binaries rather than compiling from source).

Reading values from .PKGINFO would populate the AUR with the values for whichever architecture the package uploader happened to use. (So if the maintainer changes, or the same maintainer works on different computers, a simple re-upload of an AUR package could suddenly change the package's meta-info, i.e. the AUR would become more "fragile".)

Of course, the "perfect solution" would be pretty difficult to implement. Gentoo had a GSoC project last year [1], to implement an efficient and safe (side-effect free) limited Bash parser / pseudo-interpreter in C++, sufficient to extract all necessary values from Genoo's equivalent of PKGBUILDs. Surely, this could have been useful for the AUR as well. But I can find no evidence of continued project activity after the GSoC period ended, so it appears they have given up... :(

--- [1] http://dev.gentoo.org/~qiaomuf/libbash.html

I love the fact someone could be working on a bash parser but that solution is *insane*.

It's designed to be incomplete, and the project appears very dead.

This is a solved problem: use a metafile compiled by whatever tools you use in your distro/domain that can be parsed safely and easily. For Arch, those are PKGINFOs.

No, go see historical conversations about a mythical .SRCINFO -- this is what .AURINFO is based on. .SRCINFO is vaporware right now, and I again refer you to unresolved discussions about how it would handle split packages and package-specific overrides.

Good point on the differences per arch. I guess the obvious solution that comes to mind here is to have makepkg -S generate the source files for each arch value (eg. PKGINFO.x86_64, etc) but that's not necessarily good and is the subject of another discussion imo.

To be clear, .PKGINFO is not the solution here. This file describes a built package (something the AUR explicitly does not support). d

On Wed, Dec 11, 2013 at 08:21:44PM +0000, Jerome Leclanche wrote:

On Wed, Dec 11, 2013 at 6:42 PM, Dave Reisner wrote:

...

On Wed, Dec 11, 2013 at 06:12:59PM +0000, Jerome Leclanche wrote:

...

On Wed, Dec 11, 2013 at 5:52 PM, Sam S. wrote:

...

On Tue, Dec 10, 2013 at 4:21 PM, Jerome Leclanche wrote:

...

I don't understand why makepkg -S doesn't include the .PKGINFO file from makepkg (and subsequently the AUR would use that if present instead of the grep system which fails as soon as variables/expansions are involved which is every other package). All the implementation is there.

It would probably be better then what we have now, but a perfect solution would also account for PKGBUILDs that use Bash conditionals to set different variables on i386 systems than on x86_64 systems (which is pretty common among AUR packages that package upstream binaries rather than compiling from source).

Reading values from .PKGINFO would populate the AUR with the values for whichever architecture the package uploader happened to use. (So if the maintainer changes, or the same maintainer works on different computers, a simple re-upload of an AUR package could suddenly change the package's meta-info, i.e. the AUR would become more "fragile".)

Of course, the "perfect solution" would be pretty difficult to implement. Gentoo had a GSoC project last year [1], to implement an efficient and safe (side-effect free) limited Bash parser / pseudo-interpreter in C++, sufficient to extract all necessary values from Genoo's equivalent of PKGBUILDs. Surely, this could have been useful for the AUR as well. But I can find no evidence of continued project activity after the GSoC period ended, so it appears they have given up... :(

--- [1] http://dev.gentoo.org/~qiaomuf/libbash.html

I love the fact someone could be working on a bash parser but that solution is *insane*.

It's designed to be incomplete, and the project appears very dead.

...

This is a solved problem: use a metafile compiled by whatever tools you use in your distro/domain that can be parsed safely and easily. For Arch, those are PKGINFOs.

No, go see historical conversations about a mythical .SRCINFO -- this is what .AURINFO is based on. .SRCINFO is vaporware right now, and I again refer you to unresolved discussions about how it would handle split packages and package-specific overrides.

...

Good point on the differences per arch. I guess the obvious solution that comes to mind here is to have makepkg -S generate the source files for each arch value (eg. PKGINFO.x86_64, etc) but that's not necessarily good and is the subject of another discussion imo.

To be clear, .PKGINFO is not the solution here. This file describes a built package (something the AUR explicitly does not support).

d

Care to explain the reasoning? I'm looking at a few example PKGINFOs and they contain nothing that can exclusively be generated at package() or build() time (other than pkgver but that's already the case currently).

J. Leclanche

Here's a few reasons that come to mind: 1) As you've already discovered, you'd need a .PKGINFO file for every potential architecture, rather than just a .SRCINFO which might describe what architectures are known available. A .SRCINFO could express all package and architecture specific overrides. The fact that an override exists might be a useful bit of information to convey. 2) .PKGINFO doesn't contain things that are useful for source packages, like, say... the source URL? checksums? 3) You can't possibly express split packages well. Having pkgname fields in a .SRCINFO would mean you could describe all packages created by a PKGBUILD rather than having some loose association between a PKGBUILD and some possible .PKGINFO files which it might have generated. This is about *source* packages. The metadata needs for source packages are not the same as the needs for binary packages.

On Thu, Dec 12, 2013 at 08:47:49AM +1000, Justin Dray wrote:

What is to stop us using both? I don't think anyone has said we should stop parsing the PKGBUILD for the AUR and exclusively parse a .PKGINFO instead.

We should stop parsing the PKGBUILD for the AUR and exclusively parse a .SRCINFO instead.

On 2013-12-11 22:57 +0000 Jerome Leclanche wrote:

These are all issues that can easily be fixed. Sources/checksums can be added, extra architectures can be generated, and split packages can be described in sections. (.conf aka .ini file format would be a perfect fit here and would be forwards-compatible with the current format but pretty much anything with sections would do.)

Just to be clear, I'm not advocating to specifically use .PKGINFO; just to use a compiled, parseable file (such as the .SRCINFO that was talked about).. and it seems the consensus is that this is the best solution. Am I wrong? What are the issues with .SRCINFO?

My response is not directly at you in particular. I'm reply to the thread as a whole. The real problem is that we are attacking this from the wrong end. We should no be trying to excavate machine-parsable metadata from Bash. We should be thinking of ways to encapsulate the metadata in a flexible way that can be used to generate Bash. This would solve all of these problems and open the doors for numerous packaging tools (graph generators, metadata miners, fully automated repo generators, etc.). I've played around a bit with encoding package metadata in JSON. All of the packages on my site are autogenerated from JSON files with associated Bash stubs containing functions. My own approach is customized to my release scripts and not immediately generalizable, but the idea itself is workable. The key selling point of Bash is flexibility due to it being a full scripting language, but for the metadata itself we don't need the full flexibility. I think the following would suffice: 1) A convention for expressing per-package namespace for split packages, with a way to inherit and override from other namespaces to avoid redundancies. This can be done with sub-objects and a little syntax, e.g. { "python-foo" : { ... }, "python2-foo < python-foo" : { "depends" : ["python2", ...]", ... } 2) Variables for convenience. A top-level field to specify a delimiter would suffice, e.g. {"variable delimiter" : "$", ...} and then later {..., "url" : "http://example.com/$pkgname$-$version$.tar.gz", ...}. The file could be safely interpolated. Interpolation requires some thought but it should be easy to do in all sane cases. 3) The build, package, prepare and version functions would obviously still be written in Bash, but they would be kept in a separate file and devoid of metadata. 4) If truly necessary, syntax for inserting command output could be added as well, again with a top-level limiter for the command. This would allow parsers to decide exactly which commands to run, if any. The use of such commands would be restricted to cases in which they are absolutely necessary (if there are any). Overall, it is not much more verbose than current PKGBUILDS: "powerpill": { "pkgdesc": "Pacman wrapper for parallel and segmented downloads.", "pkgrel": 1, "pkgver": "2013.5.13", "arch": "any", "backup": [ "etc/powerpill/powerpill.json" ], "depends": [ "python3", "pyalpm", "pm2ml>2012.12.12", "reflector", "aria2" ], "optdepends": { "python3-threaded_servers": "internal Pacserve support", "rsync": "Rsync download support" }, "md5sums": [ "0f0d4c0096bd60287ab923038c3bbc47", "f58432feaaeb339e5d6b3692d3cecf17" ], ... As soon as I have some time I will provide a proof-of-principle implementation and post a RFC. Regards, Xyne

On 2013-12-12 01:39, Xyne wrote:

On 2013-12-11 22:57 +0000 Jerome Leclanche wrote:

...

These are all issues that can easily be fixed. Sources/checksums can be added, extra architectures can be generated, and split packages can be described in sections. (.conf aka .ini file format would be a perfect fit here and would be forwards-compatible with the current format but pretty much anything with sections would do.)

Just to be clear, I'm not advocating to specifically use .PKGINFO; just to use a compiled, parseable file (such as the .SRCINFO that was talked about).. and it seems the consensus is that this is the best solution. Am I wrong? What are the issues with .SRCINFO?

My response is not directly at you in particular. I'm reply to the thread as a whole.

The real problem is that we are attacking this from the wrong end. We should no be trying to excavate machine-parsable metadata from Bash. We should be thinking of ways to encapsulate the metadata in a flexible way that can be used to generate Bash.

This would solve all of these problems and open the doors for numerous packaging tools (graph generators, metadata miners, fully automated repo generators, etc.).

I've played around a bit with encoding package metadata in JSON. All of the packages on my site are autogenerated from JSON files with associated Bash stubs containing functions. My own approach is customized to my release scripts and not immediately generalizable, but the idea itself is workable.

The key selling point of Bash is flexibility due to it being a full scripting language, but for the metadata itself we don't need the full flexibility. I think the following would suffice:

1) A convention for expressing per-package namespace for split packages, with a way to inherit and override from other namespaces to avoid redundancies. This can be done with sub-objects and a little syntax, e.g. { "python-foo" : { ... }, "python2-foo < python-foo" : { "depends" : ["python2", ...]", ... }

2) Variables for convenience. A top-level field to specify a delimiter would suffice, e.g. {"variable delimiter" : "$", ...} and then later {..., "url" : "http://example.com/$pkgname$-$version$.tar.gz", ...}. The file could be safely interpolated. Interpolation requires some thought but it should be easy to do in all sane cases.

3) The build, package, prepare and version functions would obviously still be written in Bash, but they would be kept in a separate file and devoid of metadata.

4) If truly necessary, syntax for inserting command output could be added as well, again with a top-level limiter for the command. This would allow parsers to decide exactly which commands to run, if any. The use of such commands would be restricted to cases in which they are absolutely necessary (if there are any).

Overall, it is not much more verbose than current PKGBUILDS:

"powerpill": { "pkgdesc": "Pacman wrapper for parallel and segmented downloads.", "pkgrel": 1, "pkgver": "2013.5.13", "arch": "any", "backup": [ "etc/powerpill/powerpill.json" ], "depends": [ "python3", "pyalpm", "pm2ml>2012.12.12", "reflector", "aria2" ], "optdepends": { "python3-threaded_servers": "internal Pacserve support", "rsync": "Rsync download support" }, "md5sums": [ "0f0d4c0096bd60287ab923038c3bbc47", "f58432feaaeb339e5d6b3692d3cecf17" ], ...

As soon as I have some time I will provide a proof-of-principle implementation and post a RFC.

Regards, Xyne

I'm very much in favour of this (I'd even though of something slightly similar at some point). We need a PKGBUILD2.0 format that can overcome of our current limitations. The downside is flexibility. What if the sources are complete different for each architecture. At the moment, we can have an "if" in place. The same applies for depends that vary according to architecture. Would we just have a bunch of differently named-fields? Before you post an RFC, a complete example of a really complex package (with wierd sources, depends and stuff) would be great! :D Anyway, if this becomes a reality, feel free to CC me; I'd be willing to help the development of tools for this format! :D -- Hugo Osvaldo Barrera