On 2013-09-13 09:57, Bartłomiej Piotrowski wrote:

On 09/07/2013 10:24 AM, Chris “Kwpolska” Warrick wrote:

...

In recent shenanigans[0], python-pyside became a split package, building Python 3 and 2 versions simultaneously. python2-pyside still exists in the AUR. Please merge python2-pyside[1] into python-pyside[2].

[0]: https://mailman.archlinux.org/pipermail/aur-general/2013-September/025115.ht... [1]: https://aur.archlinux.org/packages/python2-pyside/ [2]: https://aur.archlinux.org/packages/python-pyside/

Done, thank you.

-- Bartłomiej Piotrowski http://bpiotrowski.pl/

As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR? Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results? Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted? -- Hugo Osvaldo Barrera A. Because it breaks the logical sequence of discussion Q. Why is top posting bad?

On Mon, Sep 16, 2013 at 6:43 AM, Xyne wrote:

* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.

Could that be fixed by having makepkg --source generate a file containing metadata in an appropriate format? And have the AUR parse that file instead. J. Leclanche

On Mon, Sep 16, 2013 at 7:43 AM, Xyne wrote:

Hugo Osvaldo Barrera wrote:

...

As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR?

Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results?

The AUR does not fully parse PKGBUILDs because that would require a tool that can fully parse Bash without executing it, which no one has written yet (to my knowledge). As such, the current approach fudges it with simple regexes or whatever that only extract a subset of data. I do not think that includes "replaces".

The AUR also lacks true support for split packages (again, because there is no full Bash parser*), which is why the python2-pyside split package is not detected.

...

Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted?

A split package builds 2 or more packages. You may optionally install any of them as needed. It makes sense here to have a split package because both the Python 2 and Python 3 versions are built from the same source. The only argument against having a split package in this case is the lack of AUR support, which makes the package a little harder to find and which requires some ugly hacks to the PKGBUILD (e.g. "true &&" to trick the parser).

Regards, Xyne

* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.

-- Chris “Kwpolska” Warrick http://kwpolska.tk PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense

Chris “Kwpolska” Warrick wrote:

Why not adapt the actual Bash parser (in C) to only read and do stuff safely? In most cases, this would be enough. In the others, we already have mess in those fields in the AUR. (my C skills are not appropriate for this)

That is basically what needs to be done but it is a difficult task. Even if you can adapt the Bash source code to return the AST, you would still need to create an extensive whitelist of executables (both internal and external) that may be run in order interpolate all of the variables. The code must be able to detect variable settings nested in the package functions, skip commands that do not affect variables (which may require it to work backwards), count loop cycles to prevent infinite loops, track time to prevent timeouts, etc. I have thought about this before when I wrote the Bauerbill PKGBUILD parser, but I gave up trying to find a way to extract the AST using the Bash code. In the end my code would simply wrap the PKGBUILD in a function, source the file, spit it out with "set" to homogenize the syntax, and then parse it with regexes. I started writing a Bash parser in Haskell with Parsec but my free time ran out and I had to move on to other things. I think that approach would work quite well if the Bash sources are too tangled to extract the parser, but it is a huge task for one person (word expansion, string manipulation, all of the built-ins, etc.). I would be willing to collaborate on that as well, if there is any interest. Regards, Xyne

Dave Reisner wrote:

And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature...

I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR:

1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality.

Wouldn't that need to actually build everything to access the data nested in the package functions? That wouldn't necessarily be a bad thing as it would require packages to check that the package builds, but in that case you may as well just extract the data from .PKGINFO.

2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.

I honestly think the best approach would be to replace Bash PKGBUILDs with a versatile metadata language that can be easily and safely parsed, e.g. JSON with support for variables and maybe a limited set of custom macros. Build and package functions could be moved to external scriptlets, e.g. '{"build" : "/path/to/build.sh", ...}'. Yet another item on my todo list. :P

You'd probably be interested in shellcheck:

http://www.shellcheck.net/

It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).

That is indeed interesting.

On Mon, Sep 16, 2013 at 04:59:03PM +0000, Xyne wrote:

Dave Reisner wrote:

...

And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature...

I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR:

1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality.

Wouldn't that need to actually build everything to access the data nested in the package functions? That wouldn't necessarily be a bad thing as it would require packages to check that the package builds, but in that case you may as well just extract the data from .PKGINFO.

makepkg doesn't need to build everything to extract this metadata. It's hacky, but it can do it. Some of the code in makepkg looks like: for i in pkgver pkgrel epoch; do local indirect="${i}_override" eval $(declare -f package_$1 | sed -n "s/\(^[[:space:]]*$i=\)/${i}_override=/p") [[ -z ${!indirect} ]] && eval ${indirect}=\"${!i}\" done There's other similar stuff for the arch overrides. You can also look at run_split_packaging() for other magic.

...

2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.

I honestly think the best approach would be to replace Bash PKGBUILDs with a versatile metadata language that can be easily and safely parsed, e.g. JSON with support for variables and maybe a limited set of custom macros. Build and package functions could be moved to external scriptlets, e.g. '{"build" : "/path/to/build.sh", ...}'.

Yet another item on my todo list. :P

Let's call this PKGBUILD 2.0. We'll throw it in the same camp as AUR 2.0.

...

You'd probably be interested in shellcheck:

http://www.shellcheck.net/

It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).

That is indeed interesting.

On 2013-09-16 05:43, Xyne wrote:

Hugo Osvaldo Barrera wrote:

...

As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR?

Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results?

The AUR does not fully parse PKGBUILDs because that would require a tool that can fully parse Bash without executing it, which no one has written yet (to my knowledge). As such, the current approach fudges it with simple regexes or whatever that only extract a subset of data. I do not think that includes "replaces".

The AUR also lacks true support for split packages (again, because there is no full Bash parser*), which is why the python2-pyside split package is not detected.

...

Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted?

A split package builds 2 or more packages. You may optionally install any of them as needed. It makes sense here to have a split package because both the Python 2 and Python 3 versions are built from the same source. The only argument against having a split package in this case is the lack of AUR support, which makes the package a little harder to find and which requires some ugly hacks to the PKGBUILD (e.g. "true &&" to trick the parser).

Regards, Xyne

* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.

But AUR helpers don't really process there split packages either, so they only benefit people that will manually download and install them, and even those cases, the new package builds both the python2 and python3 versions always. I guess my real question is: do we really want to delete python2-pyside from AUR? If so, why? The fact that two packages use the same source doesn't seem enough, IMHO (we have plenty of variations of the same package in aur already). -- Hugo Osvaldo Barrera