[aur-general] Merge request: python2-pyside -> python-pyside
In recent shenanigans[0], python-pyside became a split package, building Python 3 and 2 versions simultaneously. python2-pyside still exists in the AUR. Please merge python2-pyside[1] into python-pyside[2]. [0]: https://mailman.archlinux.org/pipermail/aur-general/2013-September/025115.ht... [1]: https://aur.archlinux.org/packages/python2-pyside/ [2]: https://aur.archlinux.org/packages/python-pyside/ -- Chris “Kwpolska” Warrick <http://kwpolska.tk> PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense
bumpity bump On Sat, Sep 7, 2013 at 10:24 AM, Chris “Kwpolska” Warrick <kwpolska@gmail.com> wrote:
In recent shenanigans[0], python-pyside became a split package, building Python 3 and 2 versions simultaneously. python2-pyside still exists in the AUR. Please merge python2-pyside[1] into python-pyside[2].
[0]: https://mailman.archlinux.org/pipermail/aur-general/2013-September/025115.ht... [1]: https://aur.archlinux.org/packages/python2-pyside/ [2]: https://aur.archlinux.org/packages/python-pyside/
-- Chris “Kwpolska” Warrick <http://kwpolska.tk> PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense
On 09/07/2013 10:24 AM, Chris “Kwpolska” Warrick wrote:
In recent shenanigans[0], python-pyside became a split package, building Python 3 and 2 versions simultaneously. python2-pyside still exists in the AUR. Please merge python2-pyside[1] into python-pyside[2].
[0]: https://mailman.archlinux.org/pipermail/aur-general/2013-September/025115.ht... [1]: https://aur.archlinux.org/packages/python2-pyside/ [2]: https://aur.archlinux.org/packages/python-pyside/
Done, thank you. -- Bartłomiej Piotrowski http://bpiotrowski.pl/
On 2013-09-13 09:57, Bartłomiej Piotrowski wrote:
On 09/07/2013 10:24 AM, Chris “Kwpolska” Warrick wrote:
In recent shenanigans[0], python-pyside became a split package, building Python 3 and 2 versions simultaneously. python2-pyside still exists in the AUR. Please merge python2-pyside[1] into python-pyside[2].
[0]: https://mailman.archlinux.org/pipermail/aur-general/2013-September/025115.ht... [1]: https://aur.archlinux.org/packages/python2-pyside/ [2]: https://aur.archlinux.org/packages/python-pyside/
Done, thank you.
-- Bartłomiej Piotrowski http://bpiotrowski.pl/
As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR? Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results? Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted? -- Hugo Osvaldo Barrera A. Because it breaks the logical sequence of discussion Q. Why is top posting bad?
Hugo Osvaldo Barrera wrote:
As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR?
Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results?
The AUR does not fully parse PKGBUILDs because that would require a tool that can fully parse Bash without executing it, which no one has written yet (to my knowledge). As such, the current approach fudges it with simple regexes or whatever that only extract a subset of data. I do not think that includes "replaces". The AUR also lacks true support for split packages (again, because there is no full Bash parser*), which is why the python2-pyside split package is not detected.
Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted?
A split package builds 2 or more packages. You may optionally install any of them as needed. It makes sense here to have a split package because both the Python 2 and Python 3 versions are built from the same source. The only argument against having a split package in this case is the lack of AUR support, which makes the package a little harder to find and which requires some ugly hacks to the PKGBUILD (e.g. "true &&" to trick the parser). Regards, Xyne * This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.
On Mon, Sep 16, 2013 at 6:43 AM, Xyne <xyne@archlinux.ca> wrote:
* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.
Could that be fixed by having makepkg --source generate a file containing metadata in an appropriate format? And have the AUR parse that file instead. J. Leclanche
From: adys.wh@gmail.com Date: Mon, 16 Sep 2013 06:56:49 +0100 To: aur-general@archlinux.org Subject: Re: [aur-general] Merge request: python2-pyside -> python-pyside
On Mon, Sep 16, 2013 at 6:43 AM, Xyne <xyne@archlinux.ca> wrote:
* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.
Could that be fixed by having makepkg --source generate a file containing metadata in an appropriate format? And have the AUR parse that file instead.
J. Leclanche
We kind of have that with .AURINFO . Might be worth wile to submit patches to expand the usefullness. https://mailman.archlinux.org/pipermail/aur-dev/2013-March/002428.html https://mailman.archlinux.org/pipermail/aur-dev/2013-March/002398.html
On Mon, Sep 16, 2013 at 7:43 AM, Xyne <xyne@archlinux.ca> wrote:
Hugo Osvaldo Barrera wrote:
As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR?
Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results?
The AUR does not fully parse PKGBUILDs because that would require a tool that can fully parse Bash without executing it, which no one has written yet (to my knowledge). As such, the current approach fudges it with simple regexes or whatever that only extract a subset of data. I do not think that includes "replaces".
The AUR also lacks true support for split packages (again, because there is no full Bash parser*), which is why the python2-pyside split package is not detected.
Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted?
A split package builds 2 or more packages. You may optionally install any of them as needed. It makes sense here to have a split package because both the Python 2 and Python 3 versions are built from the same source. The only argument against having a split package in this case is the lack of AUR support, which makes the package a little harder to find and which requires some ugly hacks to the PKGBUILD (e.g. "true &&" to trick the parser).
Regards, Xyne
* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.
-- Chris “Kwpolska” Warrick <http://kwpolska.tk> PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense
whoops, gmail messed up. On Mon, Sep 16, 2013 at 7:43 AM, Xyne <xyne@archlinux.ca> wrote:
The AUR does not fully parse PKGBUILDs because that would require a tool that can fully parse Bash without executing it, which no one has written yet (to my knowledge). [snip] The AUR also lacks true support for split packages (again, because there is no full Bash parser*), which is why the python2-pyside split package is not detected. [snip] * This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.
Why not adapt the actual Bash parser (in C) to only read and do stuff safely? In most cases, this would be enough. In the others, we already have mess in those fields in the AUR. (my C skills are not appropriate for this) Even though $replaces could be easily taken out, just like $depends is. -- Chris “Kwpolska” Warrick <http://kwpolska.tk> PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense
Chris “Kwpolska” Warrick wrote:
Why not adapt the actual Bash parser (in C) to only read and do stuff safely? In most cases, this would be enough. In the others, we already have mess in those fields in the AUR. (my C skills are not appropriate for this)
That is basically what needs to be done but it is a difficult task. Even if you can adapt the Bash source code to return the AST, you would still need to create an extensive whitelist of executables (both internal and external) that may be run in order interpolate all of the variables. The code must be able to detect variable settings nested in the package functions, skip commands that do not affect variables (which may require it to work backwards), count loop cycles to prevent infinite loops, track time to prevent timeouts, etc. I have thought about this before when I wrote the Bauerbill PKGBUILD parser, but I gave up trying to find a way to extract the AST using the Bash code. In the end my code would simply wrap the PKGBUILD in a function, source the file, spit it out with "set" to homogenize the syntax, and then parse it with regexes. I started writing a Bash parser in Haskell with Parsec but my free time ran out and I had to move on to other things. I think that approach would work quite well if the Bash sources are too tangled to extract the parser, but it is a huge task for one person (word expansion, string manipulation, all of the built-ins, etc.). I would be willing to collaborate on that as well, if there is any interest. Regards, Xyne
On Mon, Sep 16, 2013 at 03:07:19PM +0000, Xyne wrote:
Chris “Kwpolska” Warrick wrote:
Why not adapt the actual Bash parser (in C) to only read and do stuff safely? In most cases, this would be enough. In the others, we already have mess in those fields in the AUR. (my C skills are not appropriate for this)
That is basically what needs to be done but it is a difficult task. Even if you can adapt the Bash source code to return the AST, you would still need to create an extensive whitelist of executables (both internal and external) that may be run in order interpolate all of the variables. The code must be able to detect variable settings nested in the package functions, skip commands that do not affect variables (which may require it to work backwards), count loop cycles to prevent infinite loops, track time to prevent timeouts, etc.
And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature... I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR: 1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality. 2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.
I have thought about this before when I wrote the Bauerbill PKGBUILD parser, but I gave up trying to find a way to extract the AST using the Bash code. In the end my code would simply wrap the PKGBUILD in a function, source the file, spit it out with "set" to homogenize the syntax, and then parse it with regexes.
I started writing a Bash parser in Haskell with Parsec but my free time ran out and I had to move on to other things. I think that approach would work quite well if the Bash sources are too tangled to extract the parser, but it is a huge task for one person (word expansion, string manipulation, all of the built-ins, etc.). I would be willing to collaborate on that as well, if there is any interest.
You'd probably be interested in shellcheck: http://www.shellcheck.net/ It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).
Dave Reisner wrote:
And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature...
I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR:
1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality.
Wouldn't that need to actually build everything to access the data nested in the package functions? That wouldn't necessarily be a bad thing as it would require packages to check that the package builds, but in that case you may as well just extract the data from .PKGINFO.
2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.
I honestly think the best approach would be to replace Bash PKGBUILDs with a versatile metadata language that can be easily and safely parsed, e.g. JSON with support for variables and maybe a limited set of custom macros. Build and package functions could be moved to external scriptlets, e.g. '{"build" : "/path/to/build.sh", ...}'. Yet another item on my todo list. :P
You'd probably be interested in shellcheck:
It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).
That is indeed interesting.
On Mon, Sep 16, 2013 at 04:59:03PM +0000, Xyne wrote:
Dave Reisner wrote:
And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature...
I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR:
1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality.
Wouldn't that need to actually build everything to access the data nested in the package functions? That wouldn't necessarily be a bad thing as it would require packages to check that the package builds, but in that case you may as well just extract the data from .PKGINFO.
Feel free to write a script that creates .AURINFO from .PKGINFO.
2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.
I honestly think the best approach would be to replace Bash PKGBUILDs with a versatile metadata language that can be easily and safely parsed, e.g. JSON with support for variables and maybe a limited set of custom macros. Build and package functions could be moved to external scriptlets, e.g. '{"build" : "/path/to/build.sh", ...}'.
Yet another item on my todo list. :P
If you combine .AURINFO and PKGBUILD you already have a format that provides pretty much everything you want.
You'd probably be interested in shellcheck:
It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).
That is indeed interesting.
On Mon, Sep 16, 2013 at 04:59:03PM +0000, Xyne wrote:
Dave Reisner wrote:
And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature...
I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR:
1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality.
Wouldn't that need to actually build everything to access the data nested in the package functions? That wouldn't necessarily be a bad thing as it would require packages to check that the package builds, but in that case you may as well just extract the data from .PKGINFO.
makepkg doesn't need to build everything to extract this metadata. It's hacky, but it can do it. Some of the code in makepkg looks like: for i in pkgver pkgrel epoch; do local indirect="${i}_override" eval $(declare -f package_$1 | sed -n "s/\(^[[:space:]]*$i=\)/${i}_override=/p") [[ -z ${!indirect} ]] && eval ${indirect}=\"${!i}\" done There's other similar stuff for the arch overrides. You can also look at run_split_packaging() for other magic.
2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.
I honestly think the best approach would be to replace Bash PKGBUILDs with a versatile metadata language that can be easily and safely parsed, e.g. JSON with support for variables and maybe a limited set of custom macros. Build and package functions could be moved to external scriptlets, e.g. '{"build" : "/path/to/build.sh", ...}'.
Yet another item on my todo list. :P
Let's call this PKGBUILD 2.0. We'll throw it in the same camp as AUR 2.0.
You'd probably be interested in shellcheck:
It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).
That is indeed interesting.
On Mon, Sep 16, 2013 at 5:59 PM, Xyne <xyne@archlinux.ca> wrote:
Dave Reisner wrote:
And you'd need to do all this work at a level lower than the parser itself to avoid subversion via aliases, functions, and scripts which mask the actual operation's nature...
I think I've mentioned this a few times, but I think there's 2 options if you want better parsing on the AUR:
1) Extend .AURINFO, implement it as .SRCINFO in makepkg proper. To date, I think there's been a number of issues which no one has been willing to address to make this a reality.
Wouldn't that need to actually build everything to access the data nested in the package functions? That wouldn't necessarily be a bad thing as it would require packages to check that the package builds, but in that case you may as well just extract the data from .PKGINFO.
2) Use a VM (e.g. http://www.vidarholen.net/contents/evalbot/) to evalulate the code. This would require something very similar to the guts of makepkg which understands per-package overrides. The output would be something similar to #1, so really... interested parties should just work on that.
I honestly think the best approach would be to replace Bash PKGBUILDs with a versatile metadata language that can be easily and safely parsed, e.g. JSON with support for variables and maybe a limited set of custom macros. Build and package functions could be moved to external scriptlets, e.g. '{"build" : "/path/to/build.sh", ...}'.
Yet another item on my todo list. :P
That makes pkgbuilds needlessly complex to write; json is not exactly a good fit to be human-written (no comments, no trailing commas, ...). makepkg / makepkg -S does run through the bash script, it can gather all the metadata at that point and autogenerate such a json file.
You'd probably be interested in shellcheck:
It's written in Haskell, and while it doesn't execute anything, it does understand a large amount of bash syntax. I found an obscure bug in it recently which was quickly fixed by the author (he's a denizen of #bash on freenode).
That is indeed interesting.
On 2013-09-16 05:43, Xyne wrote:
Hugo Osvaldo Barrera wrote:
As maintainer of python2-pyside shouldn't I have gotten a notification about this? At least an automated one from the AUR?
Also, since the PKGBUILD states "replaces=(python2-pyside...", shouldn't searching the AUR for python2-pyside yield python-pyside as one of the results?
The AUR does not fully parse PKGBUILDs because that would require a tool that can fully parse Bash without executing it, which no one has written yet (to my knowledge). As such, the current approach fudges it with simple regexes or whatever that only extract a subset of data. I do not think that includes "replaces".
The AUR also lacks true support for split packages (again, because there is no full Bash parser*), which is why the python2-pyside split package is not detected.
Finally, why does this merge make sense? It merges two *different* libraries into one; I maintained python2-pyside, but have no interest in installing the python3 version, so why is the original package deleted?
A split package builds 2 or more packages. You may optionally install any of them as needed. It makes sense here to have a split package because both the Python 2 and Python 3 versions are built from the same source. The only argument against having a split package in this case is the lack of AUR support, which makes the package a little harder to find and which requires some ugly hacks to the PKGBUILD (e.g. "true &&" to trick the parser).
Regards, Xyne
* This is the price we pay for having all metadata trapped in a general purpose, quirky scripting language that can only be evaluated by executing arbitrary code.
But AUR helpers don't really process there split packages either, so they only benefit people that will manually download and install them, and even those cases, the new package builds both the python2 and python3 versions always. I guess my real question is: do we really want to delete python2-pyside from AUR? If so, why? The fact that two packages use the same source doesn't seem enough, IMHO (we have plenty of variations of the same package in aur already). -- Hugo Osvaldo Barrera
Hugo Osvaldo Barrera wrote:
But AUR helpers don't really process there split packages either, so they only benefit people that will manually download and install them, and even those cases, the new package builds both the python2 and python3 versions always.
Split packages are created from the same compiled sources. The main work is done in the common build function. The additional overhead of creating each package is simply the overhead of archiving the files. The point is irrelevant, however, as makepkg's "--pkg" option can be used to select the desired package and skip the rest. As for whether or not it is more useful to have two packages in the AUR, I'm actually neutral. Regards, Xyne
participants (8)
-
Bartłomiej Piotrowski
-
Chris “Kwpolska” Warrick
-
Daniel Wallace
-
Dave Reisner
-
Hugo Osvaldo Barrera
-
Jerome Leclanche
-
Lukas Fleischer
-
Xyne