On Oct 1, 2009, at 2:52 PM, Allan McRae wrote:

Hi,

This is one part of the makepkg test suite I am working on. It is fairly a simple class that takes a pacman package filename and does some parsing. Currently that involves getting the file list and the info from the .PKGINFO file.

I am still fairly new to python so I am looking for comments on how this could be improved. One idea I have had is to not initialize all the fields in the pkginfo dict and add a test if an array exists before appending. It would make the code tidier but that would mean the need to test if the field exists when comparing it later.

I didn't really look at the script, but I made a similar parser [1] earlier, initially for AUR2 purposes. Looking over it, it seems I don't include the file list during parsing, but that can probably be easily added in. Perhaps it can be expanded upon, instead of creating a new one. If not it should at least help. [1] http://github.com/sebnow/parched/blob/master/parched.py

On Oct 1, 2009, at 4:32 PM, Allan McRae wrote:

Sebastian Nowicki wrote:

...

On Oct 1, 2009, at 2:52 PM, Allan McRae wrote:

...

Hi,

This is one part of the makepkg test suite I am working on. It is fairly a simple class that takes a pacman package filename and does some parsing. Currently that involves getting the file list and the info from the .PKGINFO file.

I am still fairly new to python so I am looking for comments on how this could be improved. One idea I have had is to not initialize all the fields in the pkginfo dict and add a test if an array exists before appending. It would make the code tidier but that would mean the need to test if the field exists when comparing it later.

I didn't really look at the script, but I made a similar parser [1] earlier, initially for AUR2 purposes. Looking over it, it seems I don't include the file list during parsing, but that can probably be easily added in. Perhaps it can be expanded upon, instead of creating a new one. If not it should at least help.

[1] http://github.com/sebnow/parched/blob/master/parched.py

Ah.... I had seen that before but I thought it only parsed PKGBUILDs for some reason. I have been using that for some ideas for my PKGBUILD parser but it needed a big overhaul for package splitting.

Indeed, I haven't kept it up to date. I've wanted to actually implement a tiny and safe bash parser to make it work with all PKGBUILDs, but it I could never settle on the feature list (bash is bigger than I thought). Such changes to the PKGBUILD spec shouldn't affect it then. That would be overkill for a test suite though :P. It seems our package parsers are pretty much the same, with the exception of variable naming.

On 01/10/2009, at 10:47 PM, Sebastian Nowicki wrote:

On Oct 1, 2009, at 4:32 PM, Allan McRae wrote:

...

Sebastian Nowicki wrote:

...

On Oct 1, 2009, at 2:52 PM, Allan McRae wrote:

...

Hi,

This is one part of the makepkg test suite I am working on. It is fairly a simple class that takes a pacman package filename and does some parsing. Currently that involves getting the file list and the info from the .PKGINFO file.

I am still fairly new to python so I am looking for comments on how this could be improved. One idea I have had is to not initialize all the fields in the pkginfo dict and add a test if an array exists before appending. It would make the code tidier but that would mean the need to test if the field exists when comparing it later.

I didn't really look at the script, but I made a similar parser [1] earlier, initially for AUR2 purposes. Looking over it, it seems I don't include the file list during parsing, but that can probably be easily added in. Perhaps it can be expanded upon, instead of creating a new one. If not it should at least help.

[1] http://github.com/sebnow/parched/blob/master/parched.py

Ah.... I had seen that before but I thought it only parsed PKGBUILDs for some reason. I have been using that for some ideas for my PKGBUILD parser but it needed a big overhaul for package splitting.

Indeed, I haven't kept it up to date. I've wanted to actually implement a tiny and safe bash parser to make it work with all PKGBUILDs, but it I could never settle on the feature list (bash is bigger than I thought). Such changes to the PKGBUILD spec shouldn't affect it then. That would be overkill for a test suite though :P.

It seems our package parsers are pretty much the same, with the exception of variable naming.

As you may have heard, I started a proper PKGBUILD parser[1], which parses according to shell semantics and does a little interpreting. I just released the first version, which doesn't handle errors, or multi- line values (like arrays or escaped newlines) very well. It does however support split packages. I'm in the process of modifying parched to essentially turn it into python bindings[2] for pkgparse. You probably already have a parser at this point, so I'm not sure how useful this would be to you (it might be overkill anyway), I just though I'd let you know. [1]: http://github.com/sebnow/pkgparse [2]: http://github.com/sebnow/parched/tree/pkgparse_pyrex

On Sat, Dec 12, 2009 at 3:11 PM, Allan McRae wrote:

Sebastian Nowicki wrote:

...

As you may have heard, I started a proper PKGBUILD parser[1], which parses according to shell semantics and does a little interpreting. I just released the first version, which doesn't handle errors, or multi-line values (like arrays or escaped newlines) very well. It does however support split packages. I'm in the process of modifying parched to essentially turn it into python bindings[2] for pkgparse.

You probably already have a parser at this point, so I'm not sure how useful this would be to you (it might be overkill anyway), I just though I'd let you know.

[1]: http://github.com/sebnow/pkgparse [2]: http://github.com/sebnow/parched/tree/pkgparse_pyrex

Looks interesting.  I will take it for a spin later. I assume this is going towards AUR2?

Yes.

I had not done any further work on my parser as I was uncertain what was the best way to go in developing a makepkg test suite.  Given the makepkg test suite will use a safe set of PGKBUILDs, I was thinking of just using bash to parse them.

http://wiki.archlinux.org/index.php/AUR_2#High_priority "Parsing of pkgbuilds, we can no longer use bash to do it because bash sucks and is riddled with security flaws. This is really important." It was discussed with Louipc too on #archlinux-aur earlier, and on the forum too, I don't find the log at this momment :( It's not best solution to do it in bash, lex/yacc seems a better solution for it in this case. Some documentation from Sebastian with that I'm dealing at this momment: http://github.com/sebnow/pkgparse/tree/gh-pages Best Regards, Laszlo Papp

On Sat, Dec 12, 2009 at 3:36 PM, Laszlo Papp wrote:

On Sat, Dec 12, 2009 at 3:11 PM, Allan McRae wrote:

...

Sebastian Nowicki wrote:

...

As you may have heard, I started a proper PKGBUILD parser[1], which parses according to shell semantics and does a little interpreting. I just released the first version, which doesn't handle errors, or multi-line values (like arrays or escaped newlines) very well. It does however support split packages. I'm in the process of modifying parched to essentially turn it into python bindings[2] for pkgparse.

You probably already have a parser at this point, so I'm not sure how useful this would be to you (it might be overkill anyway), I just though I'd let you know.

[1]: http://github.com/sebnow/pkgparse [2]: http://github.com/sebnow/parched/tree/pkgparse_pyrex

Looks interesting.  I will take it for a spin later. I assume this is going towards AUR2?

Yes.

...

I had not done any further work on my parser as I was uncertain what was the best way to go in developing a makepkg test suite.  Given the makepkg test suite will use a safe set of PGKBUILDs, I was thinking of just using bash to parse them.

http://wiki.archlinux.org/index.php/AUR_2#High_priority "Parsing of pkgbuilds, we can no longer use bash to do it because bash sucks and is riddled with security flaws. This is really important."

It was discussed with Louipc too on #archlinux-aur earlier, and on the forum too, I don't find the log at this momment :( It's not best solution to do it in bash, lex/yacc seems a better solution for it in this case.

Some documentation from Sebastian with that I'm dealing at this momment: http://github.com/sebnow/pkgparse/tree/gh-pages

I can't help but think this whole situation is stupid. I would suppose that PKGBUILDs were written in bash for simplicity reason : makepkg just needs to source them, and that's it. Whole parsing done for free. And now we realize that when using untrusted source, we cannot do that anymore. And now we basically have to rewrite a bash parser from scratch. I mean, it's hard to imagine a more flawed design, and more complex solution to a simple problem. Somehow we manage to go from a very KISS solution to a completely anti-KISS one. I only see two solutions : - we keep using bash, but try to do that in the most restricted environment possible (e.g. namcap way , or maybe there is something even more restrictive and secure ?) - we decide that pkgbuild format is a flawed design, and was too limited for our needs, and switch to a new one (in which case Xyne's brainstorming could help : http://xyne.archlinux.ca/ideas/pkgmeta )

Xavier wrote:

I can't help but think this whole situation is stupid.

I would suppose that PKGBUILDs were written in bash for simplicity reason : makepkg just needs to source them, and that's it. Whole parsing done for free. And now we realize that when using untrusted source, we cannot do that anymore. And now we basically have to rewrite a bash parser from scratch. I mean, it's hard to imagine a more flawed design, and more complex solution to a simple problem.

Somehow we manage to go from a very KISS solution to a completely anti-KISS one.

I think it is still a KISS solution for _building packages_. That is what PKGBUILDs and makepkg are supposed to do. Anything other than building packages that is to be done with PKGBUILDs is secondary.

I only see two solutions : - we keep using bash, but try to do that in the most restricted environment possible (e.g. namcap way , or maybe there is something even more restrictive and secure ?)

It is not particularly possible given all the bash "tricks" used in PKGBUILDs.

- we decide that pkgbuild format is a flawed design, and was too limited for our needs, and switch to a new one (in which case Xyne's brainstorming could help : http://xyne.archlinux.ca/ideas/pkgmeta )

As I said above, I do not believe it is flawed. The PKGBUILD format specifies how to create a package in a simple way that is interpreted by makepkg while still maintaining all the flexibility of bash (and all software is built from the shell...). It does its job well. I know people (including me) have encountered the need to parse PKGBUILDs for purposes other that building packages (AUR2, repo checking scripts, makepkg test suite...), but I have yet to see a suggestion that can handle all the complexities available to the current PKGBUILD format while retaining its simplicity. Allan

On 12/12/2009, at 10:11 PM, Allan McRae wrote:

Sebastian Nowicki wrote:

...

As you may have heard, I started a proper PKGBUILD parser[1], which parses according to shell semantics and does a little interpreting. I just released the first version, which doesn't handle errors, or multi-line values (like arrays or escaped newlines) very well. It does however support split packages. I'm in the process of modifying parched to essentially turn it into python bindings[2] for pkgparse. You probably already have a parser at this point, so I'm not sure how useful this would be to you (it might be overkill anyway), I just though I'd let you know. [1]: http://github.com/sebnow/pkgparse [2]: http://github.com/sebnow/parched/tree/pkgparse_pyrex

Looks interesting. I will take it for a spin later. I assume this is going towards AUR2?

I had not done any further work on my parser as I was uncertain what was the best way to go in developing a makepkg test suite. Given the makepkg test suite will use a safe set of PGKBUILDs, I was thinking of just using bash to parse them.

That would probably be the simplest and most accurate way (you don't want bugs in the parser to fail tests). The namcap method is a good way to go. You could look at the current AUR2 implementation to get information (it sources the PKGBUILD and spits out python code)[1]. It is indeed for AUR2. I thought that since you were contemplating a PKGBUILD parser, this might be of use to you. On 12/12/2009, at 11:44 PM, Xavier wrote:

I can't help but think this whole situation is stupid.

I would suppose that PKGBUILDs were written in bash for simplicity reason : makepkg just needs to source them, and that's it. Whole parsing done for free. And now we realize that when using untrusted source, we cannot do that anymore. And now we basically have to rewrite a bash parser from scratch. I mean, it's hard to imagine a more flawed design, and more complex solution to a simple problem.

It is, but as Allan pointed out, makepkg's goals are different. AUR effectively uses PKGBUILDs in an unconventional way.

Somehow we manage to go from a very KISS solution to a completely anti-KISS one.

I only see two solutions : - we keep using bash, but try to do that in the most restricted environment possible (e.g. namcap way , or maybe there is something even more restrictive and secure ?)

I don't think it's possible to make the bash environment any more restrictive than namcap's way. There are fundamental things about bash that make it insecure for blind parsing. The first example that comes to mind are infinite loops. They may not be deliberate, or occur often, but it's a possibility. The others regard various utilities which are required for a sane build environment (mv, install, cat, etc) but could possibly be dangerous when used maliciously. I recall Callan (or someone else) provided a nice example of this on the aur- dev mailing list. It's probably lost now. A custom parser omits these problems by not executing commands (and by inference possibly getting inaccurate metadata, e.g. sed) and restrictive language constructs like loops (maximum amount of iterations before an error is raised). There are of course pros and cons to both approaches.

...

It is not particularly possible given all the bash "tricks" used in PKGBUILDs.

Why is that ? Things like calling external commands and such ? And would the alternative parser support these bash tricks anyway ?

So why don't we just forbid / not support them ?

Some "tricks" are required for the PKGBUILDs to be sane and remove duplication. One particular trick that comes to mind is: if [ "$CARCH" = "x86_64"]; then ... fi. This is something that is very common with binary sources, and the PKGBUILD format has no native support for it. Removing support for it would require two PKGBUILDs, almost identical. Due to things like this, a bit of interpretation is required for the parser to work properly. I plan on supporting conditionals like these in pkgparse. I have also seen for loops populating fields (I think in the kernel package). These tricks should not be supported as it is due to the laziness of the maintainer. Essentially, for building packages, or extracting metadata from safe packages, using bash directly is the way to go. For environments where arbitrary packages are parsed and security is an issue, a parser is currently required.

- we decide that pkgbuild format is a flawed design, and was too limited for our needs, and switch to a new one (in which case Xyne's brainstorming could help : http://xyne.archlinux.ca/ideas/pkgmeta )

I have been thinking of designing a new package metadata format myself, one that is more universal (i.e. not specific to pacman). I doubt this is achievable, but it's an interesting experiment. The idea is mostly inspired by the rockspec format for luarocks[2]. It seems very flexible and provides lots of data. The Lua syntax resembles JSON, so that format can be used instead. YAML could also be used to make it "prettier" and even more flexible. Anyway that's a discussion for another thread. A new format is unlikely to be used by Archlinux any time soon. Sorry for going a little off-topic there. [1]: http://github.com/sebnow/aur2/blob/1ad21387a58eab0d3cf4296f08b6edee0a1e27d2/... [2]: http://www.luarocks.org/en/Creating_a_rock