[pacman-dev] A makepkg test suite

[Dave Reisner]

On Fri, Jul 12, 2013 at 08:00:37PM +0100, Ashley Whetter wrote:
> I've recently come across issue FS#15645 ([makepkg] needs a test
> suite) and I'd like to have a bash at implementing it. It seems like a
> good idea to get peoples ideas and thoughts first so what do you
> think?
> 
> My first reaction is to do something similar to the pacman tests and
> just write a custom framework in python. Also have the output the same
> as the pacman tests.
> 
> I think keeping the idea an environment (like in the pacman tests)
> seems a bit heavy handed because the only configuration done is in
> makepkg.conf and we can probably fake a pacman environment using a
> stub pacman. Or maybe we could utilise the pacman test environments!
> 
> 
> Unit testing should be easy enough because we can just call specific
> functions in makepkg and check the output.
> 
> For functional testing I'm not really sure.
> 
> As for acceptance testing we can just put different flag combinations
> in different test environments, and check the return code. Check the
> .PKGINFO. Maybe that files were put where they should be in pkg and
> src, as well as in the package itself.
> Have some PKGBUILDs that fail for various reasons like syntax, a
> dependency package doesn't exist. Make sure missing
> {,make,check}dependencies are installed.
> 
> 
> Any suggestions or ideas?
> Ashley
> 

Incoming wall of text and can of worms. I could have gone on, but I feel
strongly about this because I've been down this road before and it's a
shitfest.

--------

I fear that you're unfamiliar with the various pitfalls that make
testing shell scripts so incredibly unwieldy and error prone. There's
even more pitfalls than simply *writing* shell. The greatest hurdle is
this: the only parser fit to parse bash scripts is bash itself. Notice
that currently in the repo, the only shell which is under test has tests
which are also written in shell. So, when you mention Python as a basis
for testing makepkg, I want to simply discard the rest of your proposal
because it's clear to me that you haven't given this much thought. Sorry
if that's harsh.

What you describe isn't unit testing which is what makepkg really needs
most, imo. You're describing system tests which include too many
variables. Your tests will invariably be brittle if they're too
specific, or useless if they're too vague.

I need to mention offhand that the output from pactest should be
destroyed and the whole test suite rewired to use autotools' to launch
the tests. Doing this will let you parallelize the tests.

If you want to write tests for makepkg, it needs to go something like
this:

1) Break out all functionality that needs testing to a separate file.
Everything in this broken out file *must* be in a function.
2) m4_include() the functions file somewhere in the new makepkg "main".
3) Write unit tests for the various functions. This alone is NOT trivial
because you'll need to be able to import various and choice bits of
environment. There will likely be multiple patches generated as a result
of this against makepkg proper before you can even write tests.
4) Now you might be able to write tests. But, it isn't straight forward.
How do you deal with external commands? You can't simply run them and
expect to behave as you want because they may read from system-specific
files which are out of your control. This goes back to the pitfalls I
spoke of before...

So what's actually possible in this arena? I've written tests for
mkinitcpio. They live on a branch in my personal devel repo:

http://code.falconindy.com/cgit/mkinitcpio.git/tree/?h=tests

These are made possible through a test library and mocking framework
that I wrote (apron), which does a fair job of creating mocks and
providing simple wrappers for assertions and error accounting. If you've
ever used Google's gmock before, you might recognize some of the style.
I haven't committed these to master because while the tests do work, I'm
not really sure how reliable they are. They depend fairly heavily on
mocks and expectations which don't scale well with time. I could explain
why, but that's for another rant.

So in the broad sense, why doesn't this work well? This is also known
as: what did I learn from writing apron and mkinitcpio tests?

1) You have no ability to separate the environment of the test suite
itself and the code under test if you do the testing "in-process". But,
if you run the code under test in a child process, you're prohibitively
limited from monitoring the test itself. Shell functionality like traps
are needed for some behavior in apron. This means that you potentially
modify behavior of the code under test by blowing away its own traps.
Similarly, the code under test might destroy the traps that apron has
set in order to get its job done.
2) You have little to no ability to control redirections. This is
important as tests are all about watching and validating, preventing, or
controlling side effects. Once you start getting into creating wrapper
functions for things as simple as redirections in the actual code in
order to make it testable, you start impacting readability negatively.
3) You're at the mercy of the bash parser before your test code even
runs, because PKGBUILDs *are* code. How do you handle testing various
error conditions triggered by the bash parser and distinguish them from
some other failure?

Obviously, "end-to-end" tests might be a lot easier here, but could
still potentially fail for a lot of the reasons already covered here
(external dependencies, etc). You can only feed so many PKGBUILDs to
makepkg before you run out of things to test at this broad of a stance
and you won't have covered very much more ground than what one can
manually test.

Have fun,
Dave