On Fri, Jul 12, 2013 at 08:00:37PM +0100, Ashley Whetter wrote:
I've recently come across issue FS#15645 ([makepkg] needs a test suite) and I'd like to have a bash at implementing it. It seems like a good idea to get peoples ideas and thoughts first so what do you think?
My first reaction is to do something similar to the pacman tests and just write a custom framework in python. Also have the output the same as the pacman tests.
I think keeping the idea an environment (like in the pacman tests) seems a bit heavy handed because the only configuration done is in makepkg.conf and we can probably fake a pacman environment using a stub pacman. Or maybe we could utilise the pacman test environments!
Unit testing should be easy enough because we can just call specific functions in makepkg and check the output.
For functional testing I'm not really sure.
As for acceptance testing we can just put different flag combinations in different test environments, and check the return code. Check the .PKGINFO. Maybe that files were put where they should be in pkg and src, as well as in the package itself. Have some PKGBUILDs that fail for various reasons like syntax, a dependency package doesn't exist. Make sure missing {,make,check}dependencies are installed.
Any suggestions or ideas? Ashley
Incoming wall of text and can of worms. I could have gone on, but I feel strongly about this because I've been down this road before and it's a shitfest. -------- I fear that you're unfamiliar with the various pitfalls that make testing shell scripts so incredibly unwieldy and error prone. There's even more pitfalls than simply *writing* shell. The greatest hurdle is this: the only parser fit to parse bash scripts is bash itself. Notice that currently in the repo, the only shell which is under test has tests which are also written in shell. So, when you mention Python as a basis for testing makepkg, I want to simply discard the rest of your proposal because it's clear to me that you haven't given this much thought. Sorry if that's harsh. What you describe isn't unit testing which is what makepkg really needs most, imo. You're describing system tests which include too many variables. Your tests will invariably be brittle if they're too specific, or useless if they're too vague. I need to mention offhand that the output from pactest should be destroyed and the whole test suite rewired to use autotools' to launch the tests. Doing this will let you parallelize the tests. If you want to write tests for makepkg, it needs to go something like this: 1) Break out all functionality that needs testing to a separate file. Everything in this broken out file *must* be in a function. 2) m4_include() the functions file somewhere in the new makepkg "main". 3) Write unit tests for the various functions. This alone is NOT trivial because you'll need to be able to import various and choice bits of environment. There will likely be multiple patches generated as a result of this against makepkg proper before you can even write tests. 4) Now you might be able to write tests. But, it isn't straight forward. How do you deal with external commands? You can't simply run them and expect to behave as you want because they may read from system-specific files which are out of your control. This goes back to the pitfalls I spoke of before... So what's actually possible in this arena? I've written tests for mkinitcpio. They live on a branch in my personal devel repo: http://code.falconindy.com/cgit/mkinitcpio.git/tree/?h=tests These are made possible through a test library and mocking framework that I wrote (apron), which does a fair job of creating mocks and providing simple wrappers for assertions and error accounting. If you've ever used Google's gmock before, you might recognize some of the style. I haven't committed these to master because while the tests do work, I'm not really sure how reliable they are. They depend fairly heavily on mocks and expectations which don't scale well with time. I could explain why, but that's for another rant. So in the broad sense, why doesn't this work well? This is also known as: what did I learn from writing apron and mkinitcpio tests? 1) You have no ability to separate the environment of the test suite itself and the code under test if you do the testing "in-process". But, if you run the code under test in a child process, you're prohibitively limited from monitoring the test itself. Shell functionality like traps are needed for some behavior in apron. This means that you potentially modify behavior of the code under test by blowing away its own traps. Similarly, the code under test might destroy the traps that apron has set in order to get its job done. 2) You have little to no ability to control redirections. This is important as tests are all about watching and validating, preventing, or controlling side effects. Once you start getting into creating wrapper functions for things as simple as redirections in the actual code in order to make it testable, you start impacting readability negatively. 3) You're at the mercy of the bash parser before your test code even runs, because PKGBUILDs *are* code. How do you handle testing various error conditions triggered by the bash parser and distinguish them from some other failure? Obviously, "end-to-end" tests might be a lot easier here, but could still potentially fail for a lot of the reasons already covered here (external dependencies, etc). You can only feed so many PKGBUILDs to makepkg before you run out of things to test at this broad of a stance and you won't have covered very much more ground than what one can manually test. Have fun, Dave