TAP conventions for aurweb’s test suites

[Frédéric Mangano-Tarumi]

Hello everyone,

The aurweb test suites are are currently run by make, looking for all
files matching `tNNNN-*.sh`. See `test\Makefile`. Each test program is
standalone and follows the TAP conventions. More about TAP at
https://en.wikipedia.org/wiki/Test_Anything_Protocol

While make is better than nothing, every Arch Linux system comes with a
very common TAP harness: prove, shipped with Perl. The prove command
line is the most intuitive it can get, and generates reports, keeps
running tests even if one fails (unlike make which stops at the first
failure), and can even run tests in parallel provided the tests support
it. We can already use it by running `prove t*.sh` in the `test`
directory.

prove comes with its conventions, which are from the original TAP.
First, it expects test programs to end with `.t`, which is restrictive
enough to exclude non-test files, but doesn’t bind to any specific
language since the shebang is used to know how to run each test. This is
by the way a great opportunity to start writing tests in Python too.

Another convention is to name the test directory `t` and to call prove
from the root of the project. This one is harder to use since all the
existing tests assume they’re run from the `test` directory. A simple cd
at the beginning of every script is probably a good solution, but let’s
leave that problem open for now. While we can run prove from the `test`
directory, running tests from the root has the notable advantage of
letting us import aurweb Python modules without hacking PYTHONPATH.

Here are various suggestions:

1. Change the extension of all the test programs to `.t`.
2. Create `test\README` to tell developers about TAP and prove.
3. Make `make check` call prove.
4. Delete the `make t1234-xxx.sh` rules.
5. Delete the whole Makefile altogether.
6. Run the test programs from the project root.
7. Rename the `test` directory to `t`.

1 to 4 are cheap and have a significant added value in my opinion.
5 to 7 less so, but I think that if we want to use TAP, we should stick
to its conventions.

Feedback welcome!

Cheers,
-- 
fmang