TAP conventions for aurweb’s test suites

[Lukas Fleischer]

On Fri, 21 Feb 2020 at 23:41:06, Frédéric Mangano-Tarumi wrote:
> prove comes with its conventions, which are from the original TAP.
> First, it expects test programs to end with `.t`, which is restrictive
> enough to exclude non-test files, but doesn\u2019t bind to any specific
> language since the shebang is used to know how to run each test. This is
> by the way a great opportunity to start writing tests in Python too.

The current tests mostly execute commands and compare their outputs, and
I don't see much benefit in rewriting them in Python. That being said, I
think we could benefit from additional unit tests.

> Another convention is to name the test directory `t` and to call prove
> from the root of the project. This one is harder to use since all the
> existing tests assume they\u2019re run from the `test` directory. A simple cd
> at the beginning of every script is probably a good solution, but let\u2019s
> leave that problem open for now. While we can run prove from the `test`
> directory, running tests from the root has the notable advantage of
> letting us import aurweb Python modules without hacking PYTHONPATH.
> 
> Here are various suggestions:
> 
> 1. Change the extension of all the test programs to `.t`.

Sounds good.

> 2. Create `test\README` to tell developers about TAP and prove.

Great idea!

> 3. Make `make check` call prove.

I like the idea of being able to use prove but I'd prefer still having a
way to run tests without having prove installed. How about adding a
"prove" target to the Makefile instead of changing `make check`?

> 4. Delete the `make t1234-xxx.sh` rules.
> 5. Delete the whole Makefile altogether.

Why? Those are small, super low maintenance, and provide the benefit of
being able to easily run the test suite without prove.

> 6. Run the test programs from the project root.
> 7. Rename the `test` directory to `t`.

Sounds good.

> 1 to 4 are cheap and have a significant added value in my opinion.
> 5 to 7 less so, but I think that if we want to use TAP, we should stick
> to its conventions.

I agree with sticking to TAP conventions but it should be easy to keep
almost everything working as-is at the same time without any extra work.

Thanks for bringing this up!

Lukas