Lukas Fleischer [2020-02-22 09:58:15 +0100]
The current tests mostly execute commands and compare their outputs, and I don't see much benefit in rewriting them in Python. That being said, I think we could benefit from additional unit tests.
Exactly. Using .t instead of .sh is about opening doors. Rewriting existing tests would have no benefit per se. The beauty of TAP’s language-agnosticness is the liberty to use to right tool for the right job. I’ll probably bring up testing in Python specifically soon.
I like the idea of being able to use prove but I'd prefer still having a way to run tests without having prove installed. How about adding a "prove" target to the Makefile instead of changing `make check`?
In Arch Linux, prove is part of the perl package, which is a direct dependency of git and openssl. Even though Perl is being less and less used these days, I don’t think you’ll ever find a system without Perl installed. Therefore, I personally don’t think it’s worth checking whether prove is installed or not. If we do want to support systems without prove anyway, I suggest we find a way to let make detect if prove is installed, and depending on the result change the behavior of make check. `make prove` has no utility as someone who knows about prove would call it directly instead.
4. Delete the `make t1234-xxx.sh` rules. 5. Delete the whole Makefile altogether.
Why? Those are small, super low maintenance, and provide the benefit of being able to easily run the test suite without prove.
`make something.sh` should generate or update something.sh, and not run it. This is counter-intuitive to the point of being harmful, because it teaches bad practice. Usually, when a project uses make, it provides a Makefile at the root of the project, with many rules including check, and is run from the root of the project. test/Makefile is like hidden and lonely. I wouldn’t mind keeping make check, but I think most developers would rather know they can type “prove a.t b.t” to run specific tests and have nice reports. `make check` is simpler for users, but prove is better for developers, and given the nature of aurweb we should target the latter. aurweb could benefit from a Makefile if it had rules like “make test-env” to automatically execute the steps in TESTING, “make run” to start the PHP test server, etc. To exaggerate a bit, having the developer manually set up their database and then come up with “hey we have a hidden make check rule to save you from typing prove” somewhat feels weird.
I agree with sticking to TAP conventions but it should be easy to keep almost everything working as-is at the same time without any extra work.
Don’t worry! I’ll send patches progressively and you’ll tell me if something feels too disruptive.