TAP conventions for aurweb’s test suites
Hello everyone, The aurweb test suites are are currently run by make, looking for all files matching `tNNNN-*.sh`. See `test\Makefile`. Each test program is standalone and follows the TAP conventions. More about TAP at https://en.wikipedia.org/wiki/Test_Anything_Protocol While make is better than nothing, every Arch Linux system comes with a very common TAP harness: prove, shipped with Perl. The prove command line is the most intuitive it can get, and generates reports, keeps running tests even if one fails (unlike make which stops at the first failure), and can even run tests in parallel provided the tests support it. We can already use it by running `prove t*.sh` in the `test` directory. prove comes with its conventions, which are from the original TAP. First, it expects test programs to end with `.t`, which is restrictive enough to exclude non-test files, but doesn’t bind to any specific language since the shebang is used to know how to run each test. This is by the way a great opportunity to start writing tests in Python too. Another convention is to name the test directory `t` and to call prove from the root of the project. This one is harder to use since all the existing tests assume they’re run from the `test` directory. A simple cd at the beginning of every script is probably a good solution, but let’s leave that problem open for now. While we can run prove from the `test` directory, running tests from the root has the notable advantage of letting us import aurweb Python modules without hacking PYTHONPATH. Here are various suggestions: 1. Change the extension of all the test programs to `.t`. 2. Create `test\README` to tell developers about TAP and prove. 3. Make `make check` call prove. 4. Delete the `make t1234-xxx.sh` rules. 5. Delete the whole Makefile altogether. 6. Run the test programs from the project root. 7. Rename the `test` directory to `t`. 1 to 4 are cheap and have a significant added value in my opinion. 5 to 7 less so, but I think that if we want to use TAP, we should stick to its conventions. Feedback welcome! Cheers, -- fmang
On Fri, 21 Feb 2020 at 23:41:06, Frédéric Mangano-Tarumi wrote:
prove comes with its conventions, which are from the original TAP. First, it expects test programs to end with `.t`, which is restrictive enough to exclude non-test files, but doesn\u2019t bind to any specific language since the shebang is used to know how to run each test. This is by the way a great opportunity to start writing tests in Python too.
The current tests mostly execute commands and compare their outputs, and I don't see much benefit in rewriting them in Python. That being said, I think we could benefit from additional unit tests.
Another convention is to name the test directory `t` and to call prove from the root of the project. This one is harder to use since all the existing tests assume they\u2019re run from the `test` directory. A simple cd at the beginning of every script is probably a good solution, but let\u2019s leave that problem open for now. While we can run prove from the `test` directory, running tests from the root has the notable advantage of letting us import aurweb Python modules without hacking PYTHONPATH.
Here are various suggestions:
1. Change the extension of all the test programs to `.t`.
Sounds good.
2. Create `test\README` to tell developers about TAP and prove.
Great idea!
3. Make `make check` call prove.
I like the idea of being able to use prove but I'd prefer still having a way to run tests without having prove installed. How about adding a "prove" target to the Makefile instead of changing `make check`?
4. Delete the `make t1234-xxx.sh` rules. 5. Delete the whole Makefile altogether.
Why? Those are small, super low maintenance, and provide the benefit of being able to easily run the test suite without prove.
6. Run the test programs from the project root. 7. Rename the `test` directory to `t`.
Sounds good.
1 to 4 are cheap and have a significant added value in my opinion. 5 to 7 less so, but I think that if we want to use TAP, we should stick to its conventions.
I agree with sticking to TAP conventions but it should be easy to keep almost everything working as-is at the same time without any extra work. Thanks for bringing this up! Lukas
Lukas Fleischer [2020-02-22 09:58:15 +0100]
The current tests mostly execute commands and compare their outputs, and I don't see much benefit in rewriting them in Python. That being said, I think we could benefit from additional unit tests.
Exactly. Using .t instead of .sh is about opening doors. Rewriting existing tests would have no benefit per se. The beauty of TAP’s language-agnosticness is the liberty to use to right tool for the right job. I’ll probably bring up testing in Python specifically soon.
I like the idea of being able to use prove but I'd prefer still having a way to run tests without having prove installed. How about adding a "prove" target to the Makefile instead of changing `make check`?
In Arch Linux, prove is part of the perl package, which is a direct dependency of git and openssl. Even though Perl is being less and less used these days, I don’t think you’ll ever find a system without Perl installed. Therefore, I personally don’t think it’s worth checking whether prove is installed or not. If we do want to support systems without prove anyway, I suggest we find a way to let make detect if prove is installed, and depending on the result change the behavior of make check. `make prove` has no utility as someone who knows about prove would call it directly instead.
4. Delete the `make t1234-xxx.sh` rules. 5. Delete the whole Makefile altogether.
Why? Those are small, super low maintenance, and provide the benefit of being able to easily run the test suite without prove.
`make something.sh` should generate or update something.sh, and not run it. This is counter-intuitive to the point of being harmful, because it teaches bad practice. Usually, when a project uses make, it provides a Makefile at the root of the project, with many rules including check, and is run from the root of the project. test/Makefile is like hidden and lonely. I wouldn’t mind keeping make check, but I think most developers would rather know they can type “prove a.t b.t” to run specific tests and have nice reports. `make check` is simpler for users, but prove is better for developers, and given the nature of aurweb we should target the latter. aurweb could benefit from a Makefile if it had rules like “make test-env” to automatically execute the steps in TESTING, “make run” to start the PHP test server, etc. To exaggerate a bit, having the developer manually set up their database and then come up with “hey we have a hidden make check rule to save you from typing prove” somewhat feels weird.
I agree with sticking to TAP conventions but it should be easy to keep almost everything working as-is at the same time without any extra work.
Don’t worry! I’ll send patches progressively and you’ll tell me if something feels too disruptive.
On Sat, 22 Feb 2020 at 16:24:54, Frédéric Mangano-Tarumi wrote:
Exactly. Using .t instead of .sh is about opening doors. Rewriting existing tests would have no benefit per se. The beauty of TAP\u2019s language-agnosticness is the liberty to use to right tool for the right job. I\u2019ll probably bring up testing in Python specifically soon.
Sounds great!
I like the idea of being able to use prove but I'd prefer still having a way to run tests without having prove installed. How about adding a "prove" target to the Makefile instead of changing `make check`?
In Arch Linux, prove is part of the perl package, which is a direct dependency of git and openssl. Even though Perl is being less and less used these days, I don\u2019t think you\u2019ll ever find a system without Perl installed. Therefore, I personally don\u2019t think it\u2019s worth checking whether prove is installed or not.
It is true that Perl is currently installed on pretty much every Arch installation but we are working on changing that. Getting rid of the OpenSSL dependency on Perl [1] is a first step. You are right that even after those changes, Perl will likely continue to be installed on most systems; at least the interactively used ones.
If we do want to support systems without prove anyway, I suggest we find a way to let make detect if prove is installed, and depending on the result change the behavior of make check. `make prove` has no utility as someone who knows about prove would call it directly instead. [...] Usually, when a project uses make, it provides a Makefile at the root of the project, with many rules including check, and is run from the root of the project. test/Makefile is like hidden and lonely. I wouldn\u2019t mind keeping make check, but I think most developers would rather know they can type \u201cprove a.t b.t\u201d to run specific tests and have nice reports. `make check` is simpler for users, but prove is better for developers, and given the nature of aurweb we should target the latter.
I guess you are right. We could recommend using prove and provide the Makefile only for compatibility. I don't feel very strongly about this, but even though the Makefile is hidden, keeping a simple useful 7-line file won't hurt. All this should be explained in a README file.
aurweb could benefit from a Makefile if it had rules like \u201cmake test-env\u201d to automatically execute the steps in TESTING, \u201cmake run\u201d to start the PHP\u202ftest server, etc. To exaggerate a bit, having the developer manually set up their database and then come up with \u201chey we have a hidden make check rule to save you from typing prove\u201d somewhat feels weird.
True. For that matter, there are plans to add a Dockerfile to the source tree to aid with building a testing environment. We are still working on this.
Don\u2019t worry! I\u2019ll send patches progressively and you\u2019ll tell me if something feels too disruptive.
Awesome, thanks again for your work! [1] https://www.archlinux.org/todo/perl-transient-openssl-dependencies/
participants (2)
-
Frédéric Mangano-Tarumi
-
Lukas Fleischer