On Sat, 2020-03-21 at 16:51 +0100, Frédéric Mangano-Tarumi wrote:
In order to write tests in Python, we first need to make one important choice: the test framework. Our main constraint is the output format, which should be TAP to run along with the existing scripts.
My favorite candidate is pycotap, which holds in a 157-line no-dependency module that we could put in our tests directory, pretty much like we did for `test/sharness.sh`. It is implemented as a producer for Python’s standard unittest module. https://el-tramo.be/pycotap/
Its main opponent is tappy, a set of tools for working with TAP. It integrates with unittest, pytest and nose. From the documentation, my impression is that it expects to be invoked with nose or pytest’s harness, which would be inconsistent with usual TAP test invocation consisting of calling scripts directly and relying on the shebang. https://tappy.readthedocs.io/en/latest/producers.html
More alternatives at https://tappy.readthedocs.io/en/latest/alternatives.html
Unless we plan to use pytest or nose over unittest, I suggest we use use pycotap. Except for a couple of lines, tests written with pycotap are standard unittest tests, which nose and pytest can collect results from too. If our needs evolve, we could easily upgrade to another framework.
Why should we use unittest over pytest? pytest is easier to read and write, more extensible/has more integrations and has a larger userbase. The only caveat is that people would have to fo a quick read on fixtures before they start contributing. Choosing pycotap over tappy/pytest-tap would make sense if we already had a codebase written in unittest, but we don't. Lukas, what do you think? Cheers, Filipe Laíns