In order to write tests in Python, we first need to make one important choice: the test framework. Our main constraint is the output format, which should be TAP to run along with the existing scripts. My favorite candidate is pycotap, which holds in a 157-line no-dependency module that we could put in our tests directory, pretty much like we did for `test/sharness.sh`. It is implemented as a producer for Python’s standard unittest module. https://el-tramo.be/pycotap/ Its main opponent is tappy, a set of tools for working with TAP. It integrates with unittest, pytest and nose. From the documentation, my impression is that it expects to be invoked with nose or pytest’s harness, which would be inconsistent with usual TAP test invocation consisting of calling scripts directly and relying on the shebang. https://tappy.readthedocs.io/en/latest/producers.html More alternatives at https://tappy.readthedocs.io/en/latest/alternatives.html Unless we plan to use pytest or nose over unittest, I suggest we use use pycotap. Except for a couple of lines, tests written with pycotap are standard unittest tests, which nose and pytest can collect results from too. If our needs evolve, we could easily upgrade to another framework.
On Sat, 2020-03-21 at 16:51 +0100, Frédéric Mangano-Tarumi wrote:
In order to write tests in Python, we first need to make one important choice: the test framework. Our main constraint is the output format, which should be TAP to run along with the existing scripts.
My favorite candidate is pycotap, which holds in a 157-line no-dependency module that we could put in our tests directory, pretty much like we did for `test/sharness.sh`. It is implemented as a producer for Python’s standard unittest module. https://el-tramo.be/pycotap/
Its main opponent is tappy, a set of tools for working with TAP. It integrates with unittest, pytest and nose. From the documentation, my impression is that it expects to be invoked with nose or pytest’s harness, which would be inconsistent with usual TAP test invocation consisting of calling scripts directly and relying on the shebang. https://tappy.readthedocs.io/en/latest/producers.html
More alternatives at https://tappy.readthedocs.io/en/latest/alternatives.html
Unless we plan to use pytest or nose over unittest, I suggest we use use pycotap. Except for a couple of lines, tests written with pycotap are standard unittest tests, which nose and pytest can collect results from too. If our needs evolve, we could easily upgrade to another framework.
Why should we use unittest over pytest? pytest is easier to read and write, more extensible/has more integrations and has a larger userbase. The only caveat is that people would have to fo a quick read on fixtures before they start contributing. Choosing pycotap over tappy/pytest-tap would make sense if we already had a codebase written in unittest, but we don't. Lukas, what do you think? Cheers, Filipe Laíns
On Sat, 21 Mar 2020 at 12:29:41, Filipe Laíns wrote:
Why should we use unittest over pytest? pytest is easier to read and write, more extensible/has more integrations and has a larger userbase. The only caveat is that people would have to fo a quick read on fixtures before they start contributing.
Choosing pycotap over tappy/pytest-tap would make sense if we already had a codebase written in unittest, but we don't.
Lukas, what do you think?
I have a slight personal preference towards pytest. However, the much more important questions to ask are: 1. How easy/convenient will it be to write tests for aurweb using the framework? What about maintainability? 2. How easy/straightforward will it be for future contributors to write tests using the framework? 3. How well does the framework integrate with the current test suite? Having a small script that can be fully included in our source tree is a nice bonus but shouldn't be one of the main criteria. How hard would it be to make pytest-tap compatible with the usual TAP test invocation? Lukas
I’ve never used unittest, but from my experience pytest has all the qualities you could wish for. Lukas Fleischer [2020-03-22 17:35:23 -0400]
How hard would it be to make pytest-tap compatible with the usual TAP test invocation?
I didn’t realize it at first, but since pytest-tap outputs TAP already, a very simple shell wrapper invoking pytest with the right arguments should be enough to integrate it with the existing test structure. One friction point with pytest-tap is that it’s not in the official Arch repositories, unlike all the other libraries aurweb depends on. Most Python developers should be familiar with pip and virtualenv though, but that’s inconsistent with the instructions in TESTING that suggest using pacman to install Python dependencies.
On Sun, 22 Mar 2020 at 21:03:31, Frédéric Mangano-Tarumi wrote:
I didn\u2019t realize it at first, but since pytest-tap outputs TAP already, a very simple shell wrapper invoking pytest with the right arguments should be enough to integrate it with the existing test structure.
Great!
One friction point with pytest-tap is that it\u2019s not in the official Arch repositories, unlike all the other libraries aurweb depends on. Most Python developers should be familiar with pip and virtualenv though, but that\u2019s inconsistent with the instructions in TESTING that suggest using pacman to install Python dependencies.
I will work on a package and add it to the repositories tomorrow. Best regards, Lukas
participants (3)
-
Filipe Laíns
-
Frédéric Mangano-Tarumi
-
Lukas Fleischer