TAP conventions for aurweb’s test suites

[Lukas Fleischer]

On Sat, 22 Feb 2020 at 16:24:54, Frédéric Mangano-Tarumi wrote:
> Exactly. Using .t instead of .sh is about opening doors. Rewriting
> existing tests would have no benefit per se. The beauty of TAP\u2019s
> language-agnosticness is the liberty to use to right tool for the right
> job. I\u2019ll probably bring up testing in Python specifically soon.

Sounds great!

> > I like the idea of being able to use prove but I'd prefer still having a
> > way to run tests without having prove installed. How about adding a
> > "prove" target to the Makefile instead of changing `make check`?
> 
> In Arch Linux, prove is part of the perl package, which is a direct
> dependency of git and openssl. Even though Perl is being less and less
> used these days, I don\u2019t think you\u2019ll ever find a system without Perl
> installed. Therefore, I personally don\u2019t think it\u2019s worth checking
> whether prove is installed or not.

It is true that Perl is currently installed on pretty much every Arch
installation but we are working on changing that. Getting rid of the
OpenSSL dependency on Perl [1] is a first step. You are right that even
after those changes, Perl will likely continue to be installed on most
systems; at least the interactively used ones.

> If we do want to support systems without prove anyway, I suggest we find
> a way to let make detect if prove is installed, and depending on the
> result change the behavior of make check. `make prove` has no utility
> as someone who knows about prove would call it directly instead.
> [...]
> Usually, when a project uses make, it provides a Makefile at the root of
> the project, with many rules including check, and is run from the root
> of the project. test/Makefile is like hidden and lonely. I wouldn\u2019t mind
> keeping make check, but I think most developers would rather know they
> can type \u201cprove a.t b.t\u201d to run specific tests and have nice reports.
> `make check` is simpler for users, but prove is better for developers,
> and given the nature of aurweb we should target the latter.

I guess you are right. We could recommend using prove and provide the
Makefile only for compatibility. I don't feel very strongly about this,
but even though the Makefile is hidden, keeping a simple useful 7-line
file won't hurt. All this should be explained in a README file.

> aurweb could benefit from a Makefile if it had rules like \u201cmake
> test-env\u201d to automatically execute the steps in TESTING, \u201cmake run\u201d to
> start the PHP\u202ftest server, etc. To exaggerate a bit, having the
> developer manually set up their database and then come up with \u201chey we
> have a hidden make check rule to save you from typing prove\u201d somewhat
> feels weird.

True. For that matter, there are plans to add a Dockerfile to the source
tree to aid with building a testing environment. We are still working on
this.

> Don\u2019t worry! I\u2019ll send patches progressively and you\u2019ll tell me if
> something feels too disruptive.

Awesome, thanks again for your work!

[1] https://www.archlinux.org/todo/perl-transient-openssl-dependencies/