[PATCH] First HTTP functional test of the RPC interface

[Frédéric Mangano-Tarumi]

Lukas Fleischer [2020-04-16 16:58:51 -0400]
> On Wed, 15 Apr 2020 at 17:54:45, Frédéric Mangano-Tarumi wrote:
> > When I run test/t2600-rendercomment.t and measure the time, it spends
> > almost 8 seconds on setup.sh, then 3 seconds to actually run the tests.
> > That\u2019s quite significant.
> > 
> > $ ./t2600-rendercomment.t | ts -s %.s
> > 0.000010 Starting setup.sh
> > 7.887877 Compeleted setup.sh
> > 8.540758 ok 1 - Test comment rendering.
> > 9.050442 ok 2 - Test Markdown conversion.
> > 9.574958 ok 3 - Test HTML sanitizing.
> > 10.098891 ok 4 - Test link conversion.
> > 10.617952 ok 5 - Test Git commit linkification.
> > 11.156466 ok 6 - Test Flyspray issue linkification.
> > 11.657045 ok 7 - Test headings lowering.
> > 11.673559 # passed all 7 test(s)
> > 11.673628 1..7
> 
> Interesting, I get very different results here. Are you using a
> traditional HDD? Since setup.sh does a lot of disk I/O, I could imagine
> a significant slowdown with physical disks. Patches are always welcome!

That’s right, and my hard disk is harldy new, so I’m more likely to
notice slowness. Still, rendercomment should hardly perform any disk I/O
at all, except for Python’s module loading.

Here’s from my /tmp ramdisk:

	$ ~/dev/aurweb/test/t2600-rendercomment.t | ts -s %.s
	0.000010 Starting setup.sh
	0.383917 Compeleted setup.sh
	0.720444 ok 1 - Test comment rendering.
	1.050310 ok 2 - Test Markdown conversion.
	1.385062 ok 3 - Test HTML sanitizing.
	1.721737 ok 4 - Test link conversion.
	2.055162 ok 5 - Test Git commit linkification.
	2.384108 ok 6 - Test Flyspray issue linkification.
	2.710699 ok 7 - Test headings lowering.
	2.712725 # passed all 7 test(s)
	2.712772 1..7

Impressive. I’m pretty sure a large part of these 3 seconds is caused by
process spawning slowness. If the tests were written with pytest instead
of sharness, therefore using a single process, we might be able to reach
sub-second run time.

By the way, all my measures are for hot runs.

I should run all my tests from /tmp, now that they support being run
from anywhere. We could make the test suite use /tmp automatically, but
I don’t think that’s worth it, especially since it’s not desirable for
everyone.

Anyway, to get back to the topic, common initialization could be more
selective. We could update setup.sh so that, say, `./setup.sh database
git` only initializes an empty database and git, but not SSH or specific
database data. That’s kind of equivalent to splitting setup.sh though.

Concerning Python, we won’t be able to call setup.sh and call it a day.
setup.sh includes sharness, which we don’t want. It also sets bash
variables that we won’t be able to see in Python. I’m sure that with
some ingenuity we’ll find a way to concile both where it makes sense.