Functional HTTP testing

[Frédéric Mangano-Tarumi]

Hello everyone,

With the current plans of porting the PHP codebase to Python, we should
consider setting up a suite of functional tests to squash most bugs
introduced by the rewrite.

I suggest the following testing plan, to be applied on an
endpoint-by-endpoint basis:

1. Write functional HTTP tests for the existing software.
2. Port the code to Python.
3. Run the tests.

This differs from the more appealing plan to test only the new code in
Python using tools provided by the framework we’d use. I think testing
the PHP code first will help find surprising edge cases we could
accidentally change but that other parts of the system depend on.

HTTP testing being a kind of language-agnostic blackbox testing, the
tests would also resist any future rewrite or major refactoring without
requiring maintenance.

I’d now like to introduce a few solutions to implement that.

Requests
--------

Using Python’s Requests package with an HTML and a JSON parser should
cover most cases. We would typically run POST requests to change the
state then parse the HTML to validate the updated data.

Any other programming language would work too. Perl offers better
integration with our current test architecture, and its support for
regex is quite practical for matching results.

Mechanize
---------

Perl has a cool WWW::Mechanize package that provides a higher-level
interface to HTTP testing, offering basic support for forms and link
detection. We would simulate use cases like “Click the ‘login’ link,
submit the form, etc.”, ensuring the website is usable by a human user.
More at https://metacpan.org/pod/WWW::Mechanize#SYNOPSIS

Someone made a port of WWW::Mechanize for Python, called
python-mechanize.

Selenium
--------

Selenium a software suite for testing web applications among other
things. Its WebDriver component controls a full-fledged browser like
Firefox, supporting in particular JavaScript. You can’t get more
powerful than that. See https://www.selenium.dev/documentation/en/

I think it’s overkill because I believe AUR should never require
JavaScript, as it should be usable without a GUI. I myself occasionally
access it on a text browser. It’s also simpler for bots, including
search engine’s.

Conclusion
----------

I think it’s safe to start with Requests, and consider Mechanize later
if need be, or even Selenium. Mechanize and Selenium kind of overlap
with each other, so we should pick only one of the two, but Requests is
well-known and should never be unwelcome.

If it were only me, I’d pick Perl for its shell-like features and regex
support that make it a great tool for writing tests in my opinion.
However, not that many developers are familiar with Perl nowadays.