Hi,
PKGBUILDs include the check() function which allows one to run tests on the software after it has been built, but before it has been packaged.
This message proposes to add a test() function, which essentially allows to run tests after package(), and in particular after the package has been installed in a chroot (for these testing purposes).
This allows the packager to run tests against the final installation (or as final as can be in a chroot). Between package() and test(), the resulting pkg is installed in a chroot like on the regular system (i.e. /usr/bin instead of $pkgdir/usr/bin). Everything but the pkg files and the test files is removed, so only the installation is under test.
The idea for this came from the issue that Python packages usually don't need a build or rather, that that build can't be tested without fully installing it. This installation however isn't really possible since /usr/ is inaccessible for obvious reasons. Workarounds usually involve setting the PYTHONPATH, but that's tedious at best and ultimately unnecessary.
Because Python has an extensive auto-detection of libraries and dependencies, the tests can be executed easily, if Python can find them. And after installation (in /usr/lib/python*/site-packages/), the package is easily found. This of course is in contrast to most compiled languages, who would require lots of paths and links in order to be able to compile the tests. It shouldn't be this way, but it is.
A slight complication arises from the fact that tests can theoretically be anywhere in the python source code. So either makepkg figures out what is a test, or the packager specifies that. In most cases though, all tests are in a single directory, and the edge cases are vast, so the latter case with some boilerplate is probably better.
In total, a PKGBUILD might look something like this (names are subject to change):
```
build() {
cd ${pkgname}-${pkgver}/
python -m build --wheel --no-isolation
}
check() {
cd ${pkgname}-${pkgver}/
python -m pytest
}
package() {
cd ${pkgname}-${pkgver}/
python -m installer --destdir="$pkgdir" dist/*.whl
}
collect() {
cd ${pkgname}-${pkgver}/
# collects the tests and puts them into $testdir
python -m pytest --co | grep -o "[^ ]*\.py" | xargs cp -t $testdir
}
# inbetween collect() and test(), the package is installed in the chroot and $testdir is copied
test() {
# we are in $testdir (e.g. /home/builduser)
pytest
}
```
Notice the line in collect() which is responsible to collect the tests. We basically ask pytest what it wants to run, then filter out the files, and lastly copy them into $testdir, which is ultimately copied over and run.
In most cases, this line might just be something like `cp tests/ $testdir/`.
------------
Makepkg
------------
This post is already quite long, so I'll make it quick with some implementation thoughts.
Since collect() and test() always run together or don't run together (test() needs collect() and only collect() has no effect), only a single option is necessary to toggle whether to run the post-package tests.
The order of package() and collect() conceptually doesn't matter, but collect() should probably run after to guarantee that it does not influence package(). Either case, the packager is responsible.
This is a pure addition, so if collect() and test() are missing, then just nothing happens. If only collect() is implemented, that might be ignored too. If only test() exists, then an error is probably most helpful, since no packages are copied. Alternatively, maybe makepkg does some limited test identification (like looking for tests/ folder). Or just nothing happens because the packager made a mistake. Should be obvious enough.
In conclusion, two new methods are introduced to allow for better and more precise testing of the final output, as well as easier testing in case of Python.
The first function - collect() - collects all the tests in the codebase and copies them into $testdir, while package() creates the pkg in $pkgdir. After that, a chroot with all depends packages is created and the pkg installed therein. $testdir is also copied there. Lastly, the second function - test() - is called and executed to test the installation.
Thanks for your attention, if you have further ideas or questions, I'd like to hear them.
If something similar has already been discussed, feel free to point out that thread. AFAICT, it hasn't been before.
- Lamarrrk