[arch-dev-public] Python packaging future (PEP 517 & removal of setup.py calls)

Sat Feb 19 22:27:09 UTC 2022

Hi all,

As some of you might know, the Python packaging ecosystem is changing. If you
have noticed a lack of setup.py or its replacement with pyproject.toml then you
already ran into issues I want to discuss today.

I will a bit in-depth when explaining the motivation and current state of
affairs, feel free to jump directly to the "What should we do?" and "Usage
summary" sections if you do not care about that.

What is happening?

Up until now, the de-facto build system for Python has been setuptools. This
forced setuptools to try and meet every single need from users. If you are
unlucky enough to need something too different, well, you are out of luck. Your
options are to try to hack around by adding custom handling to your setup.py or
simply to create your own tooling. To prevent this, PEP 517[1] and PEP 518[2]
were proposed.

PEP 517 and 518 standardize the use of different backends. PEP 517 defines a
standard interface for build tools to call the backends (AKA build hooks) and
PEP 518 a way for users to declare which backend to use, in a new pyproject.toml
file.

TL;DR: PEP 518 introduces a pyproject.toml file where people can define the
backend entrypoint and requirements [3] and PEP 517 defines the interface hooks
for build tools to be able to interact with them [4].

Some extra info that might be relevant: PEP 517 suggests that build tools
default to the setuptools.build_meta:__legacy__ backend if no backend is
specified, making them compatible with projects that have not moved to
pyproject.toml. setuptools.build_meta:__legacy__ will essentially call the same
setuptools hooks as invoking setup.py directly.

There are several resources available that explain this much better, please have
a look at them if you need a better/more in-depth explanation. I am also
available for questions if anyone is having trouble understanding how all this
works, I understand it's very involved :P

What are the current issues?

The more pressing issue is that some project are already adopting this scheme,
and our current `setup.py build && setup.py install` approach is simply not
compatible.

There is also the issue that `setup.py install` is deprecated, and will be
removed soon. We have had very extensive conversations with the setuptools
upstream about this. See [5] and [6].

The available tooling

PEP 517 does not support direct installs, which is what we currently via
`setup.py install`. Instead it allows us to build a wheel (a Python binary
package file), which we can then install to the system. So the solution is to
use PEP 517 to build a wheel file for the package, and then install that file to
$pkgdir.

For the building step, I have written python-build[7] which provides a Python
API and a CLI tool to access the PEP 517 build hooks. The project is under the
Python Packaging Authority (PyPA)[8] umbrella.

For the install step, there is pradyunsg/installer[9] which provides a Python
API to develop Python installers, and a CLI with a simple install tool. The
project is currently in the process of being moved under the Python Packaging
Authority (PyPA)[8] umbrella[10].

Our current approach

For project that have adopted PEP 517, in most of the cases we use dephell.
dephell is a tool that reads the metadata from pyproject.toml and generates a
setup.py file. For setuptools this approach is okay, it's a direct translation,
which is simple enough, but there are several other backends that are starting
to being used (flit, poetry, etc.). When this different backends are used,
dephell tries to translate their metadata to setuptools metadata to write to
setup.py. I don't think this approach is sound, it's just a hack IMO. With
things evolving, the backends will introduce quirks, or completely new features
not available in setuptools.
This is actually happening right now, we have at least one backend in the repos
that is so simply not compatible with setuptools and the dephell approach is not
an option, maturin[11]. In this case we rely on maturin providing a front end
(CLI) to build the wheel file.

What should we do?

1) Adopt python-build/python-installer for PEP 517 projects

The Python packaging guidelines have already been updated to replace the dephell
approach with the new standard tooling.

I'd also like to see projects like maturin being built this way, instead of
using their custom CLI tool, but that is not very pressing right now.

2) Drop setup.py install in favor of python-build and python-installer

We should really move a way from setup.py calls.

There are several reasons for this, the main one being that direct setup.py
calls are deprecated. They don't issue a deprecation warning because of
technical difficulties and lack of maintainer time.

Other reason is that setup.py install will generate old-style basically
undocumented, or implementation-driven, Python metadata, also known as "eggs".
By moving to wheels, we will be installing proper standardized metadata (.dist-
info) instead of egg metadata (.egg-info).

I have filled RFC 10 [12] with a formal proposal for this, as it is a fairly big
change. Feel free to give feedback there.

Usage summary

By default, python-build will build the package in a isolated environment, as
recommended by PEP 517. We don't want this behavior in our packaging so we
should disable it with the --no-isolation/-n flag.
By default, it will also build a source distribution (sdist), basically a
release tarball, and then use it to build a wheel file, which we probably want
to avoid, thus we should pass the --wheel/-w flag too.

$ python -m build --wheel --no-isolation
or
$ python -m build -wn

This will generate a wheel in dist/.

Build backends can also take arguments, they can be passed with
--config-setting/-C arguments.

You can find out more about the python-build CLI here[13].

After that, we will want to install the wheel file into $pkgdir. That can be
done by running python-installer with the --destdir/-d argument.
By default, it will compile the Python bytecode for the 0 and 1 levels, matching
our old behavior, so we don't need to pass any extra flag.

$ python -m installer --destdir="$pkgdir" dist/*.whl
or
$ python -m installer -d="$pkgdir" dist/*.whl

Please also check the Python packaging guidelines[14], which should be up-to-
date with the new workflow.

Thank you for hanging out until the end!
Feel free to ping me on IRC or by email if you have any questions.

[1] https://www.python.org/dev/peps/pep-0517/
[2] https://www.python.org/dev/peps/pep-0518/
[3] https://www.python.org/dev/peps/pep-0517/#source-trees
[4] https://www.python.org/dev/peps/pep-0517/#build-backend-interface
[5] https://github.com/pypa/setuptools/pull/2080
[6] https://github.com/pypa/setuptools/issues/2088
[7] https://github.com/FFY00/python-build
[8] https://github.com/pypa/
[9] https://github.com/pradyunsg/installer
[10] https://discuss.python.org/t/moving-installer-under-the-pypa-umbrella/13778
[11] https://www.archlinux.org/packages/community/x86_64/maturin/
[12] https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/10
[13] https://python-build.readthedocs.io/en/latest/
[14]
https://wiki.archlinux.org/title/Python_package_guidelines#Standards_based_(PEP_517)

Cheers,
Filipe Laíns

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20220219/00b6388d/attachment.sig>