Hi all,

As some of you might know, the Python packaging ecosystem is changing. If you have noticed a lack of setup.py or its replacement with pyproject.toml then you already ran into issues I want to discuss today.

I will a bit in-depth when explaining the motivation and current state of affairs, feel free to jump directly to the "What should we do?" and "Usage summary" sections if you do not care about that.

What is happening?

Up until now, the de-facto build system for Python has been setuptools. This forced setuptools to try and meet every single need from users. If you are unlucky enough to need something too different, well, you are out of luck. Your options are to try to hack around by adding custom handling to your setup.py or simply to create your own tooling. To prevent this, PEP 517[1] and PEP 518[2] were proposed.

PEP 517 and 518 standardize the use of different backends. PEP 517 defines a standard interface for build tools to call the backends (AKA build hooks) and PEP 518 a way for users to declare which backend to use, in a new pyproject.toml file.

TL;DR: PEP 518 introduces a pyproject.toml file where people can define the backend entrypoint and requirements [3] and PEP 517 defines the interface hooks for build tools to be able to interact with them [4].

Some extra info that might be relevant: PEP 517 suggests that build tools default to the setuptools.build_meta:__legacy__ backend if no backend is specified, making them compatible with projects that have not moved to pyproject.toml. setuptools.build_meta:__legacy__ will essentially call the same setuptools hooks as invoking setup.py directly.

There are several resources available that explain this much better, please have a look at them if you need a better/more in-depth explanation. I am also available for questions if anyone is having trouble understanding how all this works, I understand it's very involved :P

What are the current issues?

The more pressing issue is that some project are already adopting this scheme, and our current `setup.py build && setup.py install` approach is simply not compatible.

There is also the issue that `setup.py install` is deprecated, and will be removed soon. We have had very extensive conversations with the setuptools upstream about this. See [5] and [6].

The available tooling

PEP 517 does not support direct installs, which is what we currently via `setup.py install`. Instead it allows us to build a wheel (a Python binary package file), which we can then install to the system. So the solution is to use PEP 517 to build a wheel file for the package, and then install that file to $pkgdir.

For the building step, I have written python-build[7] which provides a Python API and a CLI tool to access the PEP 517 build hooks. The project is under the Python Packaging Authority (PyPA)[8] umbrella.

For the install step, there is pradyunsg/installer[9] which provides a Python API to develop Python installers, and a CLI with a simple install tool. The project is currently in the process of being moved under the Python Packaging Authority (PyPA)[8] umbrella[10].

Our current approach

For project that have adopted PEP 517, in most of the cases we use dephell. dephell is a tool that reads the metadata from pyproject.toml and generates a setup.py file. For setuptools this approach is okay, it's a direct translation, which is simple enough, but there are several other backends that are starting to being used (flit, poetry, etc.). When this different backends are used, dephell tries to translate their metadata to setuptools metadata to write to setup.py. I don't think this approach is sound, it's just a hack IMO. With things evolving, the backends will introduce quirks, or completely new features not available in setuptools. This is actually happening right now, we have at least one backend in the repos that is so simply not compatible with setuptools and the dephell approach is not an option, maturin[11]. In this case we rely on maturin providing a front end (CLI) to build the wheel file.

What should we do?

1) Adopt python-build/python-installer for PEP 517 projects

The Python packaging guidelines have already been updated to replace the dephell approach with the new standard tooling.

I'd also like to see projects like maturin being built this way, instead of using their custom CLI tool, but that is not very pressing right now.

2) Drop setup.py install in favor of python-build and python-installer

We should really move a way from setup.py calls.

There are several reasons for this, the main one being that direct setup.py calls are deprecated. They don't issue a deprecation warning because of technical difficulties and lack of maintainer time.

Other reason is that setup.py install will generate old-style basically undocumented, or implementation-driven, Python metadata, also known as "eggs". By moving to wheels, we will be installing proper standardized metadata (.dist- info) instead of egg metadata (.egg-info).

I have filled RFC 10 [12] with a formal proposal for this, as it is a fairly big change. Feel free to give feedback there.

Usage summary

By default, python-build will build the package in a isolated environment, as recommended by PEP 517. We don't want this behavior in our packaging so we should disable it with the --no-isolation/-n flag. By default, it will also build a source distribution (sdist), basically a release tarball, and then use it to build a wheel file, which we probably want to avoid, thus we should pass the --wheel/-w flag too.

$ python -m build --wheel --no-isolation or $ python -m build -wn

This will generate a wheel in dist/.

Build backends can also take arguments, they can be passed with --config-setting/-C arguments.

You can find out more about the python-build CLI here[13].

After that, we will want to install the wheel file into $pkgdir. That can be done by running python-installer with the --destdir/-d argument. By default, it will compile the Python bytecode for the 0 and 1 levels, matching our old behavior, so we don't need to pass any extra flag.

$ python -m installer --destdir="$pkgdir" dist/*.whl or $ python -m installer -d="$pkgdir" dist/*.whl

Please also check the Python packaging guidelines[14], which should be up-to- date with the new workflow.

Thank you for hanging out until the end! Feel free to ping me on IRC or by email if you have any questions.

[1] https://www.python.org/dev/peps/pep-0517/ [2] https://www.python.org/dev/peps/pep-0518/ [3] https://www.python.org/dev/peps/pep-0517/#source-trees [4] https://www.python.org/dev/peps/pep-0517/#build-backend-interface [5] https://github.com/pypa/setuptools/pull/2080 [6] https://github.com/pypa/setuptools/issues/2088 [7] https://github.com/FFY00/python-build [8] https://github.com/pypa/ [9] https://github.com/pradyunsg/installer [10] https://discuss.python.org/t/moving-installer-under-the-pypa-umbrella/13778 [11] https://www.archlinux.org/packages/community/x86_64/maturin/ [12] https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/10 [13] https://python-build.readthedocs.io/en/latest/ [14] https://wiki.archlinux.org/title/Python_package_guidelines#Standards_based_(...)

Cheers, Filipe Laíns