[arch-general] Pros/Cons of Python zipapp packaging
Hi, We've been discussing the distribution mechanism for mkosi ( https://github.com/systemd/mkosi) and one of the ideas is using Python zipapp (https://docs.python.org/3/library/zipapp.html) to allow us to split mkosi up into multiple files for easier development without complicating the packaging process. zipapp takes all source files in a directory and bundles them up into a single executable python zip archive so after building the zip you can simply call ./mkosi to run mkosi and can put it anywhere in the PATH to simply run mkosi wherever you want. Are there any issues with this approach from a distro packaging perspective? Zipapp doesn't bundle a specific python version (uses system python and system python stdlib) and we don't intend on bundling any dependencies in the zipapp. I don't think I've ever seen a python application packaged this way which is why I'm asking. Cheers, Daan
On Mon, 2020-08-10 at 21:49 +0100, Daan De Meyer via arch-general wrote:
Hi,
We've been discussing the distribution mechanism for mkosi ( https://github.com/systemd/mkosi) and one of the ideas is using Python zipapp (https://docs.python.org/3/library/zipapp.html) to allow us to split mkosi up into multiple files for easier development without complicating the packaging process. zipapp takes all source files in a directory and bundles them up into a single executable python zip archive so after building the zip you can simply call ./mkosi to run mkosi and can put it anywhere in the PATH to simply run mkosi wherever you want. Are there any issues with this approach from a distro packaging perspective? Zipapp doesn't bundle a specific python version (uses system python and system python stdlib) and we don't intend on bundling any dependencies in the zipapp. I don't think I've ever seen a python application packaged this way which is why I'm asking.
Cheers,
Daan
Hi Daan, This approach works, although I am not a fan. Are there any reasons for you to want to do it this way? Which problem are you trying to solve? Python packaging is pretty standardized and I would say that most likely all the problem you will ever find have been solved already. Why not adopt the usual approach? I do not think this packaging method it would be an issue from a distro POV, unless some distro specifically has a guideline against this kind of approach, but it might be an issue for Python standalone packaging. It is something that can be worked around, but definitely troublesome. If you want to support this distribution mechanism *in addition* to the standard Python packaging, that would be okay. But a distro like arch would most probably package it the normal way. In which case I would ask, who is the user for this? Maybe just as standalone? That would make some sense, but given the Python ecosystem has decent tooling nowadays, as an upstream I would probably just tell users to pip/pipx install mkosi. I would say that in spirit it goes against the FHS[1], but that is a very opinionated and subjective take from my part. Feel free to ignore. [1] https://refspecs.linuxfoundation.org/FHS_3.0/fhs.html Cheers, Filipe Laíns
On Mon, Aug 10, 2020, 6:46 PM Filipe Laíns via arch-general < arch-general@archlinux.org> wrote:
On Mon, 2020-08-10 at 21:49 +0100, Daan De Meyer via arch-general wrote:
Hi,
We've been discussing the distribution mechanism for mkosi ( https://github.com/systemd/mkosi) and one of the ideas is using Python zipapp (https://docs.python.org/3/library/zipapp.html) to allow us to split mkosi up into multiple files for easier development without complicating the packaging process. zipapp takes all source files in a directory and bundles them up into a single executable python zip archive so after building the zip you can simply call ./mkosi to run mkosi and can put it anywhere in the PATH to simply run mkosi wherever you want. Are there any issues with this approach from a distro packaging perspective? Zipapp doesn't bundle a specific python version (uses system python and system python stdlib) and we don't intend on bundling any dependencies in the zipapp. I don't think I've ever seen a python application packaged this way which is why I'm asking.
Cheers,
Daan
Hi Daan,
This approach works, although I am not a fan. Are there any reasons for you to want to do it this way? Which problem are you trying to solve?
Python packaging is pretty standardized and I would say that most likely all the problem you will ever find have been solved already. Why not adopt the usual approach?
I do not think this packaging method it would be an issue from a distro POV, unless some distro specifically has a guideline against this kind of approach, but it might be an issue for Python standalone packaging. It is something that can be worked around, but definitely troublesome.
If you want to support this distribution mechanism *in addition* to the standard Python packaging, that would be okay. But a distro like arch would most probably package it the normal way.
In which case I would ask, who is the user for this? Maybe just as standalone? That would make some sense, but given the Python ecosystem has decent tooling nowadays, as an upstream I would probably just tell users to pip/pipx install mkosi.
I would say that in spirit it goes against the FHS[1], but that is a very opinionated and subjective take from my part. Feel free to ignore.
[1] https://refspecs.linuxfoundation.org/FHS_3.0/fhs.html
Cheers, Filipe Laíns
It looks like he wants to make an executable, instead of a module that you import or install Yash
I was evaluating whether we could simply avoid pip altogether and get by with zipapp only as it's quite simple and avoids pip altogether. As Eli deduced correctly, the big advantage is getting an executable that you put anywhere and run with sudo without python having trouble finding the corresponding python module. This would let you put the archive in ~/.local/bin (assuming that is in the PATH) and let you run sudo mkosi without getting import errors because python can't find the mkosi modules. To get that working with pip it seems you're forced to write a bash script that adds the user's site packages directory to PYTHONPATH before invoking mkosi itself with python which works but feels a bit hacky. I guess if it didn't have issues python would likely do it by default. Anyway, thanks for the discussion. For now, we'll support both approaches so users and distro packagers have the final choice in what they decide to use. Daan On Tue, 11 Aug 2020 at 01:25, karx via arch-general < arch-general@archlinux.org> wrote:
On Mon, Aug 10, 2020, 6:46 PM Filipe Laíns via arch-general < arch-general@archlinux.org> wrote:
On Mon, 2020-08-10 at 21:49 +0100, Daan De Meyer via arch-general wrote:
Hi,
We've been discussing the distribution mechanism for mkosi ( https://github.com/systemd/mkosi) and one of the ideas is using Python zipapp (https://docs.python.org/3/library/zipapp.html) to allow us to split mkosi up into multiple files for easier development without complicating the packaging process. zipapp takes all source files in a directory and bundles them up into a single executable python zip archive so after building the zip you can simply call ./mkosi to run mkosi and can put it anywhere in the PATH to simply run mkosi wherever you want. Are there any issues with this approach from a distro packaging perspective? Zipapp doesn't bundle a specific python version (uses system python and system python stdlib) and we don't intend on bundling any dependencies in the zipapp. I don't think I've ever seen a python application packaged this way which is why I'm asking.
Cheers,
Daan
Hi Daan,
This approach works, although I am not a fan. Are there any reasons for you to want to do it this way? Which problem are you trying to solve?
Python packaging is pretty standardized and I would say that most likely all the problem you will ever find have been solved already. Why not adopt the usual approach?
I do not think this packaging method it would be an issue from a distro POV, unless some distro specifically has a guideline against this kind of approach, but it might be an issue for Python standalone packaging. It is something that can be worked around, but definitely troublesome.
If you want to support this distribution mechanism *in addition* to the standard Python packaging, that would be okay. But a distro like arch would most probably package it the normal way.
In which case I would ask, who is the user for this? Maybe just as standalone? That would make some sense, but given the Python ecosystem has decent tooling nowadays, as an upstream I would probably just tell users to pip/pipx install mkosi.
I would say that in spirit it goes against the FHS[1], but that is a very opinionated and subjective take from my part. Feel free to ignore.
[1] https://refspecs.linuxfoundation.org/FHS_3.0/fhs.html
Cheers, Filipe Laíns
It looks like he wants to make an executable, instead of a module that you import or install
Yash
On 8/10/20 4:49 PM, Daan De Meyer via arch-general wrote:
Hi,
We've been discussing the distribution mechanism for mkosi ( https://github.com/systemd/mkosi) and one of the ideas is using Python zipapp (https://docs.python.org/3/library/zipapp.html) to allow us to split mkosi up into multiple files for easier development without complicating the packaging process. zipapp takes all source files in a directory and bundles them up into a single executable python zip archive so after building the zip you can simply call ./mkosi to run mkosi and can put it anywhere in the PATH to simply run mkosi wherever you want. Are there any issues with this approach from a distro packaging perspective? Zipapp doesn't bundle a specific python version (uses system python and system python stdlib) and we don't intend on bundling any dependencies in the zipapp. I don't think I've ever seen a python application packaged this way which is why I'm asking.
This is more or less the same as adding a zip file to the PYTHONPATH, and importing from it -- it works, from a python perspective, no different from unzipping the files into site-packages. The difference is it is only available when you run the file (plus special considerations for __main__). As you say, this isn't trying to bundle dependencies. It's not really functionally different from just having all code in one script file. On the other hand, this introduces size overhead for packaging due to multiple layers of compression (and zip is not very good at this anyway), probably doesn't play nice with bytecode generation, and reduces the effectiveness of a nice side effect of scripted languages, which is that users can trivially read the script, edit it, etc. Minor niggles, really. I wouldn't consider it a real blocker, personally. That being said, I'd guess the only specific advantage this has over installing as a PEP 376-style Installed Python Distribution is the fact that a single file can reliably locate itself when invoked with sudo, rather than losing track of a 'pip install --user' installed version with a builtin user-specific sys.path that isn't preserved by sudo. Do they necessarily need to be incompatible? You could have a pip-installable mkosi/ which can be used with 'python3 -m mkosi' due to possessing a __main__.py, but which can *also* be zipapp'ed for portability. It might behave very weirdly in the pip install --user case, but on the other hand Arch users will always use the bleeding-edge package, while Debian customizes their distro python package with an elaborate hack to make "sudo pip install" be officially supported and not touch dpkg-managed files. I'm not positive what guidance to give you beyond "I don't think this violates the packaging guidelines in the slightest, so have no fear on that count". -- Eli Schwartz Bug Wrangler and Trusted User
participants (4)
-
Daan De Meyer
-
Eli Schwartz
-
Filipe Laíns
-
karx