Prevent pip from breaking the system by adopting PEP 668
Some time ago, PEP 668[1] enabled distributions to mark the python installation as externally managed by a package manager, thus preventing pip from breaking the system by either installing packages to the system or locally in the home folder. Debian has already adopted this[2].
From my understanding, setting the flag EXTERNALLY-MANAGED only allows using pip inside of a virtualenv and bans using it anywhere else, unless the user uses the very self-explanatory flag --break-system-packages.
Currently on Arch, users can run either `sudo pip`, which will install packages as system and break pacman altogether, or `pip` which automatically uses the --user flag nowadays, which will still break things if done outside of virtual env. Quoting arojas[3], "There are hundreds of bug reports around caused by user installed pip packages overriding system ones" it seems like using the new EXTERNALLY-MANAGED mechanism would greatly reduce wasted time for both users and people managing the bug tracker. Is Arch going to be adopting this behavior too? I could not find a discussion anywhere else about this. Martin (C0rn3j) [1] https://peps.python.org/pep-0668/ [2] https://salsa.debian.org/python-team/packages/python-pip/-/blob/master/debia... [3] https://t.me/archlinuxgroup/662555
On 5/17/23 11:47, Martin Rys wrote:
Some time ago, PEP 668[1] enabled distributions to mark the python installation as externally managed by a package manager, thus preventing pip from breaking the system by either installing packages to the system or locally in the home folder.
...
... A good rationale - question on packaging for things that fall outside the system repos. have several packages where in the install section of PKGBUILD it does something like: pipopts="--isolated --ignore-installed --no-deps" pip $pipopts install --root=$pkgdir dist/*.whl Will this continue to work or is there some change needed to accommodate this change? I didn't see "--root" discussed in reference [1] thanks gene
I found pipenv works well. A package jrnl wouldn't run once built with pip but does run inside pipenv. This creates a virtual environment for building and running packages so once jrnl got built doing pipenv run jrnl runs the program. -- Jude <jdashiel at panix dot com> "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." Ed Howdershelt 1940. On Wed, 17 May 2023, Genes Lists wrote:
On 5/17/23 11:47, Martin Rys wrote:
Some time ago, PEP 668[1] enabled distributions to mark the python installation as externally managed by a package manager, thus preventing pip from breaking the system by either installing packages to the system or locally in the home folder.
...
...
A good rationale - question on packaging for things that fall outside the system repos.
have several packages where in the install section of PKGBUILD it does something like:
pipopts="--isolated --ignore-installed --no-deps" pip $pipopts install --root=$pkgdir dist/*.whl
Will this continue to work or is there some change needed to accommodate this change?
I didn't see "--root" discussed in reference [1]
thanks
gene
On 5/17/23 16:56, Jude DaShiell wrote:
I found pipenv works well. A package jrnl wouldn't run once built with pip but does run inside pipenv. This creates a virtual environment for building and running packages so once jrnl got built doing pipenv run jrnl runs the program.
I'm now following how the above helps package maintainers. Quite possibly I'm missing something. To build a package, makepkg first builds the wheel within the build() macro in PKGBUILD, then the next step is the install() macro which installs the wheel into $pkgdir. These are then wrapped into the package file (foo.zst) At present this is accomplished (say) by using for the install part something like pip --root $pkgdir dist/*.whl which changes the default install "root" from /usr to $pkgdir. The newly created package now has all the right files so that the package (foo.zst) is installable using pacman once its in available in a repo etc. If foo has an executable, then it should be in a standard place (/usr/bin or possibly /usr/local/bin) - no venv's should be needed by any user that installed the package. Best gene
On 5/17/23 17:35, Genes Lists wrote:
I'm now following how the above helps package maintainers. Quite ^^^ not
The question is basically will "pip --root" continue to work as it does now if we adopt 668. If not what needs to be changed in PKGBUILD for the install phase. thanks gene
On Wed, May 17, 2023 at 17:47:31 +0200, Martin Rys wrote:
Some time ago, PEP 668[1] enabled distributions to mark the python installation as externally managed by a package manager, thus preventing pip from breaking the system by either installing packages to the system or locally in the home folder.
Debian has already adopted this[2].
From my understanding, setting the flag EXTERNALLY-MANAGED only allows using pip inside of a virtualenv and bans using it anywhere else, unless the user uses the very self-explanatory flag --break-system-packages.
Currently on Arch, users can run either `sudo pip`, which will install packages as system and break pacman altogether, or `pip` which automatically uses the --user flag nowadays, which will still break things if done outside of virtual env.
Quoting arojas[3], "There are hundreds of bug reports around caused by user installed pip packages overriding system ones" it seems like using the new EXTERNALLY-MANAGED mechanism would greatly reduce wasted time for both users and people managing the bug tracker.
Recalling my days of helping new users, I cannot count how many times someone broke their system because they followed a poorly written guide on the Internet telling them to 'sudo pip' or 'sudo cpan'. I would have *loved* to have this feature back in those days. The number of users it will help is incalculable. This will help not just Arch users, but every user on every platform that ships Python. If big distros like Debian and Arch adopt it, others will learn about it and feel pressure to adopt it as well. Please, please, at least consider this. It is some work to do, sure, but with the proper error message like the Debian example in the PEP, it will make so many new users' (and experts') lives significantly better. This is the kind of safeguard we should have had all along. -- Cheers, Luna Celeste
On 17-05-2023 18:31, Genes Lists wrote:
have several packages where in the install section of PKGBUILD it does something like:
pipopts="--isolated --ignore-installed --no-deps" pip $pipopts install --root=$pkgdir dist/*.whl
Those packages are NOT following the python package guidelines in the wiki and should be adjusted. https://wiki.archlinux.org/title/Python_package_guidelines I have only glimpsed at PEP 668, but it looks to be targeted at users , not packagers .
On 5/18/23 06:26, Lone_Wolf wrote:
On 17-05-2023 18:31, Genes Lists wrote:
have several packages where in the install section of PKGBUILD it does something like:
pipopts="--isolated --ignore-installed --no-deps" pip $pipopts install --root=$pkgdir dist/*.whl
Those packages are NOT following the python package guidelines in the wiki and should be adjusted.
https://wiki.archlinux.org/title/Python_package_guidelines
I have only glimpsed at PEP 668, but it looks to be targeted at users , not packagers .
Thanks LW - it is not quite so clear, maybe it's just me. The python 3.11.3 docs recommend using pip to install python modules [1] "pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers." ... PEP 517 seems to confirm this [2] by showing pip as the frontend [3] as well as in the examples. So do the current standards/recommendations suggest 'pip install' or python -m installer? Best I could tell, pip is the recommended way forward. I'm very interested what other folks, with deeper knowledge / better understanding than mine, believe is the best approach. thanks gene [1] https://docs.python.org/3/installing/index.html [2] https://peps.python.org/pep-0517 [3] https://peps.python.org/pep-0517/#appendix-a-comparison-to-pep-516
On 18-05-2023 13:48, Genes Lists wrote:
On 5/18/23 06:26, Lone_Wolf wrote:
On 17-05-2023 18:31, Genes Lists wrote:
have several packages where in the install section of PKGBUILD it does something like:
pipopts="--isolated --ignore-installed --no-deps" pip $pipopts install --root=$pkgdir dist/*.whl
Those packages are NOT following the python package guidelines in the wiki and should be adjusted.
https://wiki.archlinux.org/title/Python_package_guidelines
I have only glimpsed at PEP 668, but it looks to be targeted at users , not packagers .
Thanks LW - it is not quite so clear, maybe it's just me.
The python 3.11.3 docs recommend using pip to install python modules [1]
"pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers." ...
PEP 517 seems to confirm this [2] by showing pip as the frontend [3] as well as in the examples.
So do the current standards/recommendations suggest 'pip install' or python -m installer?
Best I could tell, pip is the recommended way forward. I'm very interested what other folks, with deeper knowledge / better understanding than mine, believe is the best approach.
thanks
gene
[1] https://docs.python.org/3/installing/index.html [2] https://peps.python.org/pep-0517 [3] https://peps.python.org/pep-0517/#appendix-a-comparison-to-pep-516
For archlinux packaging the arch package guidelines on the wiki are the goto place for recommendations. Those pages are not static and kept uptodate . Important changes are discussed before being made. Check https://wiki.archlinux.org/title/Talk:Python_package_guidelines , chapter 3 and especially 6 relate directly to this thread. LW P.S. Keep in mind 2 things : - python is used on OSes without package managers, user doc needs to work for them also - Archlinux deals with python in a different way then Redhat/Fedora , Debian / Ubuntu and other distros. Several PEPs were created/triggered to deal with those differences. Lone_Wolf
On 5/17/23 18:47, Martin Rys wrote:
Is Arch going to be adopting this behavior too? I could not find a discussion anywhere else about this.
python 3.11.3-2 has implemented this. Thanks for sharing! -- Regards, Felix Yan
The only time I broke my Arch installation beyond reasonable repair
was when I ran `pip upgrade` against all system packages a couple
years ago,
this would have saved me, so seeing this being adopted so quickly is
amazing to see.
Thank you Felix, and thanks to anyone else involved too!
On Wed, Jun 14, 2023 at 6:28 PM Felix Yan
On 5/17/23 18:47, Martin Rys wrote:
Is Arch going to be adopting this behavior too? I could not find a discussion anywhere else about this.
python 3.11.3-2 has implemented this. Thanks for sharing!
-- Regards, Felix Yan
participants (6)
-
Felix Yan
-
Genes Lists
-
Jude DaShiell
-
Lone_Wolf
-
Luna Celeste
-
Martin Rys