On Sat, Nov 21, 2020 at 04:47:27PM +0100, David Runge wrote:

On 2020-11-21 14:34:24 (+0000), Filipe Laíns via arch-dev-public wrote:

...

Hi all,

I want to propose adding all active Python versions to [community], not just the latest one. This would only entail adding the interpreter itself, no other packages.

An alternative (in a per-user setup) can be to use pyenv [1]. It works reasonably well with tox etc. and I have used it in the past successfully to develop against several python interpreter versions.

I'm personally not very excited for the idea of adding more python interpreters. I'm a bit concerned that we today say "no packages" but in the future we relax a bit and end up with python36-$pkgname, as it's the pragmatic option as opposed to blocking entire rebuilds or package updates. What is the downsides of utilizing something like pyenv? There are user repositories providing older python interpreters as well if people need it. -- Morten Linderud PGP: 9C02FF419FECBE16

On 11/21/20 11:58 AM, Filipe Laíns via arch-dev-public wrote:

pyenv forces users to compile the Python interpreter themselves, which can take a long time with --enable-optimizations. None of the user repos available builds with optimizations, or has signed packages AFAIK. Of course they could in the future, but I think having the packages in the repos is much better in terms of security and usability. I run one of the user which provides these packages and I don't see myself fixing any of the issues I pointed out due to technical limitations.

Incidentally... if you don't see yourself fixing the --enable-optimizations or gpg --detach-sign problem for your own builds, I'm morbidly curious as to the technical limitations and would like to know why you think these technical limitations will vanish once you're uploading them to repos.archlinux.org specifically. -- Eli Schwartz Bug Wrangler and Trusted User

On Sat, 2020-11-21 at 16:58 +0100, Andreas Radke via arch-dev-public wrote:

Am Sat, 21 Nov 2020 14:34:24 +0000 schrieb Filipe Laíns via arch-dev-public :

...

Does anyone have any big issue with this? What are your thoughts?

[1] https://www.python.org/downloads/

Cheers, Filipe Laíns

-1

Arch is yours. Whoever needs more and older releases on the system - just do it yourself! In the past we said "use abs and AUR - Arch is the base to make it your own".

This argument can be used to deny adding any package to the repos. You need this library, tool, etc.? Just add it yourself. Why are we packaging software that is used by far less people but we can't package these Python interpreters which are being actively missed by people?

I don't want to see users raising bugs that something doesn't work with an older version of python. And I don't want to see these requests pop up every now and then to add even more stuff in different versions.

We already have multiple versions of Java, Ruby, Javascript, etc. hell, even Python. I don't think having people opening bugs because they are deliberately using an older version of Python is a big problem. It hasn't been for any of the other languages, I don't think it will be here. I think this is an hypothetical that doesn't really materialize into reality.

It's sad enough we still have python2 and gtk2 around. To have gcc9 around and other duplicates is not what I want to see growing in Arch.

What you call sad I call a bad UX. Why do we need to force users to compile active releases of the Python interpreters themselves, which can take a long time if they are building with optimization, or to resort to pre-built repos with much lower security standards than us, when there are people willing to maintain them? I can't understand how it's sad to help out users by not forcing them to resort the sort of things I mentioned above, as long as we are not struggling to do so. I like helping people, that's why I am a packager, that is the opposite of sad for me, so I really can't understand this.

I don't want to see our distribution transformed into another Debian.

That is not what is happening. Cheers, Filipe Laíns

On Sat, 2020-11-21 at 11:24 -0600, Doug Newgard via arch-dev-public wrote:

On Sat, 21 Nov 2020 16:59:21 +0000 Filipe Laíns via arch-dev-public wrote:

...

On Sat, 2020-11-21 at 16:58 +0100, Andreas Radke via arch-dev- public wrote:

...

Am Sat, 21 Nov 2020 14:34:24 +0000 schrieb Filipe Laíns via arch-dev-public :

 

...

Does anyone have any big issue with this? What are your thoughts?

[1] https://www.python.org/downloads/

Cheers, Filipe Laíns 

-1

Arch is yours. Whoever needs more and older releases on the system - just do it yourself! In the past we said "use abs and AUR - Arch is the base to make it your own". 

This argument can be used to deny adding any package to the repos. You need this library, tool, etc.? Just add it yourself.

Why are we packaging software that is used by far less people but we can't package these Python interpreters which are being actively missed by people?

...

I don't want to see users raising bugs that something doesn't work with an older version of python. And I don't want to see these requests pop up every now and then to add even more stuff in different versions. 

We already have multiple versions of Java, Ruby, Javascript, etc. hell, even Python. I don't think having people opening bugs because they are deliberately using an older version of Python is a big problem. It hasn't been for any of the other languages, I don't think it will be here. I think this is an hypothetical that doesn't really materialize into reality.

...

It's sad enough we still have python2 and gtk2 around. To have gcc9 around and other duplicates is not what I want to see growing in Arch.  

What you call sad I call a bad UX. Why do we need to force users to compile active releases of the Python interpreters themselves, which can take a long time if they are building with optimization, or to resort to pre-built repos with much lower security standards than us, when there are people willing to maintain them?

I can't understand how it's sad to help out users by not forcing them to resort the sort of things I mentioned above, as long as we are not struggling to do so. I like helping people, that's why I am a packager, that is the opposite of sad for me, so I really can't understand this.

It's more concerning to me that you can't understand this argument than anything else so far. Arch keeps old things around in the repos when they're required by other things in the repos. It's a necessary evil, not something to be actively encouraged.

I understand that. I am not asking to put all releases of Python on the repos, only the active ones, which people are using.

...

...

I don't want to see our distribution transformed into another Debian. 

That is not what is happening.

Cheers, Filipe Laíns

-- Filipe Laíns

On Sat, 2020-11-21 at 19:11 +0100, Bartłomiej Piotrowski via arch-dev-public wrote: On 21/11/2020 18.48, Filipe Laíns via arch-dev-public wrote:

I understand that. I am not asking to put all releases of Python on the repos, only the active ones, which people are using.

I presume you can back it up with numbers how widely 3.7 and 3.6 are used by Arch users. All I can see is that both have less than 30 votes in AUR. Even if I take into account how irrelevant AUR votes are, I assume the problem is exaggerated. I have some metrics that might be relevant. The people that are missing these interpreters generally use pyenv to make up for it, which is not an optimal solution, but the most common when the interpreters are missing. Looking at some package stats for pyenv[1], we see a usage of 3,4%. I think that is a big enough number. pyenv does have another usage, which is managing custom Python interpreters, which is mainly used in CPython development, but I'd say that only a small percentage of the users use it uniquely for that. We can also look directly at the python36 and python37 packages reported, which users might be getting from the AUR or user repos. python37 has 1.18% [2] python36 has 1.63% [3] The main reason why the packages might be required, as I explained in the original email, is for package testing. It puts every maintainer and most contributors to projects that target Python 3.6 or 3.7 in need of an interpreter. Most open source Python project target at least a few different Python versions. These metrics are not perfect by any means, but I do think they present relevant enough data to show an existent need. Why are we packaging software that is used by far less people but we can't package these Python interpreters which are being actively missed by people? Our approach of "YOLO push random new packages to repos" is something I never agreed with, for the record. Unfortunately it's a kingdom with almost no rules. BP [1] https://pkgstats.archlinux.de/api/packages/pyenv [2] https://pkgstats.archlinux.de/api/packages/python36 [3] https://pkgstats.archlinux.de/api/packages/python37 Cheers, Filipe Laíns

On 11/21/20 11:59 AM, Filipe Laíns via arch-dev-public wrote:

On Sat, 2020-11-21 at 16:58 +0100, Andreas Radke via arch-dev-public wrote:

...

Am Sat, 21 Nov 2020 14:34:24 +0000 schrieb Filipe Laíns via arch-dev-public :

...

Does anyone have any big issue with this? What are your thoughts?

[1] https://www.python.org/downloads/

Cheers, Filipe Laíns

-1

Arch is yours. Whoever needs more and older releases on the system - just do it yourself! In the past we said "use abs and AUR - Arch is the base to make it your own".

This argument can be used to deny adding any package to the repos. You need this library, tool, etc.? Just add it yourself.

Why are we packaging software that is used by far less people but we can't package these Python interpreters which are being actively missed by people?

...

I don't want to see users raising bugs that something doesn't work with an older version of python. And I don't want to see these requests pop up every now and then to add even more stuff in different versions.

We already have multiple versions of Java, Ruby, Javascript, etc. hell,

Your analysis is correct, it is indeed hell. I'm not sure why that is an argument in favor of doing even more of it though. Now, if you were proposing to get rid of some of this, I could get behind that.

even Python.

I don't know if maybe you've missed the fact that we've spent like a year now aggressively upgrading or removing leaf packages depending on python2 in search of that glorious Promised Future where we can finally remove the python2 package itself? And we're doing that even for a major incompatible release that most people argue is actually a different language. I'm not sure how this suddenly became an argument to package a ton of point releases that aren't even incompatible.

I don't think having people opening bugs because they are deliberately using an older version of Python is a big problem. It hasn't been for any of the other languages, I don't think it will be here. I think this is an hypothetical that doesn't really materialize into reality.

You're proposing something far beyond the scope of what we do for other languages, and ignoring that for other languages we only do it if driven to in desperation because another official repository package depends on it. We don't introduce leaf packages of ancient versions of currently packaged interpreters just for fun. I don't even think anyone else does that either.

...

It's sad enough we still have python2 and gtk2 around. To have gcc9 around and other duplicates is not what I want to see growing in Arch.

What you call sad I call a bad UX. Why do we need to force users to compile active releases of the Python interpreters themselves, which can take a long time if they are building with optimization, or to resort to pre-built repos with much lower security standards than us, when there are people willing to maintain them?

I can't understand how it's sad to help out users by not forcing them to resort the sort of things I mentioned above, as long as we are not struggling to do so. I like helping people, that's why I am a packager, that is the opposite of sad for me, so I really can't understand this.

I have... no idea what the problem is supposed to be here? I'm desperately trying to understand what the actual point of your proposal is. You want to package old versions of the python3 interpreter "but not modules", because "people would like to use it for tox". Old versions of stuff that people need for who knows what weird reason is *the textbook reason* why the AUR exists. How long it takes to compile stuff in the AUR is not our problem, and we don't have a rationale to upload gcc-{4,5,6,7,8,9} because actual users need to use a lot of diverse compilers "to test that it still works on really old compilers". Likewise, if people want to test support for old versions of python3, they have options which don't involve uploading trash into [community]. - Do what normal people do, and build AUR packages they need - Ask FFY00 to provide signed repos - Use travis CI, which doesn't use distro packages but provides a diverse set of hand-compiled python versions, which is apparently some kind of gold standard for doing regression testing on lots of different python releases ;) - Use pyenv Apparently none of these are viable options in your mind, and the most Arch-like of them, option #1 "use the AUR" you believe to be problematic because it takes a long time to build with optimizations... even though you yourself said those packages don't enable optimizations and don't take a long time??? (If you're using python-$old exclusively for regression tests in tox, you don't need a super optimized version.)

...

I don't want to see our distribution transformed into another Debian.

That is not what is happening.

Yeah, even Debian doesn't do this. Travis-ci.com does, though. I don't see any reason why [community] needs to grow to support the needs of exclusively Continuous Integration testsuite matrixes. -- Eli Schwartz Bug Wrangler and Trusted User

On Sun, 22 Nov 2020 15:05:04 +0000 Filipe Laíns via arch-dev-public wrote:

As long as there is someone willing to maintain the packages, why would we block it? I doesn't really have any effect on the rest of the people. If the person start having trouble maintaining them, they can be dropped. It doesn't affect you and helps other people.

Cheers, Filipe Laíns

Why did you start the discussion if you plan on ignoring other people's opinions anyway? Doug

On Sun, 22 Nov 2020 15:32:22 +0000 Filipe Laíns via arch-dev-public wrote:

On Sun, 2020-11-22 at 09:13 -0600, Doug Newgard via arch-dev-public wrote:

...

Why did you start the discussion if you plan on ignoring other people's opinions anyway?

Doug

I am not ignoring Andreas' or Eli's opinion, I understand that they don't want these packages in the repos. What I am asking is why they don't want them there?

I don't appreciate you trimming the paragraph I was replying to. I makes my comments look completely different without context. The paragraph specifically said that you see no reason to block anything as long as someone is willing to maintain it. That means that other's opinions in this disussion don't matter, unless I misunderstood? Doug

On 11/22/20 10:05 AM, Filipe Laíns wrote:

On Sat, 2020-11-21 at 20:24 -0500, Eli Schwartz via arch-dev-public wrote:

...

Your analysis is correct, it is indeed hell. I'm not sure why that is an argument in favor of doing even more of it though.

Now, if you were proposing to get rid of some of this, I could get behind that.

It was not an analysis, "hell" was an used here as an idiom.

https://i.imgur.com/axJmn.gif You were unintentionally accurate.

...

...

even Python.

I don't know if maybe you've missed the fact that we've spent like a year now aggressively upgrading or removing leaf packages depending on python2 in search of that glorious Promised Future where we can finally remove the python2 package itself?

I did not miss that. I absolutely want to see the python2 modules removed, not _necessarily_ the python2 interpreter.

Now I'm starting to see part of the reason why people are having a difference of opinions here. I think I probably speak for a bunch of people when I say I'd like to see the interpreter dropped due to not being needed by any packages *and* being an end of life interpreter. The interpreter is only useful inasmuch as people use it to run software, so if we don't have any... I expect by the time we finally remove all reverse dependencies for it, the gap between the python2 EOL and its removal from the distro will be quite a bit biger than it currently is.

...

And we're doing that even for a major incompatible release that most people argue is actually a different language.

I'm not sure how this suddenly became an argument to package a ton of point releases that aren't even incompatible.

Except they are... Python does not follow semver, only patch releases are supposed to be compatible. They do try to keep breakage to a minimum, but it definitely exists. One very obvious example is the introduction of "async" and "await" as reserved keywords. There are a bunch of other backwards incompatible changes, so yes, they are incompatible releases, to the point you could call them different languages.

This is the first time I ever heard anyone describe "backwards incompatible changes means it is now a different programming language". I had backwards incompatible changes in a patchlevel bugfix in the python 3.8.x release line causing immediate traceback and SystemExit on a program's startup, does that mean every bugfix is now a different programming language? (I mean this quite literally. Python bugfixed a function not working as expected under certain situations. The program it broke was working around the failure by manually doing $thing, the traceback happened when you try to do $thing twice.)

...

...

I don't think having people opening bugs because they are deliberately using an older version of Python is a big problem. It hasn't been for any of the other languages, I don't think it will be here. I think this is an hypothetical that doesn't really materialize into reality.

You're proposing something far beyond the scope of what we do for other languages, and ignoring that for other languages we only do it if driven to in desperation because another official repository package depends on it.

We don't introduce leaf packages of ancient versions of currently packaged interpreters just for fun. I don't even think anyone else does that either.

That was not of the point of me bringing up other languages. The whole point of me bringing up other languages was to show that Andreas' concern is unlikely to materialize into reality.

You think no one reports bugs for python-* packages that we should provide the python2 version? I promise you, you're wrong. I've closed those bugs, so I know they exist. You specifically brought up Java and Javascript, languages where each application is forced to vendor their entire dependency list rather than using system modules. For this reason, people don't report the kind of bugs Andreas is concerned about for Java/nodejs... So I assumed you could not possibly be talking about Andreas' concern, and I responded somewhat tangentially myself. We can, of course, talk about Andreas' concern, but given I've now submitted the authoritative facts on the matter, I'm not sure what else there is to say. It does indisputably happen right now. The logical inference is, it will happen *more*, if we add more runtimes for people to submit bugtracker tickets for.

...

You want to package old versions of the python3 interpreter "but not modules", because "people would like to use it for tox".

People can use it tox or to create virtual environments directly. The main point of the proposal is that people have use for the old interpreters and I would like to provide them.

The main point of my objection is that their use isn't to run software, it's to regression test it -- something more suited to Continuous Integration which already targets a wide range of operating systems *and* interpreter versions, and which can be used to block merging.

...

Old versions of stuff that people need for who knows what weird reason is *the textbook reason* why the AUR exists. How long it takes to compile stuff in the AUR is not our problem, and we don't have a rationale to upload gcc-{4,5,6,7,8,9} because actual users need to use a lot of diverse compilers "to test that it still works on really old compilers".

AFAIK, the textbook reason why the AUR exists is to provide software that isn't in the official repos. It being older versions should be completely incidental, but not the "textbook reason".

The AUR exists to provide software which is not in the official repos for practical OR ideological reasons. Old stuff is ideological reasons.

...

Likewise, if people want to test support for old versions of python3, they have options which don't involve uploading trash into [community].

I like how you refereed to the currently active Python releases as "trash", will keep that in mind.

I like how you reinterpret my reference to Python releases which are in LTS "only security fixes are permitted and even then, we won't release Windows or macOS binaries, just source code for you to compile yourself" mode as "currently active". They are not active. Active would imply that on https://devguide.python.org/#branchstatus they are marked as "bugfix" or "feature", indicating the Python developers will actually fix bugs in them and release multi-platform installers in addition to source code. So python >= 3.8 I would call "not trash", but you want to upload python 3.6 and python 3.7, which I would call trash at this point. They are expired versions. They went moldy, and the only people who should be using them are people who have been storing their entire computing environment in the freezer in order to eke out another year or two of emergency use.

...

- Do what normal people do, and build AUR packages they need - Ask FFY00 to provide signed repos - Use travis CI, which doesn't use distro packages but provides a   diverse set of hand-compiled python versions, which is apparently some   kind of gold standard for doing regression testing on lots of   different python releases ;) - Use pyenv

Apparently none of these are viable options in your mind, and the most Arch-like of them, option #1 "use the AUR" you believe to be problematic because it takes a long time to build with optimizations... even though you yourself said those packages don't enable optimizations and don't take a long time???

(If you're using python-$old exclusively for regression tests in tox, you don't need a super optimized version.)

Tests can take a long time to run, multiply that by the interpreters you need to run them against. But that is only part of the reason.

Why are you nitpicking the offhand comment I made, rather than my actual recommendations? Again, this is a non-issue if you stick the packages in an unofficial user repository and enable optimizations. And users are welcome to make the tradeoff of less time compiling in exchange for slightly longer test runtimes. Building a package is a one-time cost anyway, which you could do overnight.

As long as there is someone willing to maintain the packages, why would we block it? I doesn't really have any effect on the rest of the people. If the person start having trouble maintaining them, they can be dropped. It doesn't affect you and helps other people.

One reason is, for the same reason we ideologically don't want gcc-{4,5,6,7,8,9} in the repos, no matter how "little effect" it has on other people. We don't wish to promote using old things. Sometimes we include *one* old version of gcc in desperation, because CUDA cannot be convinced to use current gcc and cannot be fixed because it's closed-source. Even this bothers Andreas (and me, and actually, it bothers Sven too even though he's the one that uploaded gcc9 to begin with) but we put up with it because it's a dependency of something else, and we only package the one old version needed and upgrade it as soon as cuda supports something more modern. We wouldn't have gcc9 merely "for the public service convenience of anyone who needs it". -- Eli Schwartz Bug Wrangler and Trusted User

Remember, write the angry email first, wait a half hour, then re-write. Please keep the IRC flamewars to IRC where some of us can avoid it without losing contact with important discussion. :)

On Thu, 2020-11-26 at 21:21 +0100, Jelle van der Waa wrote:

On 21/11/2020 15:34, Filipe Laíns via arch-dev-public wrote:

...

Hi all,

I want to propose adding all active Python versions to [community], not just the latest one. This would only entail adding the interpreter itself, no other packages.

Having access to interpreters for older active versions is really helpful for Python developers. This allows them to easily run test suites against older versions. It is very common for developers to maintain software against a couple major releases. Tools like tox or nox are able to automate testing against multiple Python versions, just needing the interpreter.

The current active Python releases are:    - 3.9    - 3.8    - 3.7    - 3.6    - 2.7

The list can be found here[1].

So, I propose introducing 2 new packages:    - python3.7    - python3.6

And when we update the python package to 3.9:    - python3.8

I am not super in favor of providing an old version, as we strive to provide the latest and greatest. However some things come to mind:

We would still provide the latest and greatest :P

* Does the Python foundation still maintain older Python versions and do they provide proper security updates?

It depends on the version. Python 3.6 and 3.7 are in security maintenance status. I am okay limiting my proposal to interpreters which are getting security updates, which at the time being doesn't change anything regarding the versions affected. It is also not clear in the proposal, but I have mentioned in the replies. I only want the interpreters there as long as they do not give us any issues. If a interpreter starts needing older dependencies or otherwise becomes a maintenance burden, it would be removed.

* This seems to be purely for Development right? Users would use this Python version to bootstrap a virtualenv I guess?

The target is development, yes. You are correct, users, or tools, could use the interpreter to bootstrap a virtual environment using either virtualenv, which supports using external interpreters, or the builtin venv module.

Greetings,

Jelle

Cheers, Filipe Laíns