[aur-general] Python-3.x transition with python-2.7 update
Hi all, Here comes a rebuild so large that our TODO list had trouble handling it! Hopefully all packages are now in the rebuild list.... At a total of 518 packages long, it puts the combined libpng/libjpeg rebuild to shame. Python-2.7 has been releases and will be the last 2.x official release of python. So it is time to switch to python-3.x as our /usr/bin/python and python-2.7 as our /usr/bin/python2. See http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List for all the details about how to achieve this. It is actually not that hard. I had a system converted when python-3.1 was released as a test run. The main key is to build packages in a clean chroot so that they detect and point their files to /usr/bin/python2. Some packages are stupid and require a sed at the end of packaging to fix that. Because this rebuild is crazy stupid, I would like to plan when it is going to occur. We will need to clear out [testing] as much as possible over the coming week or two (what is happening with perl...). Also, a new KDE is a the beginning of next month so I would not want to conflict with that. Any other major rebuilds on the way? Should we do this in a separate repo? Allan
On Mon, Jul 5, 2010 at 4:11 PM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
Here comes a rebuild so large that our TODO list had trouble handling it! Hopefully all packages are now in the rebuild list.... At a total of 518 packages long, it puts the combined libpng/libjpeg rebuild to shame.
Python-2.7 has been releases and will be the last 2.x official release of python. So it is time to switch to python-3.x as our /usr/bin/python and python-2.7 as our /usr/bin/python2. See http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List for all the details about how to achieve this.
It is actually not that hard. I had a system converted when python-3.1 was released as a test run. The main key is to build packages in a clean chroot so that they detect and point their files to /usr/bin/python2. Some packages are stupid and require a sed at the end of packaging to fix that.
Because this rebuild is crazy stupid, I would like to plan when it is going to occur. We will need to clear out [testing] as much as possible over the coming week or two (what is happening with perl...). Also, a new KDE is a the beginning of next month so I would not want to conflict with that. Any other major rebuilds on the way? Should we do this in a separate repo?
Allan
What will be the benefits of switching to Python 3.x? The newly released Python 2.7 contains many features [1] from the 3.x branch, which makes the transition at this point less appealing. Please don't take my question the wrong way, but I read in the wiki where you say: "During the rebuild, care needs to be taken to make sure all rebuilt packages are really using the python2 binary." Is everything going to be rebuilt against Python 2? If yes, isn't that the current situation? You obviously have given this more thought that I have, but it seems to me that a more preferable option would be to just update to Python 2.7. Regards. ---- [1] http://python.org/download/releases/2.7/
On Mon, Jul 5, 2010 at 8:33 AM, Evangelos Foutras <foutrelis@gmail.com> wrote:
On Mon, Jul 5, 2010 at 4:11 PM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
Here comes a rebuild so large that our TODO list had trouble handling it! Hopefully all packages are now in the rebuild list.... At a total of 518 packages long, it puts the combined libpng/libjpeg rebuild to shame.
Python-2.7 has been releases and will be the last 2.x official release of python. So it is time to switch to python-3.x as our /usr/bin/python and python-2.7 as our /usr/bin/python2. See http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List for all the details about how to achieve this.
It is actually not that hard. I had a system converted when python-3.1 was released as a test run. The main key is to build packages in a clean chroot so that they detect and point their files to /usr/bin/python2. Some packages are stupid and require a sed at the end of packaging to fix that.
Because this rebuild is crazy stupid, I would like to plan when it is going to occur. We will need to clear out [testing] as much as possible over the coming week or two (what is happening with perl...). Also, a new KDE is a the beginning of next month so I would not want to conflict with that. Any other major rebuilds on the way? Should we do this in a separate repo?
Allan
What will be the benefits of switching to Python 3.x? The newly released Python 2.7 contains many features [1] from the 3.x branch, which makes the transition at this point less appealing. Please don't take my question the wrong way, but I read in the wiki where you say:
"During the rebuild, care needs to be taken to make sure all rebuilt packages are really using the python2 binary."
Is everything going to be rebuilt against Python 2? If yes, isn't that the current situation?
You obviously have given this more thought that I have, but it seems to me that a more preferable option would be to just update to Python 2.7.
Regards.
Hmm, the more I think about this, the more this solution sounds appealing, since it wouldn't require that much work. We could possibly move to python 3.x at a later time when one of the 2.7.x fixes comes out and when more packages have a chance to transition to python 3. I really don't know the reasons behind this move, but this is just my 2¢ for now :) Cheers!
Am 05.07.2010 16:33, schrieb Thomas Dziedzic:
Hmm, the more I think about this, the more this solution sounds appealing, since it wouldn't require that much work. We could possibly move to python 3.x at a later time when one of the 2.7.x fixes comes out and when more packages have a chance to transition to python 3. I really don't know the reasons behind this move, but this is just my 2¢ for now :) Cheers!
1+
On 05/07/10 23:11, Allan McRae wrote:
Hi all,
Here comes a rebuild so large that our TODO list had trouble handling it! Hopefully all packages are now in the rebuild list.... At a total of 518 packages long, it puts the combined libpng/libjpeg rebuild to shame.
Python-2.7 has been releases and will be the last 2.x official release of python. So it is time to switch to python-3.x as our /usr/bin/python and python-2.7 as our /usr/bin/python2. See http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List for all the details about how to achieve this.
It is actually not that hard. I had a system converted when python-3.1 was released as a test run. The main key is to build packages in a clean chroot so that they detect and point their files to /usr/bin/python2. Some packages are stupid and require a sed at the end of packaging to fix that.
Because this rebuild is crazy stupid, I would like to plan when it is going to occur. We will need to clear out [testing] as much as possible over the coming week or two (what is happening with perl...). Also, a new KDE is a the beginning of next month so I would not want to conflict with that. Any other major rebuilds on the way? Should we do this in a separate repo?
Just to clarify why this _is_ happening. python-2.7 will enable USC4 (UTF8) and will not enable the usual stupid hack of including the previous pythons include path (the past has shown that this breaks too much). So ~500 out of the 518 packages are going to need to be rebuilt anyway... I think this is a major point people are missing. So the choice is: 1) massive rebuild now _and_ even more massive rebuild later when python3 is more widely used with lots of package renames (python3-foo would need renamed python-foo and python-foo would become python2-foo). We actually can not handle that dual rename with provides/replaces/conflicts.... 2) one massive rebuild allowing the packages to gradually transition their packages to python3. As in the wiki, there will be no forced package renaming (let the maintainers introduce a python2-foo package only if they want to supply a python2 and python3 version) but the description should be updated to say python2. #1 is completely crazy and can not be done cleanly on a rolling release distro. #2 is slightly crazy and works fine (one of my installs underwent this conversion). This has been in planning for over a year. I have trialled various ways of achieving this transition over the last year and have concluded that doing it with the python-2.7 release is optimal because: - we are doing most of the rebuilds anyway - it is early enough that our repo is not filled with python3-foo packages - most major python projects have basic python3 support in their developmental code or are working on it (see the number of GSoC projects doing this transition....) so we will see python3 packages soon. I am not delaying the inevitable, especially when the delay will hurt us more in the long run. Allan
On 07/05/10 19:45, Allan McRae wrote:
Just to clarify why this _is_ happening.
...
Agreed. Also I don't think having python 2.x being called "python2" will hurt us too much (we just have to fix the packages that need fixing, in the ways Allan detailed, according to his testing-experience) and it may be a service to upstreams for us to be a distro that adopts python3 in this way early, so that these upstreams can get their own code better aligned to cope with the (hopefully-)coming reality. -Isaac
Just to clarify why this _is_ happening.
python-2.7 will enable USC4 (UTF8) and will not enable the usual stupid hack of including the previous pythons include path (the past has shown that this breaks too much). So ~500 out of the 518 packages are going to need to be rebuilt anyway... I think this is a major point people are missing.
So the choice is:
1) massive rebuild now _and_ even more massive rebuild later when python3 is more widely used with lots of package renames (python3-foo would need renamed python-foo and python-foo would become python2-foo). We actually can not handle that dual rename with provides/replaces/conflicts....
2) one massive rebuild allowing the packages to gradually transition their packages to python3. As in the wiki, there will be no forced package renaming (let the maintainers introduce a python2-foo package only if they want to supply a python2 and python3 version) but the description should be updated to say python2.
#1 is completely crazy and can not be done cleanly on a rolling release distro. #2 is slightly crazy and works fine (one of my installs underwent this conversion).
This has been in planning for over a year. I have trialled various ways of achieving this transition over the last year and have concluded that doing it with the python-2.7 release is optimal because: - we are doing most of the rebuilds anyway - it is early enough that our repo is not filled with python3-foo packages - most major python projects have basic python3 support in their developmental code or are working on it (see the number of GSoC projects doing this transition....) so we will see python3 packages soon.
I am not delaying the inevitable, especially when the delay will hurt us more in the long run.
Allan
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename. Lukas
On 07/06/10 01:57, Lukáš Jirkovský wrote:
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename.
Hi Lukas, Can you present a technical reason against doing the renaming now? Because as far as I can see, Allan has worked out the kinks and it will actually not harm you as a regular user at all... (unless you write personal scripts in python that you want to work with #!something on multiple distros? (then you probably want to run them in python version 2) .. I'm not sure I can think of an easy way to do that; maybe for each distro you use you could put a symlink in /usr/local/bin/python2 for example.) -Isaac
On 6 July 2010 10:19, Isaac Dupree <ml@isaac.cedarswampstudios.org> wrote:
On 07/06/10 01:57, Lukáš Jirkovský wrote:
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename.
Hi Lukas, Can you present a technical reason against doing the renaming now? Because as far as I can see, Allan has worked out the kinks and it will actually not harm you as a regular user at all...
(unless you write personal scripts in python that you want to work with #!something on multiple distros? (then you probably want to run them in python version 2) .. I'm not sure I can think of an easy way to do that; maybe for each distro you use you could put a symlink in /usr/local/bin/python2 for example.)
-Isaac
Hi Isaac, I don't write Python scripts but yeah, I think this is a real problem. The other problem is that there are not many users of python 3 out there. In a more subjective way I think whenever something is set as default it should be the one which has most users (in both terms of people and software). Lukas
On Tue, 2010-07-06 at 10:51 +0200, Lukáš Jirkovský wrote:
On 6 July 2010 10:19, Isaac Dupree <ml@isaac.cedarswampstudios.org> wrote:
On 07/06/10 01:57, Lukáš Jirkovský wrote:
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename.
Hi Lukas, Can you present a technical reason against doing the renaming now? Because as far as I can see, Allan has worked out the kinks and it will actually not harm you as a regular user at all...
(unless you write personal scripts in python that you want to work with #!something on multiple distros? (then you probably want to run them in python version 2) .. I'm not sure I can think of an easy way to do that; maybe for each distro you use you could put a symlink in /usr/local/bin/python2 for example.)
-Isaac
Hi Isaac, I don't write Python scripts but yeah, I think this is a real problem. The other problem is that there are not many users of python 3 out there.
In a more subjective way I think whenever something is set as default it should be the one which has most users (in both terms of people and software).
Lukas
As another user (who doesn't write Python), I'd state that 'majority usage' is a pretty poor guideline for users of a Linux distro, and a relatively small one at that. I'm all for the option which reduces workload on the packagers. Of course if things break big-time then it may be a problem, but that's what [testing] is for, and those of us using it should know what to do if/when breakage occurs.
On 6 July 2010 20:09, Ng Oon-Ee <ngoonee@gmail.com> wrote:
On Tue, 2010-07-06 at 10:51 +0200, Lukáš Jirkovský wrote:
On 6 July 2010 10:19, Isaac Dupree <ml@isaac.cedarswampstudios.org> wrote:
On 07/06/10 01:57, Lukáš Jirkovský wrote:
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename.
Hi Lukas, Can you present a technical reason against doing the renaming now? Because as far as I can see, Allan has worked out the kinks and it will actually not harm you as a regular user at all...
(unless you write personal scripts in python that you want to work with #!something on multiple distros? (then you probably want to run them in python version 2) .. I'm not sure I can think of an easy way to do that; maybe for each distro you use you could put a symlink in /usr/local/bin/python2 for example.)
-Isaac
Hi Isaac, I don't write Python scripts but yeah, I think this is a real problem. The other problem is that there are not many users of python 3 out there.
In a more subjective way I think whenever something is set as default it should be the one which has most users (in both terms of people and software).
Lukas
As another user (who doesn't write Python), I'd state that 'majority usage' is a pretty poor guideline for users of a Linux distro, and a relatively small one at that.
I'm all for the option which reduces workload on the packagers. Of course if things break big-time then it may be a problem, but that's what [testing] is for, and those of us using it should know what to do if/when breakage occurs.
Some more background info for those who are not that familiar why the Python 2 vs. 3 is such a big problem (there seem to plenty of people, and sorry for the ones who already know this inside out): http://wiki.python.org/moin/Python2orPython3
From that page: "Popular modules that don't yet support Python 3 include Twisted (for networking and a bunch of other stuff), gevent (like Twisted but different), Django and Pylons (for building websites), PyGTK and PySide (for making GUIs), py2exe (for packaging your application for Windows users), PIL (for processing images), numpy (for number crunching)..."
Thus I would mind a rebuild less, than losing my daily numpy/scipy/PyGTK... Arch is not in a position to force these packages to update to Python 3, and such I don't think it's a good idea to bump the default version up to 3. On the other hand, there are plenty of people who are experienced in code fixes, so maybe there would be some who would join the porting effort for those packages that they use on a regular basis. This could accelerate the transition. Cheers, Greg
On 06/07/10 22:54, Gergely Imreh wrote:
On 6 July 2010 20:09, Ng Oon-Ee<ngoonee@gmail.com> wrote:
On Tue, 2010-07-06 at 10:51 +0200, Lukáš Jirkovský wrote:
On 6 July 2010 10:19, Isaac Dupree<ml@isaac.cedarswampstudios.org> wrote:
On 07/06/10 01:57, Lukáš Jirkovský wrote:
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename.
Hi Lukas, Can you present a technical reason against doing the renaming now? Because as far as I can see, Allan has worked out the kinks and it will actually not harm you as a regular user at all...
(unless you write personal scripts in python that you want to work with #!something on multiple distros? (then you probably want to run them in python version 2) .. I'm not sure I can think of an easy way to do that; maybe for each distro you use you could put a symlink in /usr/local/bin/python2 for example.)
-Isaac
Hi Isaac, I don't write Python scripts but yeah, I think this is a real problem. The other problem is that there are not many users of python 3 out there.
In a more subjective way I think whenever something is set as default it should be the one which has most users (in both terms of people and software).
Lukas
As another user (who doesn't write Python), I'd state that 'majority usage' is a pretty poor guideline for users of a Linux distro, and a relatively small one at that.
I'm all for the option which reduces workload on the packagers. Of course if things break big-time then it may be a problem, but that's what [testing] is for, and those of us using it should know what to do if/when breakage occurs.
Some more background info for those who are not that familiar why the Python 2 vs. 3 is such a big problem (there seem to plenty of people, and sorry for the ones who already know this inside out): http://wiki.python.org/moin/Python2orPython3
From that page: "Popular modules that don't yet support Python 3 include Twisted (for networking and a bunch of other stuff), gevent (like Twisted but different), Django and Pylons (for building websites), PyGTK and PySide (for making GUIs), py2exe (for packaging your application for Windows users), PIL (for processing images), numpy (for number crunching)..."
Thus I would mind a rebuild less, than losing my daily numpy/scipy/PyGTK...
Do you seriously think would be removing those from the repos? That would be insane... numpy/scipy/pygtk/etc will all be in the repos and working. The only thing you will have to do is use "#!/usr/bin/env python2.7" (or just python2) at the start of your script. Allan
On Tue, 06 Jul 2010 23:25:15 +1000 Allan McRae <allan@archlinux.org> wrote:
On 06/07/10 22:54, Gergely Imreh wrote:
On 6 July 2010 20:09, Ng Oon-Ee<ngoonee@gmail.com> wrote:
On Tue, 2010-07-06 at 10:51 +0200, Lukáš Jirkovský wrote:
On 6 July 2010 10:19, Isaac Dupree<ml@isaac.cedarswampstudios.org> wrote:
On 07/06/10 01:57, Lukáš Jirkovský wrote:
Hello Allan, I know that I'm just a regular user but I'd like to express my opinion too. I think the transition should be done when most modules and applications support Python 3. I'd not be surprised if the transition of majority of modules would take several years. By that time there may be a way how to do a dual rename.
Hi Lukas, Can you present a technical reason against doing the renaming now? Because as far as I can see, Allan has worked out the kinks and it will actually not harm you as a regular user at all...
(unless you write personal scripts in python that you want to work with #!something on multiple distros? (then you probably want to run them in python version 2) .. I'm not sure I can think of an easy way to do that; maybe for each distro you use you could put a symlink in /usr/local/bin/python2 for example.)
-Isaac
Hi Isaac, I don't write Python scripts but yeah, I think this is a real problem. The other problem is that there are not many users of python 3 out there.
In a more subjective way I think whenever something is set as default it should be the one which has most users (in both terms of people and software).
Lukas
As another user (who doesn't write Python), I'd state that 'majority usage' is a pretty poor guideline for users of a Linux distro, and a relatively small one at that.
I'm all for the option which reduces workload on the packagers. Of course if things break big-time then it may be a problem, but that's what [testing] is for, and those of us using it should know what to do if/when breakage occurs.
Some more background info for those who are not that familiar why the Python 2 vs. 3 is such a big problem (there seem to plenty of people, and sorry for the ones who already know this inside out): http://wiki.python.org/moin/Python2orPython3
From that page: "Popular modules that don't yet support Python 3 include Twisted (for networking and a bunch of other stuff), gevent (like Twisted but different), Django and Pylons (for building websites), PyGTK and PySide (for making GUIs), py2exe (for packaging your application for Windows users), PIL (for processing images), numpy (for number crunching)..."
Thus I would mind a rebuild less, than losing my daily numpy/scipy/PyGTK...
Do you seriously think would be removing those from the repos? That would be insane...
numpy/scipy/pygtk/etc will all be in the repos and working. The only thing you will have to do is use "#!/usr/bin/env python2.7" (or just python2) at the start of your script.
Allan
At the moment most programs don't know to use #!/usr/bin/env python2. Therefore many python programs that users will download will not work without modification (although a very small modification). I do not think it is time yet. I do not believe that there is a large enough group of users of python3 to make it the default. Please note that unlike everyone else who has commented, I do a lot of python development. It would not affect me personally. I have already figured out that my script will work more smoothly in the future if I use "python2" over "python". Is there a good reason to ever switch python to python3? The option of just not switching at all has never come up. At some point we would need to move everything over, but why not continue to call it python3?
06.07.2010 16:48, Stephen Weinberg:
Is there a good reason to ever switch python to python3? The option of just not switching at all has never come up. At some point we would need to move everything over, but why not continue to call it python3?
I would support this idea. The amount of time needed for the transition (python->python2,python3->python) is later on needed for further incompatible upgrades of python. Just think about python4 :-). I guess this is a common problem in naming packages for major releases. Two years ago I wished they would have had decided to use 'kde4' instead of 'kde' although it might take some time until kde5 is out. Matthias
When I saw the transition plans my first thought was "oh shhhh*t!" I really would like to start using python3 for everything but I don't see this coming anytime soon. There are libraries that I need that have not been ported to python3 yet. Actually I use a lot of personal python scripts on a variety of distributions (arch, debian, centos, slackware and even freebsd) keeping them in sync using a git repository and now archlinux is going to make things complicated but I don't really see any benefit from it. Just my opinion.
On 08/07/10 22:25, Massimiliano Torromeo wrote:
When I saw the transition plans my first thought was "oh shhhh*t!"
I really would like to start using python3 for everything but I don't see this coming anytime soon. There are libraries that I need that have not been ported to python3 yet.
Actually I use a lot of personal python scripts on a variety of distributions (arch, debian, centos, slackware and even freebsd) keeping them in sync using a git repository and now archlinux is going to make things complicated but I don't really see any benefit from it.
Just put a python2 symlink somewhere in your path pointing to your chosen python version (if that distro does not already provide one) and use "/usr/bin/env python2" at the start of your scripts. In fact, that probably makes things less complicated as you can choose what 2.x version that symlink points at rather than whatever version /usr/bin/python provides. Allan
On Tue, Jul 6, 2010 at 5:54 AM, Gergely Imreh <imrehg@gmail.com> wrote:
From that page: "Popular modules that don't yet support Python 3 include Twisted (for networking and a bunch of other stuff), gevent (like Twisted but different), Django and Pylons (for building websites), PyGTK and PySide (for making GUIs), py2exe (for packaging your application for Windows users), PIL (for processing images), numpy (for number crunching)..."
Thus I would mind a rebuild less, than losing my daily numpy/scipy/PyGTK...
Actually, I know for a fact that PyGTK has a patch which has been worked on for quite a while which supports Python 3, and I'm sure at least some of the others have patches in their development branches for them, too. Those which don't have any patches (or any other means available to them) would be labeled as a python2 package instead of just python, from what I understand, and thus they wouldn't be forcing anything. It probably wouldn't be too hard to modify packages that need it to use python2 instead of just python (in many cases, I'd expect it'd just require patching the crunch bang in the beginning of the code and/or a recompile specifying the python2 directory as the one to use). I'm not a developer, though, just another user, so if you're intrested in it then please consult them on this.
Arch is not in a position to force these packages to update to Python 3, and such I don't think it's a good idea to bump the default version up to 3. On the other hand, there are plenty of people who are experienced in code fixes, so maybe there would be some who would join the porting effort for those packages that they use on a regular basis. This could accelerate the transition.
Like said above, they wouldn't be forcing anyone to do anything, and in fact they could be seen as helping python programs know where fixes are needed to make the jump from Python 2 to Python 3. This, along with the fact that Python 3 probably wouldn't come to the main repos for a while (that 500+ package rebuild would probably take a while to get through, even with the whole team just focusing on that and neglecting their jobs), would mean you'd not be losing anything very fast. Smartboy
participants (12)
-
Allan McRae
-
Evangelos Foutras
-
Gergely Imreh
-
Isaac Dupree
-
Lukáš Jirkovský
-
Massimiliano Torromeo
-
Matthias Männich
-
Ng Oon-Ee
-
Smartboy
-
Stefan Husmann
-
Stephen Weinberg
-
Thomas Dziedzic