[arch-dev-public] Plan for python upgrade
Hi everybody, Python-2.6 is out and Python-3.0 is scheduled for release in a couple of weeks. People have been asking how I am going to handle the update so I thought I'd post a summary here. There are packages that need rebuilt not matter how I handle the upgrade. I know of 39 packages in [extra] which link to libpython2.5.so. Also 8 packages add their path to the python system path and it is easier to rebuild them than hack some backwards compatibility. My current plan is that the python package will be python3.0 and add a python2 package for python2.6. This bring in a couple of issues. Everything that (make)depends on python will need changed to python2 (48 packages). More difficult is that package with "/usr/bin/python" or "/usr/bin/env python" in their scripts need this fixed (addition 30 rebuilds and effects many of the above rebuilds). Some of these packages are nice and just require a rebuild and their scripts point in the right place. Others require a bit of sed magic. This plan requires a total of 125(!) rebuilds but most of the are fairly simple to transition to python2. Of the 33 packages on my system, the only package I have had trouble rebuilding so far is subversion as autoreconf calls something that calls python but I have not looked into it too hard or even checked if it as actually a problem... The backup plan is to keep the python as the python-2 series and add a python3 package. This means less rebuilds at the moment but this will make transitioning packages to python3 harder (requiring the sed magic for scripts to point at python3 instead of python). Also, I personally think this is the less clean approach long term. But, if people feel strongly that this is the best route to go, I may be convinced (it is a lot less work at my end). Both the gnome and boost rebuilds need to be moved from [testing] before I start this due to overlapping packages with the python rebuilds. Cheers, Allan
Allan McRae wrote:
Hi everybody,
Python-2.6 is out and Python-3.0 is scheduled for release in a couple of weeks. People have been asking how I am going to handle the update so I thought I'd post a summary here.
<snip>
According to the number of emails in my inbox (which has reached well into double figures), my plan for the python 2.6 and 3.0 upgrade is rather controversial. So, I thought that I would add some more justification here and point out that if people want to comment on the strategy they can do so by replying to public mailing lists because it is getting repetitive I will no longer be replying individually to people. Here are the possible upgrade strategies. Plan 1: python = python3.0 python2 = python2.6 Plan 2: python = python2.6 python3 = python3.0 These are the advantages/disadvantages of these plans as far as I can see. Many programs point at "/usr/bin/python" or "/usr/bin/env python". For Plan 1, the /usr/bin/python binary is python3.0 so that involves rebuilding packages and changing them to point at "python2". For some, this is as simple as rebuilding. Others require a simple sed statement. Also, with Plan 1, anything with a python depends/optdepends will need rebuilt. So, Plan 1 involves *a lot* more rebuilding that Plan 2 (around 125 packages vs 47 packages in [extra]). Plan 2 will also require adjusting programs built against python3 to point at /usr/bin/python3 in their scripts and changing the dependency to python3 so this is a now versus later issue. Also, with Plan 2, at some later stage we will want the /usr/bin/python binary to be python-3.x and probably to keep a legacy python-2.x package. That tells me, that Plan 2 requires moving to Plan 1 at a later time. So in summary: Plan 1 - Big rebuild now. People complain "Allan broke it"... Plan 2 - Medium rebuild now, drawn out transition, followed by medium/big rebuild later. People complain "Allan broke it" long term... I am going to go for Plan 1 unless someone convinces me the Plan 2 is a better plan _long_ term. I prefer short term pain for long term benefit rather than long term pain for short term benefit. Cheers, Allan
On Sat, 4 Oct 2008, Allan McRae wrote:
Allan McRae wrote:
Hi everybody,
Python-2.6 is out and Python-3.0 is scheduled for release in a couple of weeks. People have been asking how I am going to handle the update so I thought I'd post a summary here.
<snip>
According to the number of emails in my inbox (which has reached well into double figures), my plan for the python 2.6 and 3.0 upgrade is rather controversial. So, I thought that I would add some more justification here and point out that if people want to comment on the strategy they can do so by replying to public mailing lists because it is getting repetitive I will no longer be replying individually to people.
Here are the possible upgrade strategies.
Plan 1: python = python3.0 python2 = python2.6
Plan 2: python = python2.6 python3 = python3.0 These are the advantages/disadvantages of these plans as far as I can see. Many programs point at "/usr/bin/python" or "/usr/bin/env python". For Plan 1, the /usr/bin/python binary is python3.0 so that involves rebuilding packages and changing them to point at "python2". For some, this is as simple as rebuilding. Others require a simple sed statement. Also, with Plan 1, anything with a python depends/optdepends will need rebuilt. So, Plan 1 involves *a lot* more rebuilding that Plan 2 (around 125 packages vs 47 packages in [extra]).
Plan 2 will also require adjusting programs built against python3 to point at /usr/bin/python3 in their scripts and changing the dependency to python3 so this is a now versus later issue. Also, with Plan 2, at some later stage we will want the /usr/bin/python binary to be python-3.x and probably to keep a legacy python-2.x package. That tells me, that Plan 2 requires moving to Plan 1 at a later time.
So in summary: Plan 1 - Big rebuild now. People complain "Allan broke it"... Plan 2 - Medium rebuild now, drawn out transition, followed by medium/big rebuild later. People complain "Allan broke it" long term...
I am going to go for Plan 1 unless someone convinces me the Plan 2 is a better plan _long_ term. I prefer short term pain for long term benefit rather than long term pain for short term benefit.
Cheers, Allan
IMO, Plan 1 is the way to go as this is how we managed the qt3/qt upgrade. Does anyone has an idea on how many packages in the repo currently works with or need python3.0? If that number is zero or very small, we could update python to 2.6 and wait until that number increases before proceeding with Plan 1. That will reduce the number of rebuilds because we will be replacing rebuilds by package update that eventually would need to be done anyway. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Eric Belanger escreveu:
On Sat, 4 Oct 2008, Allan McRae wrote:
Allan McRae wrote:
Hi everybody,
Python-2.6 is out and Python-3.0 is scheduled for release in a couple of weeks. People have been asking how I am going to handle the update so I thought I'd post a summary here.
<snip>
According to the number of emails in my inbox (which has reached well into double figures), my plan for the python 2.6 and 3.0 upgrade is rather controversial. So, I thought that I would add some more justification here and point out that if people want to comment on the strategy they can do so by replying to public mailing lists because it is getting repetitive I will no longer be replying individually to people.
Here are the possible upgrade strategies.
Plan 1: python = python3.0 python2 = python2.6
Plan 2: python = python2.6 python3 = python3.0 These are the advantages/disadvantages of these plans as far as I can see. Many programs point at "/usr/bin/python" or "/usr/bin/env python". For Plan 1, the /usr/bin/python binary is python3.0 so that involves rebuilding packages and changing them to point at "python2". For some, this is as simple as rebuilding. Others require a simple sed statement. Also, with Plan 1, anything with a python depends/optdepends will need rebuilt. So, Plan 1 involves *a lot* more rebuilding that Plan 2 (around 125 packages vs 47 packages in [extra]).
Plan 2 will also require adjusting programs built against python3 to point at /usr/bin/python3 in their scripts and changing the dependency to python3 so this is a now versus later issue. Also, with Plan 2, at some later stage we will want the /usr/bin/python binary to be python-3.x and probably to keep a legacy python-2.x package. That tells me, that Plan 2 requires moving to Plan 1 at a later time.
So in summary: Plan 1 - Big rebuild now. People complain "Allan broke it"... Plan 2 - Medium rebuild now, drawn out transition, followed by medium/big rebuild later. People complain "Allan broke it" long term...
I am going to go for Plan 1 unless someone convinces me the Plan 2 is a better plan _long_ term. I prefer short term pain for long term benefit rather than long term pain for short term benefit.
Cheers, Allan
IMO, Plan 1 is the way to go as this is how we managed the qt3/qt upgrade.
Does anyone has an idea on how many packages in the repo currently works with or need python3.0? If that number is zero or very small, we could update python to 2.6 and wait until that number increases before proceeding with Plan 1. That will reduce the number of rebuilds because we will be replacing rebuilds by package update that eventually would need to be done anyway.
As Allan know, im with you Eric. I guess it will be safe to users too. -- Douglas Soares de Andrade -- Python, Django, Zope e Plone == Archlinux Trusted User and Developer - dsa ** Quote: Old programmers never die; they exit to a higher shell.
On Fri, Oct 3, 2008 at 10:54 PM, Eric Belanger <belanger@astro.umontreal.ca> wrote:
IMO, Plan 1 is the way to go as this is how we managed the qt3/qt upgrade.
Does anyone has an idea on how many packages in the repo currently works with or need python3.0? If that number is zero or very small, we could update python to 2.6 and wait until that number increases before proceeding with Plan 1. That will reduce the number of rebuilds because we will be replacing rebuilds by package update that eventually would need to be done anyway.
Agreed, plan 1. Though, to make it easier, we could start it now. I suggest we add a python2 symlink to whatever the current python2.x version is, so that we can start adding the symlinks whenever we update a python package and now not have to wait. Also... has upstream mention any suggestions for migration?
On Sat, Oct 4, 2008 at 12:54 AM, Eric Belanger <belanger@astro.umontreal.ca> wrote:
On Sat, 4 Oct 2008, Allan McRae wrote:
Allan McRae wrote:
Hi everybody,
Python-2.6 is out and Python-3.0 is scheduled for release in a couple of weeks. People have been asking how I am going to handle the update so I thought I'd post a summary here.
<snip>
According to the number of emails in my inbox (which has reached well into double figures), my plan for the python 2.6 and 3.0 upgrade is rather controversial. So, I thought that I would add some more justification here and point out that if people want to comment on the strategy they can do so by replying to public mailing lists because it is getting repetitive I will no longer be replying individually to people.
Here are the possible upgrade strategies.
Plan 1: python = python3.0 python2 = python2.6
Plan 2: python = python2.6 python3 = python3.0 These are the advantages/disadvantages of these plans as far as I can see. Many programs point at "/usr/bin/python" or "/usr/bin/env python". For Plan 1, the /usr/bin/python binary is python3.0 so that involves rebuilding packages and changing them to point at "python2". For some, this is as simple as rebuilding. Others require a simple sed statement. Also, with Plan 1, anything with a python depends/optdepends will need rebuilt. So, Plan 1 involves *a lot* more rebuilding that Plan 2 (around 125 packages vs 47 packages in [extra]).
Plan 2 will also require adjusting programs built against python3 to point at /usr/bin/python3 in their scripts and changing the dependency to python3 so this is a now versus later issue. Also, with Plan 2, at some later stage we will want the /usr/bin/python binary to be python-3.x and probably to keep a legacy python-2.x package. That tells me, that Plan 2 requires moving to Plan 1 at a later time.
So in summary: Plan 1 - Big rebuild now. People complain "Allan broke it"... Plan 2 - Medium rebuild now, drawn out transition, followed by medium/big rebuild later. People complain "Allan broke it" long term...
I am going to go for Plan 1 unless someone convinces me the Plan 2 is a better plan _long_ term. I prefer short term pain for long term benefit rather than long term pain for short term benefit.
Cheers, Allan
IMO, Plan 1 is the way to go as this is how we managed the qt3/qt upgrade.
Does anyone has an idea on how many packages in the repo currently works with or need python3.0? If that number is zero or very small, we could update python to 2.6 and wait until that number increases before proceeding with Plan 1. That will reduce the number of rebuilds because we will be replacing rebuilds by package update that eventually would need to be done anyway.
I agree with plan 1, but perhaps we should do it as eric suggests. Upgrade "python" to 2.6, do the rebuilds, and push things out of testing. THEN we can do the python/python2 split up in testing.
IMO, Plan 1 is the way to go as this is how we managed the qt3/qt upgrade.
It's probably just as easy to find packages that went the opposite way: sqlite3 vs. sqlite2, xfce3 vs. xfce4. Also, when we did upgrade to qt3 it was after making sure most things worked with it. I don't know how long we waited to make qt3 the "official" qt package.
Does anyone has an idea on how many packages in the repo currently works with or need python3.0? If that number is zero or very small, we could update python to 2.6 and wait until that number increases before proceeding with Plan 1. That will reduce the number of rebuilds because we will be replacing rebuilds by package update that eventually would need to be done anyway.
Python 3.0 is a mostly backwards incompatible update to python. I suspect most packages will need tweaks and updates to make them work at all. The python 3.0 upgrade was never meant to be easy. They wanted to fix some of the warts in python. I'm pretty sure there will be a good number of packages that aren't python 3.0 compatible. That being said, they will be in the longer run. I really think that Python 3.0 isn't going to see mainstream support for a while. The upgrade path that the python community suggests is getting it working in python 2.6 and then fix it from there. What will you call the binaries? python and python2? I'm going with Aaron's suggestion. Don't worry about python 3.0 until more things support it. Or create a temporary python3 repo to support software in there (and call the package python3) for people who really want it. Then the python/python2 split will be gentler. Jason
On Mon, Oct 6, 2008 at 12:49 AM, Jason Chu <jason@archlinux.org> wrote:
I'm going with Aaron's suggestion. Don't worry about python 3.0 until more things support it. Or create a temporary python3 repo to support software in there (and call the package python3) for people who really want it. Then the python/python2 split will be gentler.
Yeah, instead of using 'extrapkg', just use 'archrelease python3-i686' or something, and manually scp the packages to gerolde (or maybe patch devtools to do this for you). Then, on gerolde, you can do "/arch/db-update i686 python3", though that'd require someone with superpowers manually making a top-level dir. I think this route would be best - just tell users how to get the python3 repo, and bam. All the tools are made to do exactly this, and considering the python3 upgrade path will take weeks, not days, it makes perfect sense to use a side repo for this until we're at a level of support where we want to move it into testing/extra.
Aaron Griffin wrote:
On Mon, Oct 6, 2008 at 12:49 AM, Jason Chu <jason@archlinux.org> wrote:
I'm going with Aaron's suggestion. Don't worry about python 3.0 until more things support it. Or create a temporary python3 repo to support software in there (and call the package python3) for people who really want it. Then the python/python2 split will be gentler.
Yeah, instead of using 'extrapkg', just use 'archrelease python3-i686' or something, and manually scp the packages to gerolde (or maybe patch devtools to do this for you). Then, on gerolde, you can do "/arch/db-update i686 python3", though that'd require someone with superpowers manually making a top-level dir.
I think this route would be best - just tell users how to get the python3 repo, and bam. All the tools are made to do exactly this, and considering the python3 upgrade path will take weeks, not days, it makes perfect sense to use a side repo for this until we're at a level of support where we want to move it into testing/extra.
Sounds a good plan. What I am going to do now is: 1) Transition to python-2.6. I have tried most packages that need rebuilt already and it seems to be a fairly easy transition. Will do this as soon as gnome and boost rebuilds are out of [testing]. 2) Add a python3 package to [community] so people will not complain about it not being in the repos but building against it will be discouraged (as deps in the [extra] repo will still be built against python2). 3) As devs find their packages can be built with python3, they should add them to this wiki page: http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List (python3 section at the bottom) 4) When enough (to be defined...) packages are in that list, create a [python] repo for the transition of python to python-3.x and make a python2 package. Allan
3) As devs find their packages can be built with python3, they should add them to this wiki page: http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List (python3 section at the bottom)
I'm curious if there is any reason not to use the developer to-do lists for this, and if there is anything I should do to improve them.
Dusty Phillips wrote:
3) As devs find their packages can be built with python3, they should add them to this wiki page: http://wiki.archlinux.org/index.php/DeveloperWiki:Python_Todo_List (python3 section at the bottom)
I'm curious if there is any reason not to use the developer to-do lists for this, and if there is anything I should do to improve them.
I used the DeveloperWiki for the initial rebuild information. It also allows me to annotate the rebuild process (e.g. http://wiki.archlinux.org/index.php/DeveloperWiki:Ncurses_Todo_List which includes build order information). I use the dev todo lists for keeping track of additional rebuilds that were missed during the first round, which I think they are better suited to. Allan
participants (7)
-
Aaron Griffin
-
Allan McRae
-
Douglas Soares de Andrade
-
Dusty Phillips
-
Eric Belanger
-
James Rayner
-
Jason Chu