new stable version of python has been released recently (namely 2.3) which contains many new features and is significantly faster due to code cleanups. Reproducible: Always Steps to Reproduce: 1. 2. 3.
quicker python == quicker portage :D
*** Bug 22037 has been marked as a duplicate of this bug. ***
if it was only so true .. python2.3 startup times are slower than python2.2 .. % time python2.2 -c 'print 1' 1 python2.2 -c 'print 1' 0.07s user 0.03s system 92% cpu 0.108 total % time python2.3 -c 'print 1' 1 python2.3 -c 'print 1' 0.43s user 0.01s system 103% cpu 0.427 total we've got a couple of issues before we can start upgrading. the pain is similar to perl 5.6 to 5.8 upgrade. outstanding issues off the top of my head: 1. need to re-emerge everything that depends on python-2.2 2. modify portage to NOT use python2.2 3. figuring out whether we should slot python2.2 and python2.3 4. decide if we should follow the devs recommendations on using ucs4 for unicode which redhat are using. there's a thread about this on python-dev about the pitfalls of using ucs2 or ucs4 (more memory, but more standards compliant for cjk) we can possibly make this a cjk option.
Well - it's indeed not as fast as python.org claims, but on my old pentium 233mmx it is NOT slower. Actually there was no significant difference during my short tests (including print 1 ;)
Python 2.3 doesn't start significantly slower than Python 2.2. All you've shown is Python 2.2's modules were in your VFS cache whereas Python 2.3's weren't. Run the test multiple times and you'll find that at hundredth-of-a-second resolution they both equally as fast. python2.3 -c 'print 1' 0.04s user 0.01s system 151% cpu 0.033 total functor% time python2.2 -c 'print 1' 1 python2.2 -c 'print 1' 0.04s user 0.00s system 121% cpu 0.033 total functor% time python2.3 -c 'print 1' 1 python2.3 -c 'print 1' 0.03s user 0.01s system 89% cpu 0.045 total functor% time python2.2 -c 'print 1' 1 python2.2 -c 'print 1' 0.04s user 0.00s system 128% cpu 0.031 total Besides, startup time variations on the scale of hundredths of a second are *highly* unlikely to be a bottleneck in portage.
just for the record, it is not a module caching issue, i've run these tests numerous times to make sure it isn't a disk cache issue or anything like that. i just snipped them for brevity. even the python devs acknowledge that the startup times have skyrocketed because they now import re and other modules in addition to what they used to use in python2.2. don't have a exact link, but if you search of "startup speed" on python-dev mailing list you'll know what i mean. anyway, this isn't the point of this bug. the main points i've already outlined, if someone wants to help out with those then please do. the speed of python is not the focus of this bug.
point 1: possible to make a split libpython like libperl ? all else fail we could force them to run revdep-ebuild point 2: not a problem, figure out how we're going to tackle #1 and then we'll figure out how to best to change it point 3: we slot 2.1 and 2.2, why not 2.3 point 4: why not tie it to nls ? or does it really only benefit cjk users ? nls users already 'bloat' their system, wont be anything new to em :)
point #1: python luckily doesn't need to have a libpython, because all modules statically link to libpython2.2.a, the libpython*.so is new in 2.3. but we do need to revdep-rebuild or use the script that perl does to reinstall all the old modules in /usr/lib/python2.3 and also link against the new libpython2.3.a. i'm try to make a stage that uses python-2.3 by default and make sure all the existing python modules we have don't have 2.2 hardcoded (like portage). point #3: the reason (as far as i know) for slotting python-2.1 and python-2.2 was because of zope's requirement for <=python-2.1. the problem here is if we slot them, then we need some logic to update the symlink from /usr/bin/python to /usr/bin/python2.x to point to the latest one, and make sure one doesn't clean another. i'd rather only one package made that symlink if they were slotted. although practically they can be slotted, i'm just not sure whether it is a good idea logically. point #4: yes, that would be a good idea to tie it into nls/cjk use flags. basic unicode support should be compiled in regardless of whether or not you have nls. with that said, i need to experiment with a -nls environment to see what happens to unicode. i also need to look at how other distros are doing it like debian and mandrake etc. i've heard problems with tcl/tk dealing with python's ucs4 implementation.
why not simply using utf8 for unicode? its space efficient and ANSI compatible.
the ucs2 ucs4 stuff is only for the internal representation of the unicode characters. you still interface with it using utf8 (at least under linux)
Created attachment 18639 [details] python-2.3.2_rc1.ebuild close to the final python-2.3 ebuild that will be in portage. it's missing a warning about running "python-updater" after updating to python-2.3 and just needs some more testing. be prepared for things to break though! this has only been tested by me so far. also note that this is slotted so you can have python2.2 and python2.3 on the same machine.
Created attachment 18640 [details, diff] python-2.3-add_portage_search_path.patch required for python-2.3.2_rc1 and the proposed portage ebuild that supports python2.3. moving all portage stuff to /usr/lib/portage/pym to ease upgrading pains when updating python minor versions in the future. this patch adds /usr/lib/portage/pym to the default search paths.
Created attachment 18641 [details] python-updater a script that tries to rebuild everything that has files in /usr/lib/python2.2. this is pretty similar to libperl_rebuilder except very customised to do just one specific thing. it produces a log file in /tmp/python-updater.log, i probably should document that somewhere in the source.
Created attachment 18643 [details] python-config-2.3 python-config script updated for python2.3 well, these aren't in portage yet because sys-apps/portage doesn't work with python2.3 and i don't want to risk anything by putting it in there yet. these are available for anyone to test and spot bugs before they get committed (which should be pretty soon, hopefully.)
I'm trying this now (with 2.3.2 final -- btw, there is a bz2 download you might want to use in the ebuild). python is fine. python-updater just failed with omniorb 4.0 -- access violations trying to write/unlink pyc files in /usr/lib/python-2.3.
you definately need the newest portage thats out, or the one in bug #28342. omniORB/omniORBpy stuff needs to be sorted out as well, afaik, they're currently not actively maintained. there's even a new version out that we haven't upgraded to :/
With portage 2.0.49-r7 I still get sandbox violations for every attempted emerge in python-updater -- always involving .pyc files in /usr/lib/python2.3. Also, it looks to me like every time I install a previously installed module for 2.3, the same module will be removed from 2.2, which may be inevitable, but seems problematic, since 2.3 is enriched at 2.2's expense, yet the slotting of python would suggest that they should both be able to work. Perhaps it is necessary to bite the bullet and do what debian does -- have separate packages for every python package/python version (e.g., egenix-21, egenix-22, egenix-23). Otherwise, portage can't really robustly support multiple parallel python installs, can it?
yes, the upgrade will move python modules to 2.3. the reason is that 2.3 will be the new default python version and we need to have all the modules working just like they were before. we could, for compatibility reasons, add /usr/lib/python2.3/site-packages to python2.2's search path so it finds the new modules, but gives warnings. however, i don't think that is a good strategy and i'd rather avoid it. the reason is the API has slightly changed from 2.2 to 2.3, and it may produce some undesirable effects when using python2.2. the reason why it is slotted 2.3 is because we need python2.2 installed between the python-2.3 upgrade and the portage upgrade that supports python-2.3. between then, if we replace python outright, then portage will stop working. that is pretty much the only reason why i have to keep 2.3 slotted. in bug 28342 we've decided to put the portage packages in /usr/lib/portage/pym to get around this problem for future versions of python. giving packages a -22 or -23 suffix is definitely not desirable at the moment. i'm not a fan of the -py21 suffixes we have in dev-python just to support zope either.
python 2.3.2 is in portage now. it is currently masked for testing.
after i emerged this and ran python-updater, i believe python-updater threw a "portage module not found" error. this didn't seemt o hurt it, but i aborted anyway and emerged the latest portage to add the module in to the 2.3 site-packages.
When I try to run python-updater from portage, it dies with the following when replacing the first package: --------------------------- ACCESS VIOLATION SUMMARY ---------------------------LOG FILE = "/tmp/sandbox-pysqlite-0.4.3-17987.log" unlink: /usr/lib/python2.3/distutils/command/build_ext.pyc open_wr: /usr/lib/python2.3/distutils/command/build_ext.pyc unlink: /usr/lib/python2.3/distutils/ccompiler.pyc open_wr: /usr/lib/python2.3/distutils/ccompiler.pyc unlink: /usr/lib/python2.3/distutils/unixccompiler.pyc open_wr: /usr/lib/python2.3/distutils/unixccompiler.pyc --------------------------------------------------------------------------------
Are you using sys-apps/porsys-apps/portage-2.0.49-r12 or portage-2.0.49-r14 and python-2.3.2-r1? Try setting PYTHON_DONTCOMPILE=1 in your environment. See: http://bugs.gentoo.org/show_bug.cgi?id=30725
re-assigning. probably we can unmask this for ~x86 soon. i'm going to send out a request to devs to test it out.
I hope this can move forward -- I'm going live with python 2.3.2 on several production servers in a couple of weeks! For me it was necessary to upgrade portage to r16 before attempting to upgrade python (although in the forums the opposite order is recommended); and python-updater doesn't know that some packages aren't 2.3 compatible (csv shouldn't be installed, because it will conflict with a python library module; pypgsql 2.3 won't compile, you need to use 2.4; egenix-mx-base 2.0.3 will segfault with python 2.3 at runtime). Also, I notice that python-config doesn't seem to get installed; I don't know if that matters or not. Everything else is smooth sailing so far.
grr, i hate how bugzilla only occasionally wraps bugzilla comment texts :/ thanks for the list of ebuilds, i've created a seperate bug for a comprehensive set of these (bug #33372). yes, and also python-config wasn't installed, thanks for spotting that, should be fixed in the next couple of minutes.
well, python-2.3.2 is now in ~x86. hurray .. emm, i'm bracing myself for a flood of bugs ..
