Bug 482290 – portage-2.2.1: /usr/lib/portage/bin/install.py in painfully slow (FEATURES=xattr)

• Product: Portage Development
• Status: RESOLVED DUPLICATE

[Description Andrew Savchenko 2013-08-24 02:57:46 UTC]

Hello,

when it comes to installation of packages with thousands of files (e.g. html api docs or game data) recent portage versions became unacceptably slow. Just few examples:

freedroidrpg: Mon Oct  1 16:40:11 2012: 5 minutes, 13 seconds
freedroidrpg: Sat Aug 24 04:01:44 2013: 1 hour, 4 minutes, 9 seconds (python-2.7)
freedroidrpg: Sat Aug 24 05:28:03 2013: 42 minutes, 5 seconds (python-3.3)

boost: Wed Oct  3 14:53:37 2012: 33 minutes, 55 seconds
boost: Thu Aug 22 21:16:50 2013: 4 hours, 1 minute, 28 seconds (python-3.3)

Host system is EeePC 1000H with Atom N270 CPU, 2 GB RAM, distcc and ccache enabled.

I found that compilation itself is fast: it takes ~4 minutes for freedroidrpg and ~25 minutes for boost. What takes an unacceptably long time is just installation of files: for each(!) file to be installed /usr/lib/portage/bin/install.py is called like:

/usr/bin/python3.3 /usr/lib/portage/bin/install.py -m0644 libs/spirit/doc/html/spirit/karma/quick_reference/primitive_generators/auxiliary.html /va
r/tmp/portage/dev-libs/boost-1.53.0/image//usr/share/doc/boost-1.53.0/html/libs/spirit/doc/html/spirit/karma/quick_reference/primitive_generators

Both packages have a large number of files installed:

andrew@hitomi ~ $ qlist freedroidrpg | wc -l
7770
andrew@hitomi ~ $ qlist boost | wc -l
29721

As you can see this is install rate about 2 files per second! This is absolutely unacceptable!!

And this is just few examples, virtually almost every package installs longer and those containing thousands of files are installed hundreds times slower than they should and were in the past.

System load shows that disk I/O is not an issue: it keeps below 5%. What it is an issue is 100% CPU load. Thus I need more CPU to install files than to compile the package!! This is absolutely ridiculous.

I tried with python-3.3 and python-2.7, the latter is faster by 1/3 as you can see but overall installation time is still not acceptable.

P.S. I looked in the code of install.py, dohtml and xattr/install helpers. I am shocked. Why to use all that code garbage instead of plain find -iregex ... -execdir .. \+ ? Permissions may be fixed later with chmod, links/xattrs may be preserved via cp -a, if some files/dirs must be processed specially, find can use exclusion/inclusion lists...

[Comment 1 Andrew Savchenko 2013-08-24 02:59:17 UTC]

Created attachment 356822 [details]
emerge --info

[Comment 2 Zac Medico 2013-08-24 03:21:00 UTC]

You might consider removing xattr from FEATURES, in order to bypass the install wrapper.

(In reply to Andrew Savchenko from comment #0)
> P.S. I looked in the code of install.py, dohtml and xattr/install helpers. I
> am shocked. Why to use all that code garbage instead of plain find -iregex
> ... -execdir .. \+ ? Permissions may be fixed later with chmod, links/xattrs
> may be preserved via cp -a, if some files/dirs must be processed specially,
> find can use exclusion/inclusion lists...

For internal helpers (like dohtml), it would be possible to use alternatives to the install command, but the install wrapper would still be useful for things like ebuild makefiles that call the install command directly.

[Comment 3 Andrew Savchenko 2013-08-24 03:53:31 UTC]

(In reply to Zac Medico from comment #2)
> You might consider removing xattr from FEATURES, in order to bypass the
> install wrapper.

Thanks a lot! This saved my day.

[Comment 4 Zac Medico 2013-08-24 18:24:49 UTC]

Optimize dohtml:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=3f67cb453ee1b74894c230459b43855db7fcc410

[Comment 5 Zac Medico 2013-10-24 01:17:48 UTC]

*** Bug 488368 has been marked as a duplicate of this bug. ***

[Comment 6 Martin von Gagern 2013-11-12 08:33:15 UTC]

*** Bug 490170 has been marked as a duplicate of this bug. ***

[Comment 7 Martin von Gagern 2013-11-12 08:43:18 UTC]

FEATURES=-xattr sounds like a suitable workaround, but in the long run a proper solution would be preferred. One thing that might help would be byte-compiling that python file. Another idea might be using bash instead of python, despite what SpanKY wrote in bug #465000 comment #11. A third might be a pre-compiled binary. I guess portage itself doesn't compile anything, does it? Nevertheless, it might depend on a patched version of install, which comes as a separate package and gets installed to some custom directory outside PATH.

[Comment 8 Anthony Basile 2014-01-18 14:44:01 UTC]

Work on optimizing is being discussed in bug #465000.  We're almost done and testing will be appreciated :)

*** This bug has been marked as a duplicate of bug 465000 ***

[Comment 9 Zac Medico 2014-10-07 19:34:56 UTC]

*** Bug 524706 has been marked as a duplicate of this bug. ***