Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 630292 - parallelise install-qa-check.d/60pngfix
Summary: parallelise install-qa-check.d/60pngfix
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
Keywords: InVCS
: 641700 (view as bug list)
Depends on:
Blocks: 835380 659322
  Show dependency tree
Reported: 2017-09-07 23:49 UTC by François Bissey
Modified: 2023-12-24 17:52 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description François Bissey 2017-09-07 23:49:09 UTC
I have an ebuild in the sage-on-gentoo overlay that produce a large amount of png files when building the documentation. pngfix is called on them and there is a distinct lag between the end of the installation and the start of the merge. Most of that time is spent in one huge pngfix command.

I think it would be possible to speed things up by a small parallelisation in 60pngfix. Currently all png files are gathered and feed to pngfix by the following bit on line 28:
find "${ED}" -type f -name '*.png' -exec "${pngfix}" {} +

A small amount of parallelism could be inserted by using xargs instead of the `-exec` switch of find.
find "${ED}" -type f -name '*.png' | xargs -n 1 -P $j "${pngfix}"

where $j is the number of parallel process. One of the thing I am unclear at this stage is how to get $j from MAKEOPTS and use it at this stage of the emerge process.

A more sophisticated approach would be to count the png files and adjust the argument to "-n", so only $j instances of pngfix are started, instead of one for each file.
Comment 1 François Bissey 2017-09-08 03:24:35 UTC
Using `-p 0` (GNU extension to xargs for as many processes as possible) reduced the merge time by 2 and a half minutes on this 8 core machine. Following what was happening in htop it chewed 31027 png files in about 30-40s.
Comment 2 François Bissey 2017-09-10 22:03:15 UTC
find "${ED}" -type f -name '*.png' | xargs -r -n 1 -P 0 "${pngfix}"
instead of the current expression seems to work nicely and cover all cases I have encountered. The "-r" means that there won't be any execution in the absence of png files. Without it, I had a nice message from pngfix about its usage when no png files were present in a package :)
Comment 3 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2017-12-19 11:13:50 UTC
*** Bug 641700 has been marked as a duplicate of this bug. ***
Comment 4 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2018-07-25 10:22:10 UTC
Guys, please. This is quite annoying having to wait for packages' pngfix with many png files longer than for the actual compile phase.
Comment 6 François Bissey 2018-07-26 02:41:58 UTC
Works fine for me and does have a similar impact to install time than the hack I was using so far.
Comment 7 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2018-07-26 09:36:48 UTC
I did some measuring and posted my results here:
Comment 8 Larry the Git Cow gentoo-dev 2018-07-28 06:41:25 UTC
The bug has been referenced in the following commit(s):

commit 50283f1abb77f0785ab86d41ad70d76df4e399be
Author:     Zac Medico <>
AuthorDate: 2018-07-25 19:43:24 +0000
Commit:     Zac Medico <>
CommitDate: 2018-07-28 06:31:56 +0000

    install-qa-check.d/60pngfix: parallel support (bug 630292)
    If xargs supports the --max-procs option then use the makeopts_jobs
    function from to generate a --max-procs argument.
    Use xargs -L 1 to limit the number of png files per pngfix process,
    in order to ensure that enough processes are spawned, since otherwise
    xargs minimizes the number of processes spawned. A benchmark with
    flightgear-data-2018.2.1 shows that larger values of -L only decrease
    Reviewed-by: Lars Wendler <>
    Reviewed-by: Mike Gilbert <>

 bin/install-qa-check.d/60pngfix | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)