I have an ebuild in the sage-on-gentoo overlay that produce a large amount of png files when building the documentation. pngfix is called on them and there is a distinct lag between the end of the installation and the start of the merge. Most of that time is spent in one huge pngfix command. I think it would be possible to speed things up by a small parallelisation in 60pngfix. Currently all png files are gathered and feed to pngfix by the following bit on line 28: find "${ED}" -type f -name '*.png' -exec "${pngfix}" {} + A small amount of parallelism could be inserted by using xargs instead of the `-exec` switch of find. find "${ED}" -type f -name '*.png' | xargs -n 1 -P $j "${pngfix}" where $j is the number of parallel process. One of the thing I am unclear at this stage is how to get $j from MAKEOPTS and use it at this stage of the emerge process. A more sophisticated approach would be to count the png files and adjust the argument to "-n", so only $j instances of pngfix are started, instead of one for each file.
Using `-p 0` (GNU extension to xargs for as many processes as possible) reduced the merge time by 2 and a half minutes on this 8 core machine. Following what was happening in htop it chewed 31027 png files in about 30-40s.
Using find "${ED}" -type f -name '*.png' | xargs -r -n 1 -P 0 "${pngfix}" instead of the current expression seems to work nicely and cover all cases I have encountered. The "-r" means that there won't be any execution in the absence of png files. Without it, I had a nice message from pngfix about its usage when no png files were present in a package :)
*** Bug 641700 has been marked as a duplicate of this bug. ***
Guys, please. This is quite annoying having to wait for packages' pngfix with many png files longer than for the actual compile phase.
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/d8203c237b6466d908bf5406a948350b https://github.com/gentoo/portage/pull/345
Works fine for me and does have a similar impact to install time than the hack I was using so far.
I did some measuring and posted my results here: https://github.com/gentoo/portage/pull/345#issuecomment-408038093
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=50283f1abb77f0785ab86d41ad70d76df4e399be commit 50283f1abb77f0785ab86d41ad70d76df4e399be Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-07-25 19:43:24 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-07-28 06:31:56 +0000 install-qa-check.d/60pngfix: parallel support (bug 630292) If xargs supports the --max-procs option then use the makeopts_jobs function from helper-functions.sh to generate a --max-procs argument. Use xargs -L 1 to limit the number of png files per pngfix process, in order to ensure that enough processes are spawned, since otherwise xargs minimizes the number of processes spawned. A benchmark with flightgear-data-2018.2.1 shows that larger values of -L only decrease performance. Bug: https://bugs.gentoo.org/630292 Reviewed-by: Lars Wendler <polynomial-c@gentoo.org> Reviewed-by: Mike Gilbert <floppym@gentoo.org> bin/install-qa-check.d/60pngfix | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
