Summary: | parallelise install-qa-check.d/60pngfix | ||
---|---|---|---|
Product: | Portage Development | Reporter: | François Bissey <frp.bissey> |
Component: | Enhancement/Feature Requests | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | frp.bissey, pacho, polynomial-c, tsmksubc |
Priority: | High | Keywords: | InVCS |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 835380, 659322 |
Description
François Bissey
2017-09-07 23:49:09 UTC
Using `-p 0` (GNU extension to xargs for as many processes as possible) reduced the merge time by 2 and a half minutes on this 8 core machine. Following what was happening in htop it chewed 31027 png files in about 30-40s. Using find "${ED}" -type f -name '*.png' | xargs -r -n 1 -P 0 "${pngfix}" instead of the current expression seems to work nicely and cover all cases I have encountered. The "-r" means that there won't be any execution in the absence of png files. Without it, I had a nice message from pngfix about its usage when no png files were present in a package :) *** Bug 641700 has been marked as a duplicate of this bug. *** Guys, please. This is quite annoying having to wait for packages' pngfix with many png files longer than for the actual compile phase. Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/d8203c237b6466d908bf5406a948350b https://github.com/gentoo/portage/pull/345 Works fine for me and does have a similar impact to install time than the hack I was using so far. I did some measuring and posted my results here: https://github.com/gentoo/portage/pull/345#issuecomment-408038093 The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=50283f1abb77f0785ab86d41ad70d76df4e399be commit 50283f1abb77f0785ab86d41ad70d76df4e399be Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-07-25 19:43:24 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-07-28 06:31:56 +0000 install-qa-check.d/60pngfix: parallel support (bug 630292) If xargs supports the --max-procs option then use the makeopts_jobs function from helper-functions.sh to generate a --max-procs argument. Use xargs -L 1 to limit the number of png files per pngfix process, in order to ensure that enough processes are spawned, since otherwise xargs minimizes the number of processes spawned. A benchmark with flightgear-data-2018.2.1 shows that larger values of -L only decrease performance. Bug: https://bugs.gentoo.org/630292 Reviewed-by: Lars Wendler <polynomial-c@gentoo.org> Reviewed-by: Mike Gilbert <floppym@gentoo.org> bin/install-qa-check.d/60pngfix | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) |