| Summary: | emerge feature request: Parallel emerge for non-dependent packages | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Xenos Pointman <pointman> |
| Component: | Core - Interface (emerge) | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED DUPLICATE | ||
| Severity: | enhancement | ||
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
I am not sure that all the locking bugs are out of the 2.X series. As well, I doubt this will speed your compiles up unless you are intentionally using a low -jN to force each package to build 1 file at a time on an large parallel system. Your set of packages with no dependencies will just build slower. In either case, I am not sure if Jason has lifted the feature request "ban" on the 2.X tree, and I'm sure if he did that he would want patches for any such request. So I will close for LATER with a request that you submit patches for your proposal. If you like you can join #gentoo-portage on irc.freenode.net or join the gentoo-portage-dev mailing list to talk about this and get extra input on it. This is the same as bug 147516. |
emerge -putvDN world lists many packages that have no dependencies. To make emerging faster, it could build many of the packages w/o dependencies in parallel, as set by -j# in make.conf. I don't believe this could be done for all packages, such as gcc, but in many cases packages are not inter-dependent, such as Samba and the kernel source download. After the packages without dependencies are built in parallel, many packages that had dependencies can now be built, as those dependencies were just built. Lather, rince, repeat. Reproducible: Always Steps to Reproduce: 1. emerge -putvDN world Actual Results: emerge world takes an eternity, even on a cluster of symmetric multi-processor systems with a terabyte of RAM. Expected Results: Works as designed, but see Details for much better design. This can conceptually be done as: emerge -vputDN world > emergeWorld.txt numberOfParallelCompilies = 0 echo "( " > parallelCommandFile for each file in emergeWorld.txt do if [ has no dependencies that need building ]; then if [ not a "special package" like gcc ]; then echo "emerge -q (file) 1> (/var/log/emerge.log/(file).txt 2>&1 & " >> parallelCommandFile numberOfParallelCompilies = numberOfParallelCompilies + 1 fi remove (file) from emergeWorld.txt fi for-loop echo " ); re-runThisFile" >> parallelCommandFile chmod u+x parallelCommandFile if [ numberOfParallelCompilies > 0 ]; then ./parallelCommandFile else emerge -quvDN world # This handles the case where the rest of the world needs to be built. fi