We have the Gentoo profile plus our profile to include the packages in the @profile. The packages are installed in the binary way from a binary host. sys-apps/portage-2.2.26 layout.conf masters = gentoo repo-name = nucleus profile-formats = portage-2 profile-set build-id Installing multiple packages with emerge --jobs don't have speed improvement. cat /proc/cpuinfo | grep 'model name' model name : Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz hdparm -tT /dev/sda /dev/sda: Timing cached reads: 14932 MB in 2.00 seconds = 7469.99 MB/sec Timing buffered disk reads: 478 MB in 3.02 seconds = 158.24 MB/sec We are upgrading about 410 binary packages: Total: 410 packages (275 upgrades, 1 downgrade, 68 new, 13 in new slots, 53 reinstalls, 410 binaries, 2 uninstalls), Size of downloads: 563,789 KiB For the tests we pre-downloaded the packages with emerge -f . Test1 - 1 cpu and 512GB of ram: time emerge --jobs --load-average 4 --with-bdeps=y -vtuND world real 47m5.936s user 30m52.380s sys 6m4.630s Test2 - 4 cpu and 2GB of ram: time emerge --jobs -vtuND --with-bdeps=y world real 42m45.309s user 32m50.050s sys 9m32.210s Test3 - 1 cpu and 512MB of ram NO --JOBS time emerge -vtuND --with-bdeps=y world real 41m6.100s user 26m48.230s sys 4m58.730s It is very strange that emerge is quicker without the --jobs. I'm missing something to speed up the update procedure ? Thanks Reproducible: Always Expected Results: Speed up the update procedure with binary packages.
You can maximize performance by setting FEATURES="parallel-install -ebuild-locks" but beware that -ebuild-locks makes it possible for package ebuild phases to interfere with each other in subtle ways.
(In reply to Zac Medico from comment #1) > You can maximize performance by setting FEATURES="parallel-install > -ebuild-locks" but beware that -ebuild-locks makes it possible for package > ebuild phases to interfere with each other in subtle ways. I test with parallel-install and without "-ebuild-locks" to remain more conservative i think ( right ? ) . time FEATURES="parallel-install" emerge -avtuND --jobs --with-bdeps=y world I try with FEATURES="parallel-install" only, but during the install process happended this error. >>> Emerging binary (278 of 410) net-dns/bind-tools-9.10.3_p2::gentoo >>> Emerging binary (279 of 410) net-dns/bind-9.10.3_p2::gentoo >>> Emerging binary (280 of 410) app-shells/bash-completion-2.1_p20141224-r1::gentoo >>> Emerging binary (281 of 410) net-misc/iputils-20121221-r1::gentoo ..... >>> Installing (269 of 410) dev-perl/Geo-IP-1.450.0::gentoo >>> Installing (280 of 410) app-shells/bash-completion-2.1_p20141224-r1::gentoo >>> Installing (279 of 410) net-dns/bind-9.10.3_p2::gentoo >>> Installing (278 of 410) net-dns/bind-tools-9.10.3_p2::gentoo ..... >>> Jobs: 284 of 410 complete, 1 failed Load avg: 2.90, 2.90, 1.97 >>> Extracting info * Package: net-dns/bind-tools-9.10.3_p2 * Repository: gentoo * USE: userland_GNU elibc_glibc readline amd64 ssl abi_x86_64 seccomp urandom kernel_linux * FEATURES: preserve-libs sandbox userpriv usersandbox >>> Extracting net-dns/bind-tools-9.10.3_p2 * checking 29 files for package collisions >>> Merging net-dns/bind-tools-9.10.3_p2 to / --- /usr/ --- /usr/share/ --- /usr/share/man/ --- /usr/share/man/man8/ >>> /usr/share/man/man8/dnssec-revoke.8.bz2 >>> /usr/share/man/man8/dnssec-verify.8.bz2 >>> /usr/share/man/man8/dnssec-settime.8.bz2 >>> /usr/share/man/man8/dnssec-keygen.8.bz2 >>> /usr/share/man/man8/dnssec-keyfromlabel.8.bz2 >>> /usr/share/man/man8/dnssec-importkey.8.bz2 >>> /usr/share/man/man8/dnssec-dsfromkey.8.bz2 >>> /usr/share/man/man8/dnssec-signzone.8.bz2 --- /usr/share/man/man1/ >>> /usr/share/man/man1/host.1.bz2 >>> /usr/share/man/man1/nsupdate.1.bz2 >>> /usr/share/man/man1/dig.1.bz2 >>> /usr/share/man/man1/delv.1.bz2 >>> /usr/share/man/man1/nslookup.1.bz2 --- /usr/share/doc/ >>> /usr/share/doc/bind-tools-9.10.3_p2/ >>> /usr/share/doc/bind-tools-9.10.3_p2/CHANGES.bz2 >>> /usr/share/doc/bind-tools-9.10.3_p2/README.bz2 >>> /usr/share/doc/bind-tools-9.10.3_p2/FAQ.bz2 --- /usr/bin/ >>> /usr/bin/dnssec-keyfromlabel >>> /usr/bin/nsupdate >>> /usr/bin/dnssec-signzone >>> /usr/bin/nslookup >>> /usr/bin/dnssec-dsfromkey >>> /usr/bin/dnssec-settime >>> /usr/bin/dnssec-verify >>> /usr/bin/dnssec-revoke >>> /usr/bin/dnssec-keygen >>> /usr/bin/dnssec-importkey >>> /usr/bin/host >>> /usr/bin/delv >>> /usr/bin/dig >>> Safely unmerging already-installed instance... >>> Original instance of package unmerged safely. Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/portage/dbapi/_MergeProcess.py", line 235, in _spawn prev_mtimes=self.prev_mtimes, counter=counter) File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 5013, in merge counter=counter) File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 4297, in treewalk relative_paths=False) File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 1063, in removeFromContents self.writeContentsToContentsFile(pkg, new_contents) File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 1075, in writeContentsToContentsFile f = atomic_ofstream(os.path.join(pkg.dbdir, "CONTENTS")) File "/usr/lib64/python3.4/site-packages/portage/util/__init__.py", line 1288, in __init__ mode=mode, **portage._native_kwargs(kargs))) FileNotFoundError: [Errno 2] No such file or directory: b'/var/db/pkg/net-dns/bind-9.9.5-r3/CONTENTS.12168' dnssec bin and man pages moved from net-dns/bind-9.9.5-r3 to net-dns/bind-tools-9.10.3_p2 and emerge failed, i think, because with parallel-install the var/db/pkg/net-dns/bind-9.9.5-r3/CONTENTS disappeared by the parallel installation of net-dns/bind-9.10.3_p2 and net-dns/bind-tools-9.10.3_p2 . It is normal this time of error with parallel-install ( i hope no :) ) or the lock method didn't work ? How parallel-install and lock of ebuilds works ? Thanks
(In reply to marco from comment #2) > FileNotFoundError: [Errno 2] No such file or directory: > b'/var/db/pkg/net-dns/bind-9.9.5-r3/CONTENTS.12168' > > > dnssec bin and man pages moved from net-dns/bind-9.9.5-r3 to > net-dns/bind-tools-9.10.3_p2 and emerge failed, i think, because with > parallel-install the var/db/pkg/net-dns/bind-9.9.5-r3/CONTENTS disappeared > by the parallel installation of net-dns/bind-9.10.3_p2 and > net-dns/bind-tools-9.10.3_p2 . > > > It is normal this time of error with parallel-install ( i hope no :) ) or > the lock method didn't work ? Well apparently some assumptions made by the parallel-install code break down when resolving file collisions for soft-blockers. I've opened bug 576888 to track this. > > > How parallel-install and lock of ebuilds works ? By default, packages wait in a queue to go through the installation step one at a time. With parallel-install, there is no queue, so multiple packages are allowed to go through the installation step simultaneously. The ebuild-locks feature involves the execution of unsandboxed ebuild phases that execute with root privileges, like pkg_setup, pkg_prerm, and pkg_postrm. By default, packages wait in a queue to execute these phases one at a time. With -ebuild-locks, there is no queue, so multiple unsandboxed ebuild phases can execute simultaneously.
(In reply to marco from comment #2) > It is normal this time of error with parallel-install ( i hope no :) ) or > the lock method didn't work ? You can apply the patch for bug 576888 as follows: mkdir -p /etc/portage/patches/sys-apps/portage wget -O /etc/portage/patches/sys-apps/portage/bug_576888.patch \ https://github.com/zmedico/portage/commit/20bcc9aecaaf4ad5346d662e326fffa5c38fe701.patch emerge -1 portage
(In reply to Zac Medico from comment #4) > (In reply to marco from comment #2) > > It is normal this time of error with parallel-install ( i hope no :) ) or > > the lock method didn't work ? > > You can apply the patch for bug 576888 as follows: > > mkdir -p /etc/portage/patches/sys-apps/portage > wget -O /etc/portage/patches/sys-apps/portage/bug_576888.patch \ > https://github.com/zmedico/portage/commit/ > 20bcc9aecaaf4ad5346d662e326fffa5c38fe701.patch > emerge -1 portage I'm testing the patch and i'll notify you about the results. time FEATURES="parallel-install" emerge -vtuND --jobs=16 --with-bdeps=y world Total: 410 packages (275 upgrades, 1 downgrade, 68 new, 13 in new slots, 53 reinstalls, 410 binaries, 2 uninstalls), Size of downloads: 563,789 KiB Now during the tests ,with FEATURES="parallel-install", the update of 410 binary pkg moves from 42 min to 28 min :) with 4 cpu and 2GB of ram but i will test more times to trigger a possible race condition. a new question: is it possible to compress xpak files with different compression algorithm than bzip2, maybe gzip ? to speed up the installation ?
(In reply to marco from comment #5) > Now during the tests ,with FEATURES="parallel-install", the update of 410 > binary pkg moves from 42 min to 28 min :) with 4 cpu and 2GB of ram but i > will test more times to trigger a possible race condition. Thanks, testing is very much appreciated. > a new question: is it possible to compress xpak files with different > compression algorithm than bzip2, maybe gzip ? to speed up the installation ? Not yet, but it's very easy to implement this, since we already have support to detect the compression type and select an appropriate decompressor. Bug 142579 tracks this issue. You can set PORTAGE_BUNZIP2_COMMAND="pbzip2" to speed up installation wih parallel decompression.
Hi, with the patch you provided i didn't find any problems during updates on more than 20 servers. Thanks
(In reply to marco from comment #7) > Hi, > with the patch you provided i didn't find any problems during updates on > more than 20 servers. > > Thanks Thanks.
