| Summary: | emerge --sync freezes my machine for up to 5 minutes | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Martin Doucha <next_ghost> |
| Component: | Core | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | kfm |
| Priority: | Normal | Keywords: | InVCS |
| Version: | unspecified | ||
| Hardware: | AMD64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 650144, 651804 | ||
|
Description
Martin Doucha
2018-03-17 10:38:22 UTC
What filesystem are you using for /usr/portage? You can use a setting like this to reduce the number of gemato processes in /etc/portage/repos.conf: [gentoo] sync-rsync-verify-jobs = 1 /usr/portage on my machine is stored on an Ext4 partition. Setting sync-rsync-verify-jobs has solved the issue for me. I suggest reducing the default value to something even less powerful machines can handle. (In reply to Martin Doucha from comment #2) > /usr/portage on my machine is stored on an Ext4 partition. Setting > sync-rsync-verify-jobs has solved the issue for me. I suggest reducing the > default value to something even less powerful machines can handle. The thing is, performance varies widely depending on the disk type, filesystem type, RAID, and the I/O scheduler configured in the kernel. The current defaults perform well when the all of these factors are nicely tuned. Maybe switching your filesystem from ext4 to btrfs would be enough to optimize it for you. Default settings are supposed to work everywhere, not just on carefully tuned machines with special high-performance hardware. Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/7a7dce75680a1454eb3e1d84a1e4f8c1 https://github.com/gentoo/portage/pull/300 If I may ask, how many processes of gemato do you typically see running in the freeze scenario? 4 or more? (In reply to Fabian Groffen from comment #6) > If I may ask, how many processes of gemato do you typically see running in > the freeze scenario? 4 or more? I can't reproduce the issue right now and Emerge doesn't seem to launch standalone gemato processes anymore (I can see only multiple instances of Emerge in Top). But I /think/ Emerge launched at least 6 gemato processes when I reported the issue, maybe 8. Thanks, that's what I wanted to know. Your system should have 2 or 4 cpus as detected, is that right? Note that in gemato-13.0, multiprocessing is disabled due to the unresolved deadlock issue reported in bug 647964. For earlier versions of gemato, this should report the default number of processes: > python -c 'import multiprocessing; print(multiprocessing.cpu_count())' That should be the same number as the default --jobs setting that you can see here: > gemato verify --help | grep default (In reply to Fabian Groffen from comment #8) > Thanks, that's what I wanted to know. Your system should have 2 or 4 cpus > as detected, is that right? I have Intel Core i5, dual core with hyperthreading, so 4 virtual CPUs total. I still have gemato-12.2 and `gemato verify --help` reports "default: 4" for --jobs $ python -c 'import multiprocessing; print(multiprocessing.cpu_count())' 4 The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=7b448e90034de00bc177b3d809aeaf9b94d55ee2 commit 7b448e90034de00bc177b3d809aeaf9b94d55ee2 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-13 17:01:45 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-17 02:25:25 +0000 rsync: default to sync-rsync-verify-jobs = 1 (bug 650696) Some users have reported that using all processors to verify manifests results in overloading, therefore default to using a single processor. On modern hardware, verification of the gentoo repository completes in less than 20 seconds, so using multiple processors is not really necessary. Also, gemato-13.0 disables parallel verification due to the unresolved deadlock issue reported in bug 647964, so this brings the default portage configuration into alignment with current gemato behavior. Bug: https://bugs.gentoo.org/650696 Bug: https://bugs.gentoo.org/647964 cnf/repos.conf | 1 + man/portage.5 | 8 +++++--- pym/portage/sync/modules/rsync/rsync.py | 7 ++++++- 3 files changed, 12 insertions(+), 4 deletions(-)} Fixed in portage-2.3.40-r1. |
