Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 650696 - emerge --sync freezes my machine for up to 5 minutes
Summary: emerge --sync freezes my machine for up to 5 minutes
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Portage team
Keywords: InVCS
Depends on:
Blocks: 650144 651804
  Show dependency tree
Reported: 2018-03-17 10:38 UTC by Martin Doucha
Modified: 2018-07-02 18:50 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Martin Doucha 2018-03-17 10:38:22 UTC
When I run emerge --sync nowadays, it'll make my system unresponsive for about 5 minutes through heavy disk load. My system doesn't run out of memory (over 2GB RAM free during the freeze) but `top` reports that load average shoots up to 16 (on a 2-core CPU with HyperThreading) and almost all of the load are processes waiting for I/O operations. The main problem seems to be that `emerge` launches too many instances of `gemato` at the same time.

emerge --info:
Portage 2.3.24 (python 3.5.4-final-0, default/linux/amd64/17.0/desktop/plasma, gcc-7.3.0, glibc-2.26-r6, 4.15.7-gentoo-r1 x86_64)
System uname: Linux-4.15.7-gentoo-r1-x86_64-Intel-R-_Core-TM-_i5-4200M_CPU_@_2.50GHz-with-gentoo-2.4.1
KiB Mem:     7851596 total,   2769320 free
KiB Swap:   16777212 total,  16777212 free
Timestamp of repository gentoo: Sat, 17 Mar 2018 09:15:01 +0000
Head commit of repository gentoo: 7b8449420e13c43dddd4daf62bcfddff5bf986b3
sh bash 4.4_p19
ld GNU ld (Gentoo 2.30 p1) 2.30.0
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
app-shells/bash:          4.4_p19::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.26.1-r2::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.4-r1::gentoo, 3.6.4::gentoo
dev-util/cmake:           3.10.2::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.35.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.14.1-r2::gentoo, 1.15.1-r2::gentoo, 1.16-r1::gentoo
sys-devel/binutils:       2.30::gentoo
sys-devel/gcc:            6.4.0::gentoo, 7.2.0-r1::gentoo, 7.3.0::gentoo
sys-devel/gcc-config:     1.9.1::gentoo
sys-devel/libtool:        2.4.6-r4::gentoo
sys-devel/make:           4.2.1-r2::gentoo
sys-kernel/linux-headers: 4.15::gentoo (virtual/os-headers)
sys-libs/glibc:           2.26-r6::gentoo

    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://
    priority: -1000
    sync-rsync-extra-opts: --timeout=300
    sync-rsync-verify-metamanifest: yes

    location: /usr/local/portage
    masters: gentoo
    priority: 0

    location: /var/lib/layman/gamerlay
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA AdobeFlash-11.x dlj-1.1 PUEL RTCW-ETEULA Oracle-BCLA-JavaSE"
CFLAGS="-O2 -pipe"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/X11/xkb/symbols /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/apache2-php7.1/ext-active/ /etc/php/apache2-php7.2/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="cs en"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
USE="X a52 aac acl acpi activities alsa amd64 avx avx2 bash-completion branding bzip2 cairo cdda cdr cjk cli consolekit crypt cups curl cxx dbus declarative djvu dri dts dvd dvdr dvdread emboss encode exif faac fam ffmpeg flac fontconfig gdbm gif glamor gnutls gpm gtk ibus iconv ipv6 javascript jbig jce jpeg jpeg2k kde kipi kwallet laptop lcms libass libnotify lm_sensors logrotate lzma mad mmx mmxext mng modules mp3 mp4 mpeg mtp multilib ncurses nls nptl nptlonly nsplugin ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds qml qt3support qt5 readline real rtc sdl seccomp semantic-desktop spell sse sse2 sse3 sse4 sse4_1 sse4_2 ssl ssse3 startup-notification svg tcpd tga theora tiff truetype udev udisks unicode upower usb vaapi vcd vdpau vorbis vpx webp widgets wifi win32codecs wxwidgets x264 x265 xattr xcb xcomposite xinerama xml xscreensaver xv xvid xvmc zlib" ABI_X86="32 64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="krita" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="avx avx2 mmx mmxext sse sse2 sse3 ssse3 sse4 sse4_1 sse4_2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="evdev keyboard mouse synaptics" KERNEL="linux" L10N="cs en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_5 python3_6" RUBY_TARGETS="ruby24 ruby25" SANE_BACKENDS="net" USERLAND="GNU" VIDEO_CARDS="intel i915 i965 radeon radeonsi amdgpu" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Reproducible: Always
Comment 1 Zac Medico gentoo-dev 2018-03-17 18:01:48 UTC
What filesystem are you using for /usr/portage?

You can use a setting like this to reduce the number of gemato processes in /etc/portage/repos.conf:

sync-rsync-verify-jobs = 1
Comment 2 Martin Doucha 2018-03-18 11:28:35 UTC
/usr/portage on my machine is stored on an Ext4 partition. Setting sync-rsync-verify-jobs has solved the issue for me. I suggest reducing the default value to something even less powerful machines can handle.
Comment 3 Zac Medico gentoo-dev 2018-03-18 18:52:35 UTC
(In reply to Martin Doucha from comment #2)
> /usr/portage on my machine is stored on an Ext4 partition. Setting
> sync-rsync-verify-jobs has solved the issue for me. I suggest reducing the
> default value to something even less powerful machines can handle.

The thing is, performance varies widely depending on the disk type, filesystem type, RAID, and the I/O scheduler configured in the kernel. The current defaults perform well when the all of these factors are nicely tuned. Maybe switching your filesystem from ext4 to btrfs would be enough to optimize it for you.
Comment 4 Martin Doucha 2018-03-20 14:38:47 UTC
Default settings are supposed to work everywhere, not just on carefully tuned machines with special high-performance hardware.
Comment 6 Fabian Groffen gentoo-dev 2018-04-13 18:13:18 UTC
If I may ask, how many processes of gemato do you typically see running in the freeze scenario? 4 or more?
Comment 7 Martin Doucha 2018-04-13 18:42:03 UTC
(In reply to Fabian Groffen from comment #6)
> If I may ask, how many processes of gemato do you typically see running in
> the freeze scenario? 4 or more?

I can't reproduce the issue right now and Emerge doesn't seem to launch standalone gemato processes anymore (I can see only multiple instances of Emerge in Top). But I /think/ Emerge launched at least 6 gemato processes when I reported the issue, maybe 8.
Comment 8 Fabian Groffen gentoo-dev 2018-04-13 18:47:54 UTC
Thanks, that's what I wanted to know.  Your system should have 2 or 4 cpus as detected, is that right?
Comment 9 Zac Medico gentoo-dev 2018-04-13 18:50:51 UTC
Note that in gemato-13.0, multiprocessing is disabled due to the unresolved deadlock issue reported in bug 647964.

For earlier versions of gemato, this should report the default number of processes:

> python -c 'import multiprocessing; print(multiprocessing.cpu_count())'

That should be the same number as the default --jobs setting that you can see here:

> gemato verify --help | grep default
Comment 10 Martin Doucha 2018-04-13 19:06:23 UTC
(In reply to Fabian Groffen from comment #8)
> Thanks, that's what I wanted to know.  Your system should have 2 or 4 cpus
> as detected, is that right?

I have Intel Core i5, dual core with hyperthreading, so 4 virtual CPUs total.

I still have gemato-12.2 and `gemato verify --help` reports "default: 4" for --jobs

$ python -c 'import multiprocessing; print(multiprocessing.cpu_count())'
Comment 11 Larry the Git Cow gentoo-dev 2018-04-17 02:26:43 UTC
The bug has been referenced in the following commit(s):

commit 7b448e90034de00bc177b3d809aeaf9b94d55ee2
Author:     Zac Medico <>
AuthorDate: 2018-04-13 17:01:45 +0000
Commit:     Zac Medico <>
CommitDate: 2018-04-17 02:25:25 +0000

    rsync: default to sync-rsync-verify-jobs = 1 (bug 650696)
    Some users have reported that using all processors to verify
    manifests results in overloading, therefore default to using
    a single processor. On modern hardware, verification of the
    gentoo repository completes in less than 20 seconds, so using
    multiple processors is not really necessary. Also, gemato-13.0
    disables parallel verification due to the unresolved deadlock
    issue reported in bug 647964, so this brings the default
    portage configuration into alignment with current gemato

 cnf/repos.conf                          | 1 +
 man/portage.5                           | 8 +++++---
 pym/portage/sync/modules/rsync/ | 7 ++++++-
 3 files changed, 12 insertions(+), 4 deletions(-)}
Comment 12 Zac Medico gentoo-dev 2018-07-02 18:50:56 UTC
Fixed in portage-2.3.40-r1.