When I'm building chromium, I see ninja does not use distcc. Even worse: while I'm specifying the number of jobs (-j70 -l12), it starts 70 compile jobs, but keeps all of them asleep. Only 4 of them are running. Not with distcc, but as "portage" user. I'll attach a ps axjf output to illustrate. The load of the system never rises above 4. Reproducible: Always Steps to Reproduce: 1. compile chromium with portage 2. wait a bit until ninja kicks in 3. watch the build system with ps axjf Actual Results: no distcc is used, 70 clang compiler jobs are started but kept asleep, only 4 are running Expected Results: compiler jobs are offloaded to distcc, and if this is not possible, at least more than 4 jobs are running on a 12 core cpu with -j70 -l10 make opts CPU: ryzen 5 4650G (12 core) RAM: 64GB /etc/distcc/hosts: remote/64 localhost/8 emerge --info: Portage 3.0.66.1 (python 3.12.6-final-0, default/linux/amd64/23.0/split-usr/desktop/plasma, gcc-14, glibc-2.40-r3, 6.10.7 x86_64) ================================================================= System uname: Linux-6.10.7-x86_64-AMD_Ryzen_5_PRO_4650GE_with_Radeon_Graphics-with-glibc2.40 KiB Mem: 65243696 total, 22228280 free KiB Swap: 30023672 total, 30023672 free Timestamp of repository gentoo: Fri, 04 Oct 2024 22:15:00 +0000 Head commit of repository gentoo: b1d63bd43391738a0ae1f99a489208ddea1953b1 sh bash 5.2_p26-r6 ld GNU ld (Gentoo 2.43 p2) 2.43.1 distcc 3.4 x86_64-pc-linux-gnu [enabled] app-misc/pax-utils: 1.3.7::gentoo app-shells/bash: 5.2_p26-r6::gentoo dev-build/autoconf: 2.13-r8::gentoo, 2.72-r1::gentoo dev-build/automake: 1.16.5-r2::gentoo dev-build/cmake: 3.30.3::gentoo dev-build/libtool: 2.4.7-r4::gentoo dev-build/make: 4.4.1-r100::gentoo dev-build/meson: 1.5.2::gentoo dev-java/java-config: 2.3.4::gentoo dev-lang/perl: 5.40.0::gentoo dev-lang/python: 3.11.10_p1::gentoo, 3.12.6_p4::gentoo dev-lang/rust: 1.81.0::gentoo sys-apps/baselayout: 2.15::gentoo sys-apps/openrc: 0.54.2::gentoo sys-apps/sandbox: 2.39::gentoo sys-devel/binutils: 2.43-r1::gentoo sys-devel/binutils-config: 5.5.2::gentoo sys-devel/clang: 17.0.6::gentoo, 18.1.8::gentoo, 19.1.0::gentoo sys-devel/gcc: 13.3.1_p20240920::gentoo, 14.2.1_p20240921::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/lld: 18.1.8::gentoo, 19.1.0::gentoo sys-devel/llvm: 17.0.6-r3::gentoo, 18.1.8-r4::gentoo, 19.1.0::gentoo sys-kernel/linux-headers: 6.10::gentoo (virtual/os-headers) sys-libs/glibc: 2.40-r3::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: rsync sync-uri: rsync://linuxserver/gentoo-portage priority: -1000 volatile: False sync-rsync-verify-max-age: 24 sync-rsync-extra-opts: sync-rsync-verify-jobs: 1 sync-rsync-verify-metamanifest: yes ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=znver1 -mtune=znver2 -flto -O3 -pipe " CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=znver1 -mtune=znver2 -flto -O3 -pipe " DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS=" --usepkg" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-march=znver1 -mtune=znver2 -flto -O3 -pipe " FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg buildpkg-live config-protect-if-modified distcc distlocks fixlafiles merge-sync merge-wait multilib-strict news nostrip parallel-fetch pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-march=znver1 -mtune=znver2 -flto -O3 -pipe " GENTOO_MIRRORS="ftp://ftp.free.fr/mirrors/ftp.gentoo.org/ rsync://mirror.eu.oneandone.net/gentoo/ rsync://mirror.netcologne.de/gentoo/ rsync://mirror.leaseweb.com/gentoo/ rsync://mirror.bytemark.co.uk/gentoo/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs" LEX="flex" MAKEOPTS="-j70 -l12" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="X a52 aac acl acpi activities alsa amd64 bluetooth branding bzip2 cairo caps cdda cdr cet crypt cups custom-cflags dbus declarative dri dts dvd dvdr elogind encode exif ffmpeg flac gdbm gif gnutls gpm gtk gui i18n iconv icu ipv6 java jpeg kde kf6compat kwallet lcms libkms libnotify libtirpc lm-sensors lto mad mng mp3 mp4 mpeg multilib ncurses networkmanager nfs nls nptl ogg opengl openmp pam pango pcap pcre pdf pipewire plasma png policykit ppds pulseaudio qml qt5 qt6 readline screencast sdl seccomp semantic-desktop sound spell split-usr ssl startup-notification svg test-rust tiff truetype udev udisks unicode upower usb vaapi vorbis vulkan wayland widgets wxwidgets x264 xattr xcb xft xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes sse sse2 sse3 ssse3 pni sse4_1 sse4_2 avx avx2 bmi1 bmi2 fma3 mmx mmxext sha f16c rdrand pclmul popcnt" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GRUB_PLATFORMS="efi-64" GUILE_SINGLE_TARGET="3-0" GUILE_TARGETS="3-0" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-GB nl fr" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LLVM_SLOT="18" LLVM_TARGETS="AMDGPU X86 SPIRV" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres16" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_12 python3_11" QEMU_SOFTMMU_TARGETS="x86_64 i386 aarch64" QEMU_USER_TARGETS="x86_64 i386 aarch64 arm" RUBY_TARGETS="ruby32" SANE_BACKENDS="xerox_mfp net" VIDEO_CARDS="amdgpu radeonsi radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
I don't think distcc gets used at all with Chromium indeed, nor can it given how it relies on bundled toolchain a lot of the time. It's a miracle to be able to build Chromium at all outside of Google's tooling.
Created attachment 904947 [details] ps axjf output when compiling
(In reply to Sam James from comment #1) > I don't think distcc gets used at all with Chromium indeed, nor can it given > how it relies on bundled toolchain a lot of the time. It's a miracle to be > able to build Chromium at all outside of Google's tooling. Yes, but even when not using distcc, it should NOT start 70 compiler jobs and keep 66 asleep on a 12 core cpu ... that does not make sense
Please try with ninja-1.12.1.
(In reply to Mike Gilbert from comment #4) > Please try with ninja-1.12.1. no luck, unfortunately: keeps compiling with only 4 active processes, system load tops out at 4.0
Created attachment 905031 [details] build log with first 10000 lines
Does the behavior change with FEATURES="-distcc"?
hi, sorry for the delayed info: using FEATURES="-distcc" solves it for me. So for whatever reason, using the distcc flag triggers a -j4 limit ...
I suspect the 4 threads limit is a distcc config issue. I also suspect that ninja is not to blame for remote distcc nodes not working. That is more likely some issue with www-client/chromium specifically. If you have another example of a ninja-based package that exhibits the behavior, please share.
Do your distcc nodes all have "x86_64-pc-linux-gnu-clang++-18" as a symlink in /usr/lib/distcc/bin?
(In reply to Mike Gilbert from comment #10) > Do your distcc nodes all have "x86_64-pc-linux-gnu-clang++-18" as a symlink > in /usr/lib/distcc/bin? Also in /usr/lib/distcc.
