the ebuild checks if there are ~28GB of available RAM before starting the build process. However, building for CUDA requires far more... My default MAKEOPTS = -j6 and I have 32 GB of RAM and another 32GB of swap in 2 partitions laid on 2 different NVME drives. The build for CUDA ate all 64 GB of my virtual memory and OOM terminated the build. this happened when 6 instances of ptxas were running, each consuming over 10GB of RAM. I tried to build with -j3 - no luck The build with -j1 succeeded. Can you propose a way to limit the parallel build for these memory hungry operations only?
PF16W6Y2 / # emerge --info Portage 3.0.64 (python 3.11.9-final-0, default/linux/amd64/17.1/desktop/gnome/systemd/merged-usr, gcc-13, glibc-2.39-r6, 6.6.30-gentoo x86_64) ================================================================= System uname: Linux-6.6.30-gentoo-x86_64-Intel-R-_Core-TM-_i7-8850H_CPU_@_2.60GHz-with-glibc2.39 KiB Mem: 32561988 total, 15642092 free KiB Swap: 33554424 total, 33098008 free Timestamp of repository gentoo: Mon, 20 May 2024 16:03:50 +0000 Head commit of repository gentoo: 154cb890c159bdb8c4d2ef939b9bd3e09dbcdb51 Timestamp of repository audio-overlay: Sun, 12 May 2024 05:07:13 +0000 Head commit of repository audio-overlay: a44d2eba7a59b0dd48dae45ad1ca675fba86cfe6 Timestamp of repository guru: Mon, 20 May 2024 00:33:35 +0000 Head commit of repository guru: fd9c52b7541d9331e5708f05722c40f1972e8172 Timestamp of repository vowstar: Fri, 17 May 2024 12:20:59 +0000 Head commit of repository vowstar: b21fd42a8a235279755f6007bcec40e4a239be48 Head commit of repository calculate: fcfa6fc0c50d77a0dd5fbff31b25b3840a5cf514 Head commit of repository dlang: e1447e5adf99d1384904d0893ad038884f4124d5 Head commit of repository gentoo-zh: 076ff4f635ac63333faad237009d70520f352901 Head commit of repository go-overlay: bde6b0fe1edcb9202c5c95b1f286cdf1fa64ddea Head commit of repository lto-overlay: a46a5749ee6e70bcf2317f9d6e24037779b643a8 Head commit of repository mv: 01e4cba9a9aa27badc903a60847717992df9dc7e Head commit of repository steam-overlay: fc142da83eb4eb440c787e9f1befe5945b09f485 Head commit of repository tlp: af2991838d36eb8337616cb66dcda93d455fdd66 Head commit of repository rugubara: 65136b7a37e25e498d66a1f6844dfb1b5cc20bbf sh bash 5.2_p26-r3 ld GNU ld (Gentoo 2.42 p3) 2.42.0 ccache version 4.9.1 [disabled] app-misc/pax-utils: 1.3.7::gentoo app-shells/bash: 5.2_p26-r3::gentoo dev-build/autoconf: 2.13-r8::gentoo, 2.72-r1::gentoo dev-build/automake: 1.16.5-r2::gentoo dev-build/cmake: 3.29.3::gentoo dev-build/libtool: 2.4.7-r4::gentoo dev-build/make: 4.4.1-r1::gentoo dev-build/meson: 1.4.0-r1::gentoo dev-java/java-config: 2.3.3-r1::gentoo dev-lang/perl: 5.38.2-r5::gentoo dev-lang/python: 3.10.14_p1::gentoo, 3.11.9::gentoo, 3.12.3::gentoo, 3.13.0_beta1_p1::gentoo dev-lang/rust: 1.77.1::gentoo dev-util/ccache: 4.9.1-r1::gentoo sys-apps/baselayout: 2.15::gentoo sys-apps/sandbox: 2.38::gentoo sys-apps/systemd: 255.6::gentoo sys-devel/binutils: 2.42-r1::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/clang: 15.0.7-r3::gentoo, 17.0.6::gentoo, 18.1.6::gentoo sys-devel/gcc: 12.3.1_p20240209::gentoo, 13.2.1_p20240503::gentoo, 14.1.1_p20240518::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/lld: 17.0.6::gentoo, 18.1.6::gentoo sys-devel/llvm: 15.0.7-r3::gentoo, 17.0.6::gentoo, 18.1.6::gentoo sys-kernel/linux-headers: 6.9::gentoo (virtual/os-headers) sys-libs/glibc: 2.39-r6::gentoo Repositories: gentoo location: /usr/portage sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo.git priority: -1000 volatile: True audio-overlay location: /var/db/repos/audio-overlay sync-type: git sync-uri: https://github.com/gentoo-mirror/audio-overlay.git masters: gentoo volatile: False guru location: /var/db/repos/guru sync-type: git sync-uri: https://github.com/gentoo-mirror/guru.git masters: gentoo volatile: False vowstar location: /var/db/repos/vowstar sync-type: git sync-uri: https://github.com/gentoo-mirror/vowstar.git masters: gentoo volatile: False calculate location: /var/lib/layman/calculate sync-type: git sync-uri: https://git.calculate-linux.org/calculate/calculate-overlay.git masters: gentoo priority: 50 volatile: True dlang location: /var/lib/layman/dlang sync-type: git sync-uri: https://github.com/gentoo/dlang.git masters: gentoo priority: 50 volatile: True gentoo-zh location: /var/lib/layman/gentoo-zh sync-type: git sync-uri: https://github.com/microcai/gentoo-zh.git masters: gentoo priority: 50 volatile: True go-overlay location: /var/lib/layman/go-overlay sync-type: git sync-uri: https://github.com/Dr-Terrible/go-overlay.git masters: gentoo priority: 50 volatile: True lto-overlay location: /var/lib/layman/lto-overlay sync-type: git sync-uri: https://github.com/InBetweenNames/gentooLTO.git masters: gentoo mv priority: 50 volatile: True mv location: /var/lib/layman/mv sync-type: git sync-uri: https://anongit.gentoo.org/git/user/mv.git masters: gentoo priority: 50 volatile: True steam-overlay location: /var/lib/layman/steam-overlay sync-type: git sync-uri: https://github.com/anyc/steam-overlay.git masters: gentoo priority: 50 volatile: True tlp location: /var/lib/layman/tlp sync-type: git sync-uri: https://github.com/dywisor/tlp-portage.git masters: gentoo priority: 50 volatile: True rugubara location: /var/lib/layman/rugubara sync-type: git sync-uri: https://github.com/rugubara/overlay.git masters: gentoo priority: 100 volatile: True ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA Vivaldi" CBUILD="x86_64-pc-linux-gnu" CC="gcc" CFLAGS="-march=native -O3 -fuse-linker-plugin -falign-functions=32 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXX="g++" CXXFLAGS="-march=native -O3 -fuse-linker-plugin -falign-functions=32 -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-march=native -O3 -fuse-linker-plugin -falign-functions=32 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-march=native -O3 -fuse-linker-plugin -falign-functions=32 -pipe" GENTOO_MIRRORS="http://gentoo.bloodhost.ru/ rsync://gentoo.bloodhost.ru/gentoo-distfiles http://ftp.ntua.gr/pub/linux/gentoo/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" MAKEOPTS="-j6" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="X a52 aac acl acpi aio alsa amd64 ap bash-completion bluetooth branding bzip2 cacert cairo caps cdda cdr cgroup-hybrid chromecast clang cli client colord crypt cups dbus device-mapper djvu dri dri3 dts dvd dvdr eds egl encode evo exif fat ffmpeg fftw flac fortran gdbm gif gimp gles2 gnome gnome-keyring gnome-online-accounts gnome-shell gpm graphite gstreamer gtk gtk3 gui iconv icu introspection io-uring ipv6 iwd jack jpeg jpeg2k keyring lcms libglvnd libnotify libtirpc libzfs lm_sensors lto lv2 mad mng mp3 mp4 mpeg multilib native-headset nautilus ncat ncurses networkmanager nfs nls ntfs nvenc nvidia nvme ogg opencl opengl openmp opus p2p pam pango pcre pdf pipewire png policykit ppds pulseaudio qt5 raw readline screencast sdl sdl2 seccomp smp sna sound sound-server spell spice ssl startup-notification svg synaptics sysprof system-llvm systemd test-rust thunderbolt tiff tracker truetype udev udisks uefi unicode upower usb usbredir user-session vaapi vala vdpau virgl vlc vorbis vulkan wayland webdav wireguard wxwidgets x264 x265 xattr xcb xft xml xv xvid zeroconf zfs zlib zstd" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="libinput wacom" KERNEL="linux" L10N="en ru" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31 ruby32" VIDEO_CARDS="intel i965 iris nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Not an actual solution to the point but a mere work-around. In a similar configuration (default MAKEOPTS and RAM equipment) I created a custom env. That env has the notmpfs characteristics (so saving same actual space, since my emerge default is to mount TMPDIR on tmpfs) PORTAGE_TMPDIR="/var/tmp/notmpfs" and redefine MAKEOPTS to -j1 for safety I also provided the env with: EMERGE_DEFAULT_OPTS="--jobs 1" (but it seems not to influence emerge that much, anyway it did not break it) By listing sci-libs/vtk in the corresponding package.env got a straight-forward maintenance of sci-libs/vtk and adapt the check of available RAM to less than 8GB. Some minor RAM consumption may (not sure and not tested) be pursued by some setting of NVCC flags (see: https://bugs.gentoo.org/show_bug.cgi?id=930031 https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html )
i can confirm this. i set the env for vtk to use MAKEOPTS=-j1 even though i have 64GB of ram to prevent oom.
Can you tell me where exactly it OOMs? USE-Flags and other details, like the gpu would also help. VTK is huge and pin-pointing the exact issue blindly is impossible. For vtkm a job pool is used, we might increase the required memory size for that.
I'm rebuilding vtk with MAKEOPTS=-j6 [ebuild R ] sci-libs/vtk-9.3.0:0/9.3::gentoo USE="boost cuda ffmpeg freetype logging openmp qt6 rendering sdl threads views vtkm -all-modules (-debug) -doc -examples -gdal -gles2-only -imaging -java -las -mpi -mysql -odbc -opencascade -openvdb -pdal -postgres -python -qt5 -tbb -test -tk -web" PYTHON_SINGLE_TARGET="python3_12 -python3_10 -python3_11" VIDEO_CARDS="nvidia" 0 KiB I'll post the log and also extract the OOM messages from the journal.
Created attachment 895818 [details] build log and env This is build log and environment. The OOM message is too big, will be in a separate file
Created attachment 895819 [details] OOM message
(In reply to Paul Zander from comment #4) > > For vtkm a job pool is used, we might increase the required memory size for > that. I'm not sure what is the average system used to build vtk... I wouldn't be happy if the package declares over 64GB of RAM + SWAP requirement... Can we serialize the compilation of the most memory-hungry parts not to allow them to run in parallel?
Increasing the "required memory size" would mean more memory is required per started job. So overall less jobs in parallel.
