Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 932464 - sci-libs/vtk-9.3.0[cuda] huge RAM requirements for compilation
Summary: sci-libs/vtk-9.3.0[cuda] huge RAM requirements for compilation
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Paul Zander
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-05-22 12:59 UTC by Anton Gubarkov
Modified: 2024-07-10 18:54 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build log and env (build-env.zip,218.24 KB, application/zip)
2024-06-14 12:35 UTC, Anton Gubarkov
Details
OOM message (vtk-oom.txt,34.14 KB, text/plain)
2024-06-14 12:36 UTC, Anton Gubarkov
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Anton Gubarkov 2024-05-22 12:59:17 UTC
the ebuild checks if there are ~28GB of available RAM before starting the build process.

However, building for CUDA requires far more... 
My default MAKEOPTS = -j6 and I have 32 GB of RAM and another 32GB of swap in 2 partitions laid on 2 different NVME drives.

The build for CUDA ate all 64 GB of my virtual memory and OOM terminated the build. this happened when 6 instances of ptxas were running, each consuming over 10GB of RAM.

I tried to build with -j3 - no luck
The build with -j1 succeeded.

Can you propose a way to limit the parallel build for these memory hungry operations only?
Comment 1 Anton Gubarkov 2024-05-22 12:59:43 UTC
PF16W6Y2 / # emerge --info
Portage 3.0.64 (python 3.11.9-final-0, default/linux/amd64/17.1/desktop/gnome/systemd/merged-usr, gcc-13, glibc-2.39-r6, 6.6.30-gentoo x86_64)
=================================================================
System uname: Linux-6.6.30-gentoo-x86_64-Intel-R-_Core-TM-_i7-8850H_CPU_@_2.60GHz-with-glibc2.39
KiB Mem:    32561988 total,  15642092 free
KiB Swap:   33554424 total,  33098008 free
Timestamp of repository gentoo: Mon, 20 May 2024 16:03:50 +0000
Head commit of repository gentoo: 154cb890c159bdb8c4d2ef939b9bd3e09dbcdb51

Timestamp of repository audio-overlay: Sun, 12 May 2024 05:07:13 +0000
Head commit of repository audio-overlay: a44d2eba7a59b0dd48dae45ad1ca675fba86cfe6

Timestamp of repository guru: Mon, 20 May 2024 00:33:35 +0000
Head commit of repository guru: fd9c52b7541d9331e5708f05722c40f1972e8172

Timestamp of repository vowstar: Fri, 17 May 2024 12:20:59 +0000
Head commit of repository vowstar: b21fd42a8a235279755f6007bcec40e4a239be48

Head commit of repository calculate: fcfa6fc0c50d77a0dd5fbff31b25b3840a5cf514

Head commit of repository dlang: e1447e5adf99d1384904d0893ad038884f4124d5

Head commit of repository gentoo-zh: 076ff4f635ac63333faad237009d70520f352901

Head commit of repository go-overlay: bde6b0fe1edcb9202c5c95b1f286cdf1fa64ddea

Head commit of repository lto-overlay: a46a5749ee6e70bcf2317f9d6e24037779b643a8

Head commit of repository mv: 01e4cba9a9aa27badc903a60847717992df9dc7e

Head commit of repository steam-overlay: fc142da83eb4eb440c787e9f1befe5945b09f485

Head commit of repository tlp: af2991838d36eb8337616cb66dcda93d455fdd66

Head commit of repository rugubara: 65136b7a37e25e498d66a1f6844dfb1b5cc20bbf

sh bash 5.2_p26-r3
ld GNU ld (Gentoo 2.42 p3) 2.42.0
ccache version 4.9.1 [disabled]
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p26-r3::gentoo
dev-build/autoconf:        2.13-r8::gentoo, 2.72-r1::gentoo
dev-build/automake:        1.16.5-r2::gentoo
dev-build/cmake:           3.29.3::gentoo
dev-build/libtool:         2.4.7-r4::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.4.0-r1::gentoo
dev-java/java-config:      2.3.3-r1::gentoo
dev-lang/perl:             5.38.2-r5::gentoo
dev-lang/python:           3.10.14_p1::gentoo, 3.11.9::gentoo, 3.12.3::gentoo, 3.13.0_beta1_p1::gentoo
dev-lang/rust:             1.77.1::gentoo
dev-util/ccache:           4.9.1-r1::gentoo
sys-apps/baselayout:       2.15::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-apps/systemd:          255.6::gentoo
sys-devel/binutils:        2.42-r1::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           15.0.7-r3::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-devel/gcc:             12.3.1_p20240209::gentoo, 13.2.1_p20240503::gentoo, 14.1.1_p20240518::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/lld:             17.0.6::gentoo, 18.1.6::gentoo
sys-devel/llvm:            15.0.7-r3::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-kernel/linux-headers:  6.9::gentoo (virtual/os-headers)
sys-libs/glibc:            2.39-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    priority: -1000
    volatile: True

audio-overlay
    location: /var/db/repos/audio-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/audio-overlay.git
    masters: gentoo
    volatile: False

guru
    location: /var/db/repos/guru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/guru.git
    masters: gentoo
    volatile: False

vowstar
    location: /var/db/repos/vowstar
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/vowstar.git
    masters: gentoo
    volatile: False

calculate
    location: /var/lib/layman/calculate
    sync-type: git
    sync-uri: https://git.calculate-linux.org/calculate/calculate-overlay.git
    masters: gentoo
    priority: 50
    volatile: True

dlang
    location: /var/lib/layman/dlang
    sync-type: git
    sync-uri: https://github.com/gentoo/dlang.git
    masters: gentoo
    priority: 50
    volatile: True

gentoo-zh
    location: /var/lib/layman/gentoo-zh
    sync-type: git
    sync-uri: https://github.com/microcai/gentoo-zh.git
    masters: gentoo
    priority: 50
    volatile: True

go-overlay
    location: /var/lib/layman/go-overlay
    sync-type: git
    sync-uri: https://github.com/Dr-Terrible/go-overlay.git
    masters: gentoo
    priority: 50
    volatile: True

lto-overlay
    location: /var/lib/layman/lto-overlay
    sync-type: git
    sync-uri: https://github.com/InBetweenNames/gentooLTO.git
    masters: gentoo mv
    priority: 50
    volatile: True

mv
    location: /var/lib/layman/mv
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/user/mv.git
    masters: gentoo
    priority: 50
    volatile: True

steam-overlay
    location: /var/lib/layman/steam-overlay
    sync-type: git
    sync-uri: https://github.com/anyc/steam-overlay.git
    masters: gentoo
    priority: 50
    volatile: True

tlp
    location: /var/lib/layman/tlp
    sync-type: git
    sync-uri: https://github.com/dywisor/tlp-portage.git
    masters: gentoo
    priority: 50
    volatile: True

rugubara
    location: /var/lib/layman/rugubara
    sync-type: git
    sync-uri: https://github.com/rugubara/overlay.git
    masters: gentoo
    priority: 100
    volatile: True

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA Vivaldi"
CBUILD="x86_64-pc-linux-gnu"
CC="gcc"
CFLAGS="-march=native -O3     -fuse-linker-plugin -falign-functions=32 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXX="g++"
CXXFLAGS="-march=native -O3     -fuse-linker-plugin -falign-functions=32 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-march=native -O3     -fuse-linker-plugin -falign-functions=32 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-march=native -O3     -fuse-linker-plugin -falign-functions=32 -pipe"
GENTOO_MIRRORS="http://gentoo.bloodhost.ru/ rsync://gentoo.bloodhost.ru/gentoo-distfiles http://ftp.ntua.gr/pub/linux/gentoo/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LEX="flex"
MAKEOPTS="-j6"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X a52 aac acl acpi aio alsa amd64 ap bash-completion bluetooth branding bzip2 cacert cairo caps cdda cdr cgroup-hybrid chromecast clang cli client colord crypt cups dbus device-mapper djvu dri dri3 dts dvd dvdr eds egl encode evo exif fat ffmpeg fftw flac fortran gdbm gif gimp gles2 gnome gnome-keyring gnome-online-accounts gnome-shell gpm graphite gstreamer gtk gtk3 gui iconv icu introspection io-uring ipv6 iwd jack jpeg jpeg2k keyring lcms libglvnd libnotify libtirpc libzfs lm_sensors lto lv2 mad mng mp3 mp4 mpeg multilib native-headset nautilus ncat ncurses networkmanager nfs nls ntfs nvenc nvidia nvme ogg opencl opengl openmp opus p2p pam pango pcre pdf pipewire png policykit ppds pulseaudio qt5 raw readline screencast sdl sdl2 seccomp smp sna sound sound-server spell spice ssl startup-notification svg synaptics sysprof system-llvm systemd test-rust thunderbolt tiff tracker truetype udev udisks uefi unicode upower usb usbredir user-session vaapi vala vdpau virgl vlc vorbis vulkan wayland webdav wireguard wxwidgets x264 x265 xattr xcb xft xml xv xvid zeroconf zfs zlib zstd" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="libinput wacom" KERNEL="linux" L10N="en ru" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31 ruby32" VIDEO_CARDS="intel i965 iris nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Comment 2 fosco 2024-06-03 12:46:41 UTC
Not an actual solution to the point but a mere work-around.
In a similar configuration (default MAKEOPTS and RAM equipment) I created a custom env.
That env has the notmpfs characteristics (so saving same actual space, since my emerge default is to mount TMPDIR on tmpfs)

PORTAGE_TMPDIR="/var/tmp/notmpfs"

and redefine 
MAKEOPTS to -j1

for safety I also provided the env with:
EMERGE_DEFAULT_OPTS="--jobs 1"
(but it seems not to influence emerge that much, anyway it did not break it)

By listing sci-libs/vtk in the corresponding package.env got a straight-forward maintenance of sci-libs/vtk and adapt the check of available RAM to less than 8GB.

Some minor RAM consumption may (not sure and not tested) be pursued by some setting of NVCC flags (see:
https://bugs.gentoo.org/show_bug.cgi?id=930031
https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html
)
Comment 3 Miroslav Šulc gentoo-dev 2024-06-07 11:19:31 UTC
i can confirm this. i set the env for vtk to use MAKEOPTS=-j1 even though i have 64GB of ram to prevent oom.
Comment 4 Paul Zander 2024-06-13 18:33:26 UTC
Can you tell me where exactly it OOMs? USE-Flags and other details, like the gpu would also help. VTK is huge and pin-pointing the exact issue blindly is impossible. 

For vtkm a job pool is used, we might increase the required memory size for that.
Comment 5 Anton Gubarkov 2024-06-14 07:26:34 UTC
I'm rebuilding vtk with MAKEOPTS=-j6

[ebuild   R    ] sci-libs/vtk-9.3.0:0/9.3::gentoo  USE="boost cuda ffmpeg freetype logging openmp qt6 rendering sdl threads views vtkm -all-modules (-debug) -doc -examples -gdal -gles2-only -imaging -java -las -mpi -mysql -odbc -opencascade -openvdb -pdal -postgres -python -qt5 -tbb -test -tk -web" PYTHON_SINGLE_TARGET="python3_12 -python3_10 -python3_11" VIDEO_CARDS="nvidia" 0 KiB

I'll post the log and also extract the OOM messages from the journal.
Comment 6 Anton Gubarkov 2024-06-14 12:35:37 UTC
Created attachment 895818 [details]
build log and env

This is build log and environment.
The OOM message is too big, will be in a separate file
Comment 7 Anton Gubarkov 2024-06-14 12:36:42 UTC
Created attachment 895819 [details]
OOM message
Comment 8 Anton Gubarkov 2024-06-14 12:39:33 UTC
(In reply to Paul Zander from comment #4)

> 
> For vtkm a job pool is used, we might increase the required memory size for
> that.

I'm not sure what is the average system used to build vtk... I wouldn't be happy if the package declares over 64GB of RAM + SWAP requirement...  Can we serialize the compilation of the most memory-hungry parts not to allow them to run in parallel?
Comment 9 Paul Zander 2024-06-14 16:27:44 UTC
Increasing the "required memory size" would mean more memory is required per started job. So overall less jobs in parallel.