Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 626134 - dev-lang/rust-1.19.0 could check for required RAM if -ggdb in CFLAGS
Summary: dev-lang/rust-1.19.0 could check for required RAM if -ggdb in CFLAGS
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Rust Project
URL:
Whiteboard:
Keywords:
: 626288 (view as bug list)
Depends on:
Blocks: insufficient-RAM
  Show dependency tree
 
Reported: 2017-07-25 11:27 UTC by Marien Zwart
Modified: 2024-02-14 10:26 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marien Zwart 2017-07-25 11:27:40 UTC
Building rust-1.19.0 with -ggdb in CFLAGS on a machine with a decent number of cores/hyperthreads requires quite a bit of RAM. Towards the end of building the stage0 compiler, it links a few binaries that take about 2.5GB per binary. This part of the build ignores -j from MAKEOPTS (bug 626080), so on a cpu with 8 hyperthreads it consumes >16GB of RAM.

No build issues with -ggdb dropped from CFLAGS and CXXFLAGS.

Maybe the ebuild could use check-reqs.eclass to check for this up front? Prior art: chromium and pypy ebuilds.

This would be annoying since the amount of memory consumed depends on the build's parallelism (so maybe fix bug 626080 first, so you can use something like makeopts_jobs). But running out of memory is an unpleasant failure mode, so...

Portage 2.3.6 (python 2.7.13-final-0, default/linux/amd64/13.0/desktop/gnome/systemd, gcc-6.3.0, glibc-2.24-r3, 4.12.3-gentoo-m22 x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.12.3-gentoo-m22-x86_64-Intel-R-_Core-TM-_i7-4771_CPU_@_3.50GHz-with-gentoo-2.4.1
KiB Mem:    16376584 total,   6896608 free
KiB Swap:    2097148 total,   1981620 free
Timestamp of repository gentoo: Tue, 25 Jul 2017 08:45:01 +0000
sh bash 4.4_p12
ld GNU ld (Gentoo 2.28 p1.2) 2.28
ccache version 3.3.4 [disabled]
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.2::gentoo
dev-lang/python:          2.7.13::gentoo, 3.5.3::gentoo, 3.6.1-r1::gentoo
dev-util/ccache:          3.3.4::gentoo
dev-util/cmake:           3.9.0::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1::gentoo
sys-apps/sandbox:         2.10-r4::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r3::gentoo
sys-devel/automake:       1.11.6-r2::gentoo, 1.15.1::gentoo
sys-devel/binutils:       2.28-r2::gentoo
sys-devel/gcc:            6.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r4::gentoo
sys-devel/make:           4.2.1-r1::gentoo
sys-kernel/linux-headers: 4.10::gentoo (virtual/os-headers)
sys-libs/glibc:           2.24-r3::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

marienz
    location: /usr/local/portage/private
    masters: gentoo

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="@FREE @BINARY-REDISTRIBUTABLE fping freedist google-chrome unRAR repoze MSttfEULA linux-firmware"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-ggdb -O2 -pipe -march=native"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/chromium/policies/managed/chrome-gnome-shell.json /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/opt/chrome/policies/managed/chrome-gnome-shell.json /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-ggdb -O2 -pipe -march=native"
DISTDIR="/var/cache/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs candy collision-protect compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms sign splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirror.internode.on.net/pub/gentoo/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,--hash-style=gnu -Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j8"
PKGDIR="/var/tmp/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi adns alsa amd64 avahi branding btrfs bzip2 cairo canberra caps cdda cdio cdr clang cli clutter colord corefonts crypt cxx dbus device-mapper distinct-l doc dos dot dri dri3 drm dts dvd dvdr dvi eds efi egl emacs emboss encode equalizer evo exif expat ffmpeg firefox flac fontconfig fortran fuse g3dvl gbm gegl gflags ghcbootstrap gif git glamor gles gles2 gmp gnome gnome-keyring gnome-online-accounts gnuefi go google gpg gstreamer gtk gtk3 gtkstyle http iconv icu idn imap inotify introspection ipv6 irc jpeg lame latex lcms libass libcaca libffi libidn2 libinput libkms libnotify libressl libsecret libvisual llvm llvm-shared-libs lua lzma mad maildir minizip mng modules mp3 mp4 mpeg mtp multilib multimedia ncurses nfsidmap nls nptl numpy objc ogg opengl openmp opus pam pango pch pdf playlist png policykit postscript ppds prelink preview-latex projectm pulseaudio python python3 qml readline realtime s3tc sandbox schroedinger seccomp semantic-desktop session sip smtp speex spell spice sqlite ssl startup-notification svc svg symlink system-harfbuzz system-icu system-jflex system-jpeg system-jsoncpp systemd sysv-utils theora tiff tokyocabinet toolkit-scroll-bars tools truetype udev udisks udisks2 uefi unicode unicode3 unwind upower urwid usb usbredir vaapi vala valgrind vorbis vpx wayland webkit webp widevine widgets wxwidgets x264 xattr xcb xcomposite xetex xft xinerama xml xmp xv xvmc xwayland zeroconf zlib zsh-completion" ABI_X86="64" ALSA_CARDS="hda_intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" CURL_SSL="nss" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-GB en-US nl fy" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB en_US nl fy fy_NL" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5 python3_6 pypy pypy3" QEMU_SOFTMMU_TARGETS="x86_64" RUBY_TARGETS="ruby22" SANE_BACKENDS="u12" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

dev-lang/rust-1.16.0::gentoo was built with the following:
USE="clang doc -debug -libcxx" ABI_X86="(64)"
Comment 1 Mike Auty (RETIRED) gentoo-dev 2017-08-02 19:54:52 UTC
*** Bug 626288 has been marked as a duplicate of this bug. ***
Comment 2 Mike Auty (RETIRED) gentoo-dev 2017-08-02 20:00:45 UTC
As noted in bug 626288, it should also check for disk space in /var/tmp/portage as well, using something like the following (mercilessly ripped from the Chromium ebuilds) in the EAPI-6 pre_build_checks phase (with the various values changed in accordance with rust's needs):

        CHECKREQS_MEMORY="3G"
        CHECKREQS_DISK_BUILD="5G"
        eshopts_push -s extglob
        if is-flagq '-g?(gdb)?([1-9])'; then
                CHECKREQS_DISK_BUILD="25G"
                if ! use component-build; then
                        CHECKREQS_MEMORY="16G"
                fi
        fi
        eshopts_pop
        check-reqs_pkg_setup
Comment 3 Larry the Git Cow gentoo-dev 2018-11-19 16:42:16 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c0718dbc84ed51edd184446fc0d9cde7195e25a0

commit c0718dbc84ed51edd184446fc0d9cde7195e25a0
Author:     Dirkjan Ochtman <djc@gentoo.org>
AuthorDate: 2018-11-19 16:41:47 +0000
Commit:     Dirkjan Ochtman <djc@gentoo.org>
CommitDate: 2018-11-19 16:42:01 +0000

    dev-lang/rust: check reqs, ewarn about cargo symlink
    
    Fixes: https://bugs.gentoo.org/626134
    Fixes: https://bugs.gentoo.org/626742
    Fixes: https://bugs.gentoo.org/663354
    Fixes: https://bugs.gentoo.org/671182
    Signed-off-by: Dirkjan Ochtman <djc@gentoo.org>
    Package-Manager: Portage-2.3.51, Repoman-2.3.11

 dev-lang/rust/rust-1.30.1-r1.ebuild | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
Comment 4 Ulrich Müller gentoo-dev 2018-12-22 14:39:13 UTC
Reopening, because the test that is currently in the ebuild will be triggered on false positives:

    if is-flagq '-g?(gdb)?([1-9])'; then
        CHECKREQS_DISK_BUILD="10G"
        CHECKREQS_MEMORY="16G"
    fi

For example, I have -ggdb in my CFLAGS, but override it by adding -g0 for certain packages (including rust) via package.env. The above test will be triggered nevertheless.
Comment 5 Till Schäfer 2019-01-30 13:26:03 UTC
The pre-emerge check is currently triggered for me, even though I have a machine with 16GiB RAM (gddb enabled). Also my laptop with 4GiB RAM and with gddb disabled is running against the pre_build_checks-wall. This is most likely because the internal intel grafics take a few megabytes for their video RAM. 

I think this is the reason, why most ebuild check against "desired GiB RAM - 1" in CHECKREQS_MEMORY. Thus, I would propose to check against 3GiB for non-gddb builds and 15 GiB for gddb builds.
Comment 6 Larry the Git Cow gentoo-dev 2019-05-13 13:01:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=baefac48f4aff148b8c92531380fead020abfb98

commit baefac48f4aff148b8c92531380fead020abfb98
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2019-05-13 13:00:15 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2019-05-13 13:01:13 +0000

    dev-lang/rust: drop memory requirement check
    
    CHECKREQS_MEMORY should only be used when there's a known
    minimum memory requirement to run/build packages with MAKEOPTS=-j1.
    
    If user builds package in parallel it is expected that the build process
    will require more memory. However, in most cases we don't know how much
    memory is required because we don't know how much memory an additional
    job could use at maximum (and if we would know, just add distcc, which
    will move memory requirement down to distcc host...).
    
    Otherwise we would cause unnecessarily problems for systems with low
    memory but still able to build the package.
    
    Bug: https://bugs.gentoo.org/626134
    Package-Manager: Portage-2.3.66, Repoman-2.3.12
    Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>

 dev-lang/rust/rust-1.30.1-r1.ebuild | 2 --
 dev-lang/rust/rust-1.31.1.ebuild    | 2 --
 dev-lang/rust/rust-1.32.0.ebuild    | 2 --
 dev-lang/rust/rust-1.33.0.ebuild    | 2 --
 dev-lang/rust/rust-1.34.0-r2.ebuild | 2 --
 dev-lang/rust/rust-1.34.1.ebuild    | 2 --
 6 files changed, 12 deletions(-)
Comment 7 Michael Hofmann 2020-07-07 21:58:02 UTC
I wonder what's going on here.

We had a similar case in the German Gentoo forums today. A user compiled "clang" with "-ggdb" in CFLAGS. The linker allocated huge amounts of memory and was finally killed by the OOM killer.

- https://forums.gentoo.org/viewtopic-t-1115824.html

The issue was reported to GCC developers last year:

- https://gcc.gnu.org/legacy-ml/gcc/2019-04/msg00270.html
- https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90273

It seems that GCC developers developed a patch last year. This patch is included in GCC 9.3 (I verified it), but the linker still needs huge of memory if CFLAGS contains "-ggdb", at least for "clang".