| Summary: | as: out of memory allocating 288230376151711728 bytes after a total of 272400 bytes | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Jocelyn Mayer <l_indien> |
| Component: | Hardened | Assignee: | The Gentoo Linux Hardened Team <hardened> |
| Status: | RESOLVED INVALID | ||
| Severity: | normal | ||
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | AMD64 | ||
| OS: | Linux | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=538026 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
Jocelyn Mayer
2015-01-28 07:52:06 UTC
Usually you install more memory to fix problems with programs that complain about memory problems. But maybe there is a huge memory leak here - who knows? this does not look like a memory leak: as tells us it tries to allocate about 288000 TB of memory at once. And the failed allocation size do not seem to be an "innocent" one, feels like a rounded negative size (-16 ?):
> printf "%016x\n" 288230376151711728
03fffffffffffff0
IMHO, for the Gentoo distribution, a bug in a stable build toolchain is a major one as the whole distribution is to be built from sources.
(In reply to Jocelyn Mayer from comment #2) > this does not look like a memory leak: as tells us it tries to allocate > about 288000 TB of memory at once. And the failed allocation size do not > seem to be an "innocent" one, feels like a rounded negative size (-16 ?): > > printf "%016x\n" 288230376151711728 > 03fffffffffffff0 > > IMHO, for the Gentoo distribution, a bug in a stable build toolchain is a > major one as the whole distribution is to be built from sources. I'm not sure its a bug yet. This appears related to your bug #538026. I'm suspecting something is seriously broken on your system. Are you hitting both bugs on the same box? (In reply to Anthony Basile from comment #3) > (In reply to Jocelyn Mayer from comment #2) > > this does not look like a memory leak: as tells us it tries to allocate > > about 288000 TB of memory at once. And the failed allocation size do not > > seem to be an "innocent" one, feels like a rounded negative size (-16 ?): > > > printf "%016x\n" 288230376151711728 > > 03fffffffffffff0 > > > > IMHO, for the Gentoo distribution, a bug in a stable build toolchain is a > > major one as the whole distribution is to be built from sources. > > I'm not sure its a bug yet. This appears related to your bug #538026. I'm > suspecting something is seriously broken on your system. Are you hitting > both bugs on the same box? Yes, the 2 problems appear on the same box. (In reply to Jocelyn Mayer from comment #4) > (In reply to Anthony Basile from comment #3) > > (In reply to Jocelyn Mayer from comment #2) > > > this does not look like a memory leak: as tells us it tries to allocate > > > about 288000 TB of memory at once. And the failed allocation size do not > > > seem to be an "innocent" one, feels like a rounded negative size (-16 ?): > > > > printf "%016x\n" 288230376151711728 > > > 03fffffffffffff0 > > > > > > IMHO, for the Gentoo distribution, a bug in a stable build toolchain is a > > > major one as the whole distribution is to be built from sources. > > > > I'm not sure its a bug yet. This appears related to your bug #538026. I'm > > suspecting something is seriously broken on your system. Are you hitting > > both bugs on the same box? > > Yes, the 2 problems appear on the same box. Having made more investigations, it happens that bug #538026 can be reproduced on other boxes with different configuration (not hardened gentoo) while this one only shows up on my hardened gentoo server. Then, I'm quite sure now this bug is not related to #538026. In case it's useful, here are USE flags differences between the faulty box and the one(s) which do not show the bug:
Faulty box, running hardened gentoo configuration:
# equery uses sys-devel/binutils
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
* Found these USE flags for sys-devel/binutils-2.24-r3:
U I
- - cxx : Build support for C++ (bindings, extra libraries, code
generation, ...)
- - multitarget : Adds support to binutils for cross compiling (does not work
with gas)
- - nls : Add Native Language Support (using gettext - GNU locale
utilities)
- - static-libs : Build static versions of dynamic libraries as well
- - test : Workaround to pull in packages needed to run with
FEATURES=test. Portage-2.1.2 handles this internally, so
don't set it in make.conf/package.use anymore
- - vanilla : Do not add extra patches which change default behaviour; DO
NOT USE THIS ON A GLOBAL SCALE as the severity of the
meaning changes drastically
+ + zlib : Add support for zlib (de)compression
# equery uses sys-devel/gcc
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
* Found these USE flags for sys-devel/gcc-4.8.3:
U I
- - awt : Useful only when building GCJ, this enables Abstract
Window Toolkit (AWT) peer support on top of GTK+
+ + cxx : Build support for C++ (bindings, extra libraries, code
generation, ...)
- - doc : Add extra documentation (API, Javadoc, etc). It is
recommended to enable per package instead of globally
- - fortran : Add support for fortran
- - gcj : Enable building with gcj (The GNU Compiler for the
Javatm Programming Language)
- - go : Build the GCC Go language frontend.
- - graphite : Add support for the framework for loop optimizations
based on a polyhedral intermediate representation
+ + hardened : Activate default security enhancements for toolchain
(gcc, glibc, binutils)
- - mudflap : Add support for mudflap, a pointer use checking library
- - nls : Add Native Language Support (using gettext - GNU locale
utilities)
- - nopie : Disable PIE support (NOT FOR GENERAL USE)
- - nossp : Disable SSP support (NOT FOR GENERAL USE)
+ + nptl : Enable support for Native POSIX Threads Library, the new
threading module (requires linux-2.6 or better usually)
- - objc : Build support for the Objective C code language
- - objc++ : Build support for the Objective C++ language
- - objc-gc : Build support for the Objective C code language Garbage
Collector
- - openmp : Build support for the OpenMP (support parallel
computing), requires >=sys-devel/gcc-4.2 built with
USE="openmp"
- - regression-test : Run the testsuite and install the results (requires
FEATURES=test)
- - sanitize : Build support various sanitizer functions
(ASAN/TSAN/etc...)
- - vanilla : Do not add extra patches which change default behaviour;
DO NOT USE THIS ON A GLOBAL SCALE as the severity of the
meaning changes drastically
# equery uses sys-libs/glibc
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
* Found these USE flags for sys-libs/glibc-2.19-r1:
U I
- - debug : When USE=hardened, allow fortify/stack violations to dump core
(SIGABRT) and not kill self (SIGKILL)
+ + gd : build memusage and memusagestat tools
+ + hardened : Activate default security enhancements for toolchain (gcc,
glibc, binutils)
- - nscd : Build, and enable support for, the Name Service Cache Daemon
- - profile : Add support for software performance analysis (will likely
vary from ebuild to ebuild)
- - suid : Make internal pt_chown helper setuid -- not needed if using
Linux and have /dev/pts mounted with gid=5
- - systemtap : enable systemtap static probe points
- - vanilla : Do not add extra patches which change default behaviour; DO
NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning
changes drastically
Box where the kernel and nss library build without any problem:
# equery uses sys-devel/binutils
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
* Found these USE flags for sys-devel/binutils-2.24-r3:
U I
+ + cxx : Build support for C++ (bindings, extra libraries, code
generation, ...)
+ + multitarget : Adds support to binutils for cross compiling (does not work
with gas)
+ + nls : Add Native Language Support (using gettext - GNU locale
utilities)
+ + static-libs : Build static versions of dynamic libraries as well
- - test : Workaround to pull in packages needed to run with
FEATURES=test. Portage-2.1.2 handles this internally, so
don't set it in make.conf/package.use anymore
- - vanilla : Do not add extra patches which change default behaviour; DO
NOT USE THIS ON A GLOBAL SCALE as the severity of the
meaning changes drastically
+ + zlib : Add support for zlib (de)compression
# equery uses sys-devel/gcc
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
* Found these USE flags for sys-devel/gcc-4.8.3:
U I
- - awt : Useful only when building GCJ, this enables Abstract
Window Toolkit (AWT) peer support on top of GTK+
+ + cxx : Build support for C++ (bindings, extra libraries, code
generation, ...)
+ + doc : Add extra documentation (API, Javadoc, etc). It is
recommended to enable per package instead of globally
+ + fortran : Add support for fortran
- - gcj : Enable building with gcj (The GNU Compiler for the
Javatm Programming Language)
- - go : Build the GCC Go language frontend.
+ + graphite : Add support for the framework for loop optimizations
based on a polyhedral intermediate representation
- - mudflap : Add support for mudflap, a pointer use checking library
+ + nls : Add Native Language Support (using gettext - GNU locale
utilities)
- - nopie : Disable PIE support (NOT FOR GENERAL USE)
- - nossp : Disable SSP support (NOT FOR GENERAL USE)
+ + nptl : Enable support for Native POSIX Threads Library, the new
threading module (requires linux-2.6 or better usually)
+ + objc : Build support for the Objective C code language
+ + objc++ : Build support for the Objective C++ language
- - objc-gc : Build support for the Objective C code language Garbage
Collector
+ + openmp : Build support for the OpenMP (support parallel
computing), requires >=sys-devel/gcc-4.2 built with
USE="openmp"
- - regression-test : Run the testsuite and install the results (requires
FEATURES=test)
+ + sanitize : Build support various sanitizer functions
(ASAN/TSAN/etc...)
- - vanilla : Do not add extra patches which change default behaviour;
DO NOT USE THIS ON A GLOBAL SCALE as the severity of the
meaning changes drastically
# equery uses sys-libs/glibc
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
* Found these USE flags for sys-libs/glibc-2.19-r1:
U I
- - debug : When USE=hardened, allow fortify/stack violations to dump core
(SIGABRT) and not kill self (SIGKILL)
+ + gd : build memusage and memusagestat tools
- - nscd : Build, and enable support for, the Name Service Cache Daemon
- - profile : Add support for software performance analysis (will likely
vary from ebuild to ebuild)
- - suid : Make internal pt_chown helper setuid -- not needed if using
Linux and have /dev/pts mounted with gid=5
- - systemtap : enable systemtap static probe points
- - vanilla : Do not add extra patches which change default behaviour; DO
NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning
changes drastically
(In reply to Jocelyn Mayer from comment #0) > as from binutils 2.23.2 and 2.24 runs out of memory while assembling amd64 > arch dependant linux kernel cryptographic helpers. > This also happens when trying to compile dev-libs/nss package. > > Reproducible: Always This one, I can't reproduce. I'm not sure how you're getting into this state. I gather from your previous comments that this is the only box where you're getting the strange memory allocation. (In reply to Anthony Basile from comment #7) > (In reply to Jocelyn Mayer from comment #0) > > as from binutils 2.23.2 and 2.24 runs out of memory while assembling amd64 > > arch dependant linux kernel cryptographic helpers. > > This also happens when trying to compile dev-libs/nss package. > > > > Reproducible: Always > > This one, I can't reproduce. I'm not sure how you're getting into this > state. I gather from your previous comments that this is the only box where > you're getting the strange memory allocation. Still trying to reproduce --- can I have your kernel config file so I'm sure I'm doing the same thing here as you are there. (In reply to Anthony Basile from comment #8) > (In reply to Anthony Basile from comment #7) > > (In reply to Jocelyn Mayer from comment #0) > > > as from binutils 2.23.2 and 2.24 runs out of memory while assembling amd64 > > > arch dependant linux kernel cryptographic helpers. > > > This also happens when trying to compile dev-libs/nss package. > > > > > > Reproducible: Always > > > > This one, I can't reproduce. I'm not sure how you're getting into this > > state. I gather from your previous comments that this is the only box where > > you're getting the strange memory allocation. > > Still trying to reproduce --- can I have your kernel config file so I'm sure > I'm doing the same thing here as you are there. Last ping before closing this need info. Reopen when you get more info. I finally found and solved this issue. The point was that CFLAGS with -march=amdfam10 is not correct anymore for this specific CPU (_X2_Dual_Core_Processor_BE-2300). It happens to be -march=k8-sse3 with recent gcc, missing SSE4a and ABM instruction sets; the difference in CPU options tuning is then -mno-popcnt -mno-abm -mno-lzcnt. It seems like this CPU implements (some of) the missing instructions but not giving the correct result. As a consequence, there was a problem in zlib which made as do weird things while compiling kernel and dev-libs/nss package. Recompiling zlib with -march=native solved the issue. Then I did recompile the whole system changing my CFLAGS from CFLAGS="-march=amdfam10 -O2 -funroll-loops -pipe -fexpensive-optimizations -fomit-frame-pointer" to CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" in order to be sure no other package is affected by this issue (doing emerge --verbose --empty @system ; emerge --verbose --empty @world) Thanks for your help. |
