Bugzilla Bug 215291

On my ~amd64 system I am running out of 1.5G memory and 2G swap upon
compilation of app-office/openoffice-2.4.0 when making qnametostr.obj. Trying
to compile for ~x86 (in a clean 32-bit chroot with KDE 3.5.9) resulted in about
900M memory usage at this exact stage, but the package still compiled fine.

Reproducible: Always

Steps to Reproduce:




# emerge --info
Portage 2.1.4.4 (default-linux/amd64/2007.0/desktop, gcc-4.2.3, glibc-2.7-r2,
2.6.24-gentoo-r3-alchark-u3s x86_64)                                            
=================================================================               
System uname: 2.6.24-gentoo-r3-alchark-u3s x86_64 Intel(R) Core(TM)2 Duo CPU
T7500 @ 2.20GHz                                                                 
Timestamp of tree: Sat, 29 Mar 2008 00:00:02 +0000                              
app-shells/bash:     3.2_p33                                                    
dev-java/java-config: 1.3.7, 2.1.5                                              
dev-lang/python:     2.5.1-r5                                                   
sys-apps/baselayout: 1.12.11.1                                                  
sys-apps/sandbox:    1.2.18.1-r2                                                
sys-devel/autoconf:  2.13, 2.61-r1                                              
sys-devel/automake:  1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1                
sys-devel/binutils:  2.18-r1                                                    
sys-devel/gcc-config: 1.4.0-r4                                                  
sys-devel/libtool:   1.5.26                                                     
virtual/os-headers:  2.6.24                                                     
ACCEPT_KEYWORDS="amd64 ~amd64"                                                  
CBUILD="x86_64-pc-linux-gnu"                                                    
CFLAGS="-O2 -march=nocona -pipe"                                                
CHOST="x86_64-pc-linux-gnu"                                                     
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config
/usr/kde/3.5/shutdown /usr/kde/svn/env /usr/kde/svn/share/config
/usr/kde/svn/shutdown /usr/share/config"       
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf
/etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c
/etc/udev/rules.d"                         
CXXFLAGS="-O2 -march=nocona -pipe"                                              
DISTDIR="/usr/portage/distfiles"                                                
FEATURES="distlocks metadata-transfer parallel-fetch sandbox sfperms strict
unmerge-orphans userfetch"                                                      
GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles
http://distfiles.gentoo.org"   
LANG="ru_RU.UTF-8"                                                              
LC_ALL="ru_RU.UTF-8"                                                            
LINGUAS="ru en"                                                                 
MAKEOPTS="-j4"                                                                  
PKGDIR="/usr/portage/packages"                                                  
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --stats --timeout=180 --exclude=/distfiles
--exclude=/local --exclude=/packages"                                           
PORTAGE_TMPDIR="/var/tmp"                                                       
PORTDIR="/usr/portage"                                                          
PORTDIR_OVERLAY="/usr/portage/local/layman/kde /usr/portage/local"              
SYNC="rsync://mirror.yandex.ru/gentoo-portage"                                  
USE="X a52 aac acl acpi alsa amd64 amr amrnb amrwb archive avahi berkdb
bluetooth bzip2 cairo cdparanoia cdr chm cli cracklib crypt cups curl dbus dhcp
djvu dri dts dvd dvdr dvdread emboss encode evo exif fam fbcon ffmpeg firefox
flac foomaticdb fortran ftp gdbm gif gimp gnutls gpm gs gstreamer gtk hal
htmlhandbook iconv id3tag ieee1394 imagemagick ipv6 ipw3945 isdnlog javascript
jingle jpeg kde kerberos lame latex lcms ldap lzo mad matroska midi mikmod mmx
mng mp3 mp4 mpeg mudflap musepack ncurses networkmanager nls nptlnptlonly
nsplugin ogg opengl openmp oscar oss pam pcre pdf perl png ppds pppd python
qt-copy qt3 qt3support qt4 quicktime rar readline reflection samba scanner sdl
session smpspell spl sse sse2 ssl ssse3 svg symlink taglib tcpd tetex theora
tiff truetype unicodeusb v4l v4l2 vnc vorbis wavpack wifi wmf x264 xcomposite
xine xml xorg xv xvid xvmc zeroconf zip zlib" ALSA_CARDS="hda-intel"
ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file
hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route
share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias
authn_anon authn_dbm authn_default authn_file authz_dbm authz_default
authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs
dav_lock deflate dir disk_cache env expires ext_filter file_cache filter
headers include info log_config logio mem_cache mime mime_magic negotiation
rewrite setenvif speling status unique_id userdir usertrack vhost_alias"
ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux"
LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses
text" LINGUAS="ru en" USERLAND="GNU" VIDEO_CARDS="nvidia i810"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS,
PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Created an attachment (id=147610) [details]
Compressed build log

Memory usage is normal everywhere except for the last 50 lines.

I've managed to build it with 2GB of RAM and 4GB of swap - took several minutes
due to heavy swap usage, but it compiled.

(In reply to comment #2)
> I've managed to build it with 2GB of RAM and 4GB of swap - took several minutes
> due to heavy swap usage, but it compiled.
> 

Did you compile for ~amd64 or for ~x86? For me it take more than six (!) hours
with a SATA HDD for swap, compiling for ~amd64. Then I got several 'out of
memory' messages in syslog before the system restarted without completing
compilation.

Update:
It did not really restart, as it turned out. Instead, kdeinit4 was killed, thus
killing the Konsole window and the compilation process inside it and leaving me
with a KDM prompt. To prevent further questions: I had scrollback limited to
1000 lines in Konsole, exactly to prevent it from taking up my whole memory, so
it is not the culprit.

(In reply to comment #3)
> Did you compile for ~amd64 or for ~x86? For me it take more than six (!) hours
> with a SATA HDD for swap, compiling for ~amd64. Then I got several 'out of
> memory' messages in syslog before the system restarted without completing
> compilation.

~amd64 - I managed to build it due to huge (4GB) swap.

Well, I guess I'll try to loopmount some extra gigabytes of swap, but certainly
this is not normal. No other build, openoffice included, lacked memory on my
machine.

Me too - amd64
1gb RAM and 1gb swap.
Openoffice 2.3.1 compiles fine. 

With gcc-4.3.0 qnametostr.cxx uses 140megs.
Well, won't help you guys, though. :(

# g++ -fmem-report -Wreturn-type -fmessage-length=0 -c -O0   -I. 
-I../../unxlngx6.pro/inc/resourcemodel -I../inc -I../../inc/pch -I../../inc
-I../../unx/inc -I../../unxlngx6.pro/inc -I.
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/stl
-I/mnt/data/tmp/portage/app-office/openof    out << "<theid
name=\"rtf:rgbrc\">20001</theid>" << endl;
fice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/external
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solenv/unxlngx6/inc
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solenv/inc
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/res
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/stl
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solenv/inc/Xp31
-I/opt/sun-jdk-1.6.0.04/include -I/opt/sun-jdk-1.6.0.04/include/linux
-I/opt/sun-jdk-1.6.0.04/include/native_threads/include -Idefault_x_includes    
-I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/offuh
-I. -I../../res -I. -pipe -O2 -mtune=core2 -march=core2 -pipe
-fomit-frame-pointer -fvisibility-inlines-hidden -w -Wno-ctor-dtor-privacy
-fno-use-cxa-atexit -fvisibility-inlines-hidden -Wall -Wextra -Wendif-labels
-Wshadow -Wno-ctor-dtor-privacy     -Wno-non-virtual-dtor   -fpic -DLINUX -DUNX
-DVCL -DGCC -DC341 -DX86_64 -DCVER=C341 -DNPTL -DGLIBC=2 -DX86_64 -D_PTHREADS
-D_REENTRANT -DNEW_SOLAR -D_USE_NAMESPACE=1 -DSTLPORT_VERSION=400
-DHAVE_GCC_VISIBILITY_FEATURE -D__DMAKE -DUNIX -DCPPU_ENV=gcc3
-DGXX_INCLUDE_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.0/include/g++-v4
-DSUPD=680 -DPRODUCT -DNDEBUG -DPRODUCT_FULL -DOSL_DEBUG_LEVEL=0 -DGSTREAMER
-DCUI -DSOLAR_JAVA -DOOH680=OOH680   -DWRITERFILTER_DLLIMPLEMENTATION
-DWRITERFILTER_DLLIMPLEMENTATION -DSHAREDLIB -D_DLL_   -fexceptions
-fno-enforce-eh-specs -DEXCEPTIONS_ON  -o ../../unxlngx6.pro/slo/qnametostr.o
../../unxlngx6.pro/misc/qnametostr.cxx
...
Total        140M        122M       1846k
...

Forgot to mention that there is a patch in ooo-build that adds '-O0' to
qnametostr.cxx compile which the ARCH_FLAGS overwrite later on for this reason
I guess. See the compile line.

Yuppeeee, I have just passed through this vicious stage by the means of
mounting extra 4GB of swap. An advice for those who will have to do the same:
try not to split your resulting swap space into several parts across the same
HDD to reduce stylus movements and thus seek times. So, if you already have,
say, a 2GB swap partition and want to add a 4GB swap file on the same physical
drive, it is better then to disable the 2GB partition, you will save lots of
time.

NB: In my case total memory usage has been slightly below 4GB (running also
Firefox, KDE4 and Konsole). Compilation of this particular module alone took
about 3-4 hours, with RES memory consumption in between 1g and 1.4g.

I was able to successfully build for ~amd64 with only 2GB RAM and 2.5GB swap.
It did take something like 5 hours (Core 2 Duo T7300, 2.0 GHz), but I had
Firefox, Thunderbird, and Pidgin open the whole time too.

*** Bug 215539 has been marked as a duplicate of this bug. ***

*** Bug 215586 has been marked as a duplicate of this bug. ***

Workaround is to build without -O* in CFLAGS. Most files will still be compiled
with -O2, but some with -O0 (including qnametostr.cxx). Even if building with
-O2 succeeds, the result may be broken, because I get lots of strict-aliasing
warnings for files intended to be compiled with -O0 (they get no
-fno-strict-aliasing as optimized files do). So maybe the ebuild should filter
-O* instead of doing |replace-flags "-O?" "-O2"|.

*** Bug 216205 has been marked as a duplicate of this bug. ***

(In reply to comment #14)
> Workaround is to build without -O* in CFLAGS. Most files will still be compiled
> with -O2, but some with -O0 (including qnametostr.cxx). Even if building with
> -O2 succeeds, the result may be broken, because I get lots of strict-aliasing
> warnings for files intended to be compiled with -O0 (they get no
> -fno-strict-aliasing as optimized files do). So maybe the ebuild should filter
> -O* instead of doing |replace-flags "-O?" "-O2"|.
> 
If we replace -O? with -O0 or -O1 openoffice builds fine with gcc-4.2.3 with
512M ram

(In reply to comment #16)
> If we replace -O? with -O0 or -O1 openoffice builds fine with gcc-4.2.3 with
> 512M ram

If you replace it, then it will be built without optimization (or only -O1). If
you delete it, then it will be built with -O2, except for some problematic
files.

By the way, I confirm that it compiles completely without this issue if one
uses GCC 4.3.0 (presumably due to reworked optimization algorithms in this
release). I did not even notice the moment when qnametostr.obj was compiled
(using CFLAGS="-O2 -march=core2 -pipe"). However, a patch for 'strdup not
declared in this scope' is needed, like the one provided in bug 216205.

*** Bug 217685 has been marked as a duplicate of this bug. ***

Hmm, not quite sure how to proceed here, guess forcing everyone on -01 is not
going to make us very popular ;) So open for good ideas, patches, whatever ;)

(In reply to comment #17)
> If you delete it, then it will be built with -O2, except for some problematic files.

+1

So that silent miscompiles and other issues which are already catched by
openoffice build system take effect.

(In reply to comment #20)
> Hmm, not quite sure how to proceed here, guess forcing everyone on -01 is not
> going to make us very popular ;) So open for good ideas, patches, whatever ;)

As I already said, replace

  replace-flags "-O?" "-O2"

by

  filter-flags "-O[s2-9]"

You're already forcing everyone to -O2, and that's the default without flags
too (for most files). The ARCH_FLAGS aren't thought for optimization flags, and
if they include -Os, -O2 or higher, then -fno-strict-aliasing must be added
too.

A better solution would be to patch CFLAGSOPT in the makefiles to contain -O*
from user CFLAGS (especially to allow -Os), but that's not easy because they
are still packed when the build starts. Maybe CFLAGSNOOPT could be set to -O1
without causing real problems.

Probably we could have USE=custom-cflags to allow the adventurous ones to try
and build the package with -O2, -Os or whatever, and have all the others rely
on defaults shipped with the package, as we have in case of MPlayer?

How could this become 'stable' on amd64, while nobody with more than 3 GB RAM
is able to even build it?

(In reply to comment #22)
> As I already said, replace
> 
>   replace-flags "-O?" "-O2"
> 
> by
> 
>   filter-flags "-O[s2-9]"

I've done that now, thanks to everyone giving advice on this. New build
finished here normal levels of memory usage, again ;)

Closing