Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 215291 - Insane memory usage on app-office/openoffice-2.4.0 compilation
Summary: Insane memory usage on app-office/openoffice-2.4.0 compilation
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: High normal with 2 votes (vote)
Assignee: Gentoo Office Team
URL:
Whiteboard:
Keywords:
: 215539 215586 217685 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-03-29 09:37 UTC by Alexey Charkov
Modified: 2008-10-21 00:28 UTC (History)
18 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Compressed build log (build.log.bz2,488.14 KB, application/x-bzip2)
2008-03-29 09:45 UTC, Alexey Charkov
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Charkov 2008-03-29 09:37:09 UTC
On my ~amd64 system I am running out of 1.5G memory and 2G swap upon compilation of app-office/openoffice-2.4.0 when making qnametostr.obj. Trying to compile for ~x86 (in a clean 32-bit chroot with KDE 3.5.9) resulted in about 900M memory usage at this exact stage, but the package still compiled fine.

Reproducible: Always

Steps to Reproduce:




# emerge --info
Portage 2.1.4.4 (default-linux/amd64/2007.0/desktop, gcc-4.2.3, glibc-2.7-r2, 2.6.24-gentoo-r3-alchark-u3s x86_64)                                                              
=================================================================                       
System uname: 2.6.24-gentoo-r3-alchark-u3s x86_64 Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz                                                                                    
Timestamp of tree: Sat, 29 Mar 2008 00:00:02 +0000                                      
app-shells/bash:     3.2_p33                                                            
dev-java/java-config: 1.3.7, 2.1.5                                                      
dev-lang/python:     2.5.1-r5                                                           
sys-apps/baselayout: 1.12.11.1                                                          
sys-apps/sandbox:    1.2.18.1-r2                                                        
sys-devel/autoconf:  2.13, 2.61-r1                                                      
sys-devel/automake:  1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1                        
sys-devel/binutils:  2.18-r1                                                            
sys-devel/gcc-config: 1.4.0-r4                                                          
sys-devel/libtool:   1.5.26                                                             
virtual/os-headers:  2.6.24                                                             
ACCEPT_KEYWORDS="amd64 ~amd64"                                                          
CBUILD="x86_64-pc-linux-gnu"                                                            
CFLAGS="-O2 -march=nocona -pipe"                                                        
CHOST="x86_64-pc-linux-gnu"                                                             
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/kde/svn/env /usr/kde/svn/share/config /usr/kde/svn/shutdown /usr/share/config"       
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d"                         
CXXFLAGS="-O2 -march=nocona -pipe"                                                      
DISTDIR="/usr/portage/distfiles"                                                        
FEATURES="distlocks metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch"                                                                          
GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles http://distfiles.gentoo.org"   
LANG="ru_RU.UTF-8"                                                                      
LC_ALL="ru_RU.UTF-8"                                                                    
LINGUAS="ru en"                                                                         
MAKEOPTS="-j4"                                                                          
PKGDIR="/usr/portage/packages"                                                          
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"                                                                          
PORTAGE_TMPDIR="/var/tmp"                                                               
PORTDIR="/usr/portage"                                                                  
PORTDIR_OVERLAY="/usr/portage/local/layman/kde /usr/portage/local"                      
SYNC="rsync://mirror.yandex.ru/gentoo-portage"                                          
USE="X a52 aac acl acpi alsa amd64 amr amrnb amrwb archive avahi berkdb bluetooth bzip2 cairo cdparanoia cdr chm cli cracklib crypt cups curl dbus dhcp djvu dri dts dvd dvdr dvdread emboss encode evo exif fam fbcon ffmpeg firefox flac foomaticdb fortran ftp gdbm gif gimp gnutls gpm gs gstreamer gtk hal htmlhandbook iconv id3tag ieee1394 imagemagick ipv6 ipw3945 isdnlog javascript jingle jpeg kde kerberos lame latex lcms ldap lzo mad matroska midi mikmod mmx mng mp3 mp4 mpeg mudflap musepack ncurses networkmanager nls nptlnptlonly nsplugin ogg opengl openmp oscar oss pam pcre pdf perl png ppds pppd python qt-copy qt3 qt3support qt4 quicktime rar readline reflection samba scanner sdl session smpspell spl sse sse2 ssl ssse3 svg symlink taglib tcpd tetex theora tiff truetype unicodeusb v4l v4l2 vnc vorbis wavpack wifi wmf x264 xcomposite xine xml xorg xv xvid xvmc zeroconf zip zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="ru en" USERLAND="GNU" VIDEO_CARDS="nvidia i810"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Alexey Charkov 2008-03-29 09:45:13 UTC
Created attachment 147610 [details]
Compressed build log

Memory usage is normal everywhere except for the last 50 lines.
Comment 2 Krzysztof Pawlik (RETIRED) gentoo-dev 2008-03-29 10:35:20 UTC
I've managed to build it with 2GB of RAM and 4GB of swap - took several minutes due to heavy swap usage, but it compiled.
Comment 3 Alexey Charkov 2008-03-29 12:27:31 UTC
(In reply to comment #2)
> I've managed to build it with 2GB of RAM and 4GB of swap - took several minutes
> due to heavy swap usage, but it compiled.
> 

Did you compile for ~amd64 or for ~x86? For me it take more than six (!) hours with a SATA HDD for swap, compiling for ~amd64. Then I got several 'out of memory' messages in syslog before the system restarted without completing compilation.
Comment 4 Alexey Charkov 2008-03-29 12:33:52 UTC
Update:
It did not really restart, as it turned out. Instead, kdeinit4 was killed, thus killing the Konsole window and the compilation process inside it and leaving me with a KDM prompt. To prevent further questions: I had scrollback limited to 1000 lines in Konsole, exactly to prevent it from taking up my whole memory, so it is not the culprit.
Comment 5 Krzysztof Pawlik (RETIRED) gentoo-dev 2008-03-29 12:34:42 UTC
(In reply to comment #3)
> Did you compile for ~amd64 or for ~x86? For me it take more than six (!) hours
> with a SATA HDD for swap, compiling for ~amd64. Then I got several 'out of
> memory' messages in syslog before the system restarted without completing
> compilation.

~amd64 - I managed to build it due to huge (4GB) swap.
Comment 6 Alexey Charkov 2008-03-29 14:03:45 UTC
Well, I guess I'll try to loopmount some extra gigabytes of swap, but certainly this is not normal. No other build, openoffice included, lacked memory on my machine.
Comment 7 Yury Zhuravlev 2008-03-29 17:54:21 UTC
Me too - amd64
1gb RAM and 1gb swap.
Openoffice 2.3.1 compiles fine. 
Comment 8 Hanno Zysik (geki) 2008-03-29 18:15:26 UTC
With gcc-4.3.0 qnametostr.cxx uses 140megs.
Well, won't help you guys, though. :(

# g++ -fmem-report -Wreturn-type -fmessage-length=0 -c -O0   -I.  -I../../unxlngx6.pro/inc/resourcemodel -I../inc -I../../inc/pch -I../../inc -I../../unx/inc -I../../unxlngx6.pro/inc -I. -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/stl -I/mnt/data/tmp/portage/app-office/openof    out << "<theid name=\"rtf:rgbrc\">20001</theid>" << endl;
fice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/external -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solenv/unxlngx6/inc -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solenv/inc -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/res -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/stl -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solenv/inc/Xp31 -I/opt/sun-jdk-1.6.0.04/include -I/opt/sun-jdk-1.6.0.04/include/linux -I/opt/sun-jdk-1.6.0.04/include/native_threads/include -Idefault_x_includes     -I/mnt/data/tmp/portage/app-office/openoffice-2.4.0_pre12/work/ooo-build/build/ooh680-m12/solver/680/unxlngx6.pro/inc/offuh -I. -I../../res -I. -pipe -O2 -mtune=core2 -march=core2 -pipe -fomit-frame-pointer -fvisibility-inlines-hidden -w -Wno-ctor-dtor-privacy -fno-use-cxa-atexit -fvisibility-inlines-hidden -Wall -Wextra -Wendif-labels -Wshadow -Wno-ctor-dtor-privacy     -Wno-non-virtual-dtor   -fpic -DLINUX -DUNX -DVCL -DGCC -DC341 -DX86_64 -DCVER=C341 -DNPTL -DGLIBC=2 -DX86_64 -D_PTHREADS -D_REENTRANT -DNEW_SOLAR -D_USE_NAMESPACE=1 -DSTLPORT_VERSION=400 -DHAVE_GCC_VISIBILITY_FEATURE -D__DMAKE -DUNIX -DCPPU_ENV=gcc3 -DGXX_INCLUDE_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.0/include/g++-v4 -DSUPD=680 -DPRODUCT -DNDEBUG -DPRODUCT_FULL -DOSL_DEBUG_LEVEL=0 -DGSTREAMER -DCUI -DSOLAR_JAVA -DOOH680=OOH680   -DWRITERFILTER_DLLIMPLEMENTATION -DWRITERFILTER_DLLIMPLEMENTATION -DSHAREDLIB -D_DLL_   -fexceptions -fno-enforce-eh-specs -DEXCEPTIONS_ON  -o ../../unxlngx6.pro/slo/qnametostr.o ../../unxlngx6.pro/misc/qnametostr.cxx
...
Total        140M        122M       1846k
...
Comment 9 Hanno Zysik (geki) 2008-03-29 18:17:58 UTC
Forgot to mention that there is a patch in ooo-build that adds '-O0' to qnametostr.cxx compile which the ARCH_FLAGS overwrite later on for this reason I guess. See the compile line.
Comment 10 Alexey Charkov 2008-03-29 19:07:06 UTC
Yuppeeee, I have just passed through this vicious stage by the means of mounting extra 4GB of swap. An advice for those who will have to do the same: try not to split your resulting swap space into several parts across the same HDD to reduce stylus movements and thus seek times. So, if you already have, say, a 2GB swap partition and want to add a 4GB swap file on the same physical drive, it is better then to disable the 2GB partition, you will save lots of time.

NB: In my case total memory usage has been slightly below 4GB (running also Firefox, KDE4 and Konsole). Compilation of this particular module alone took about 3-4 hours, with RES memory consumption in between 1g and 1.4g.
Comment 11 Randall Wald 2008-03-30 23:00:40 UTC
I was able to successfully build for ~amd64 with only 2GB RAM and 2.5GB swap. It did take something like 5 hours (Core 2 Duo T7300, 2.0 GHz), but I had Firefox, Thunderbird, and Pidgin open the whole time too.
Comment 12 Carsten Lohrke (RETIRED) gentoo-dev 2008-03-31 19:23:03 UTC
*** Bug 215539 has been marked as a duplicate of this bug. ***
Comment 13 Carsten Lohrke (RETIRED) gentoo-dev 2008-03-31 19:36:06 UTC
*** Bug 215586 has been marked as a duplicate of this bug. ***
Comment 14 x00000000 2008-04-02 07:42:33 UTC
Workaround is to build without -O* in CFLAGS. Most files will still be compiled with -O2, but some with -O0 (including qnametostr.cxx). Even if building with -O2 succeeds, the result may be broken, because I get lots of strict-aliasing warnings for files intended to be compiled with -O0 (they get no -fno-strict-aliasing as optimized files do). So maybe the ebuild should filter -O* instead of doing |replace-flags "-O?" "-O2"|.
Comment 15 Carsten Lohrke (RETIRED) gentoo-dev 2008-04-04 23:13:39 UTC
*** Bug 216205 has been marked as a duplicate of this bug. ***
Comment 16 Alexey Shvetsov archtester gentoo-dev 2008-04-14 12:40:16 UTC
(In reply to comment #14)
> Workaround is to build without -O* in CFLAGS. Most files will still be compiled
> with -O2, but some with -O0 (including qnametostr.cxx). Even if building with
> -O2 succeeds, the result may be broken, because I get lots of strict-aliasing
> warnings for files intended to be compiled with -O0 (they get no
> -fno-strict-aliasing as optimized files do). So maybe the ebuild should filter
> -O* instead of doing |replace-flags "-O?" "-O2"|.
> 
If we replace -O? with -O0 or -O1 openoffice builds fine with gcc-4.2.3 with 512M ram

Comment 17 x00000000 2008-04-14 18:29:38 UTC
(In reply to comment #16)
> If we replace -O? with -O0 or -O1 openoffice builds fine with gcc-4.2.3 with
> 512M ram

If you replace it, then it will be built without optimization (or only -O1). If you delete it, then it will be built with -O2, except for some problematic files.
Comment 18 Alexey Charkov 2008-04-14 20:01:21 UTC
By the way, I confirm that it compiles completely without this issue if one uses GCC 4.3.0 (presumably due to reworked optimization algorithms in this release). I did not even notice the moment when qnametostr.obj was compiled (using CFLAGS="-O2 -march=core2 -pipe"). However, a patch for 'strdup not declared in this scope' is needed, like the one provided in bug 216205.
Comment 19 Krzysztof Pawlik (RETIRED) gentoo-dev 2008-04-14 20:02:54 UTC
*** Bug 217685 has been marked as a duplicate of this bug. ***
Comment 20 Andreas Proschofsky (RETIRED) gentoo-dev 2008-04-15 11:38:21 UTC
Hmm, not quite sure how to proceed here, guess forcing everyone on -01 is not going to make us very popular ;) So open for good ideas, patches, whatever ;)
Comment 21 Hanno Zysik (geki) 2008-04-15 19:25:10 UTC
(In reply to comment #17)
> If you delete it, then it will be built with -O2, except for some problematic files.

+1

So that silent miscompiles and other issues which are already catched by openoffice build system take effect.
Comment 22 x00000000 2008-04-15 21:56:33 UTC
(In reply to comment #20)
> Hmm, not quite sure how to proceed here, guess forcing everyone on -01 is not
> going to make us very popular ;) So open for good ideas, patches, whatever ;)

As I already said, replace

  replace-flags "-O?" "-O2"

by

  filter-flags "-O[s2-9]"

You're already forcing everyone to -O2, and that's the default without flags too (for most files). The ARCH_FLAGS aren't thought for optimization flags, and if they include -Os, -O2 or higher, then -fno-strict-aliasing must be added too.

A better solution would be to patch CFLAGSOPT in the makefiles to contain -O* from user CFLAGS (especially to allow -Os), but that's not easy because they are still packed when the build starts. Maybe CFLAGSNOOPT could be set to -O1 without causing real problems.
Comment 23 Alexey Charkov 2008-04-16 10:10:34 UTC
Probably we could have USE=custom-cflags to allow the adventurous ones to try and build the package with -O2, -Os or whatever, and have all the others rely on defaults shipped with the package, as we have in case of MPlayer?
Comment 24 Patrick 2008-04-21 01:41:35 UTC
How could this become 'stable' on amd64, while nobody with more than 3 GB RAM is able to even build it?
Comment 25 Andreas Proschofsky (RETIRED) gentoo-dev 2008-04-21 12:52:20 UTC
(In reply to comment #22)
> As I already said, replace
> 
>   replace-flags "-O?" "-O2"
> 
> by
> 
>   filter-flags "-O[s2-9]"

I've done that now, thanks to everyone giving advice on this. New build finished here normal levels of memory usage, again ;)

Closing