Bug 181002 – net-www/apache-2.0.x - severe memory leaks and/or race conditions

• Product: Gentoo Linux
• Status: RESOLVED NEEDINFO

[Description georgia_tech_swagger 2007-06-05 20:37:43 UTC]

http://www.ncaabbs.com/forums/

We recently upgraded apache and were at 2.0.58.   We have since had terrible problems with apache "blowing up" and eating all 3 GB of server RAM and 512 MB of server swap and taking us down.   Problems persisted for every 2.0.x version of apache in portage.  Apache 2.0.59-r2 was the worst, consuming 2 GB of RAM in under 5 minutes.   We'd have to kill it and restart apache every 5 minutes just to stay up!! 

Tried apache 2.2.4-r4 ... and it consumes much less memory and stays at reasonable levels.   It has been stable for us thus far, with no signs of growing memory consumption.

Apache error_logs were depressingly empty after every crash :(

Reproducible: Always




# eix mysql
[I] dev-db/mysql
     Available versions:  [M]*3.23.58-r1 4.0.27-r1 4.1.22-r1 [M]~4.1.23_alpha20070101-r61 5.0.26-r2 ~5.0.32 ~5.0.34 5.0.38 5.0.40
     Installed versions:  5.0.40(22:33:32 06/03/07)(berkdb -big-tables -cluster -debug -embedded -extraengine latin1 -max-idx-128 -minimal -perl -selinux ssl -static)
     Homepage:            http://www.mysql.com/
     Description:         A fast, multi-threaded, multi-user SQL database server.



# emerge --info
ncaabbs mysql # emerge --info && php -v && eix dev-db/mysql
Portage 2.1.2.7 (hardened/amd64, gcc-3.4.6, glibc-2.5-r3, 2.6.16-hardened-r11 x86_64)
=================================================================
System uname: 2.6.16-hardened-r11 x86_64 Dual Core AMD Opteron(tm) Processor 265
Gentoo Base System release 1.12.9
Timestamp of tree: Tue, 05 Jun 2007 17:30:09 +0000
ccache version 2.4 [enabled]
dev-lang/python:     2.3.5-r3, 2.4.4-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.4-r7
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.14
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.17-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=k8 -pipe -O2"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo"
CXXFLAGS="-march=k8 -pipe -O2"
DISTDIR="/usr/portage/distfiles"
FEATURES="ccache distlocks metadata-transfer sandbox sfperms strict"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext 64bit amd64 apache apache2 berkdb bzip2 ccache cdrom clamav clamd cli crypt dedicated dict expat freetype ftp gd hardened hardenedphp imagemagick imap innodb justify latin1 libclamav lm_sensors logrotate mbox midi mime mysql mysqli ncurses nls offensive openssh openssl pam pcre php pic python rar readline sasl scp session sftp spamassassin ssl unicode urandom xml xmlrpc xorg zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY



# php -v
PHP 5.2.2-pl1-gentoo (cli) (built: Jun  5 2007 15:15:28)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
    with eAccelerator v0.9.5.1, Copyright (c) 2004-2006 eAccelerator, by eAccelerator

(eAccelerator was not a factor in this bug ... the problems persisted with eAccelerator on or off)

[Comment 1 Mike Doty (RETIRED) 2007-06-05 20:51:06 UTC]

I'm running apache-2.2.* on a bunch of my hardened amd64 servers, and I haven't had a memory leak yet.  I think it's somewhere else(php is a good place to start looking)

[Comment 2 Jakub Moc (RETIRED) 2007-06-05 20:55:19 UTC]

How exactly are we supposed to reproduce this? I can't see any memory leaks on the servers where I'm using apache-2.0.x. It's a required information when filing a bug, so post some test case here.

[Comment 3 georgia_tech_swagger 2007-06-05 21:08:28 UTC]

(In reply to comment #2)
> How exactly are we supposed to reproduce this? I can't see any memory leaks on
> the servers where I'm using apache-2.0.x. It's a required information when
> filing a bug, so post some test case here.
> 

Would love to help .. any thoughts on this?  As I said, logs were fairly empty.   Steps to reproduce?  uhhhh....
1)  Get a message board with a few million posts
2)  Put several hundred people on it at the same time
3)  *boom*

Apache 2.2.x has now been running over over an hour.  It is consuming less than HALF of the memory 2.0.x was consuming in FIVE MINUTES.   That's pretty insane.

[Comment 4 georgia_tech_swagger 2007-06-07 03:39:00 UTC]

Days later, Apache 2.2.x is still running like a lean champ.   Apache was the only thing we changed.  MySQL and PHP remained constant.  This absolutely has to be a problem in Apache.

[Comment 5 Jakub Moc (RETIRED) 2007-06-10 12:12:44 UTC]

(In reply to comment #4)
> Days later, Apache 2.2.x is still running like a lean champ.   Apache was the
> only thing we changed.  MySQL and PHP remained constant.  This absolutely has
> to be a problem in Apache.
> 

Shrug; so use apache-2.2.x. We have no issues with apache-2.0.x on forums.g.o., which is about similar scale as you've suggested in comment #3, so we really need some better information to reproduce your issues.

http://forums.gentoo.org/statistics.php