When an "attacker" (by intention or by accident) raises many of the following hphp alerts, it may bring down the webserver: ALERT - canary mismatch on efree() - heap overflow or double efree detected The webserver (apache1 or apache2 - doesn't matter, tried both) still runs, but doesn't accept any more connections. Some virtual servers still run but after a while any virtual server refuses connections. Usually webcrawlers like GoogleBot hit this bug, and it brings down the webserver. Whether or not this is a bug in PHP, Hardened PHP or any PHP script, it should not be able to bring down my apache to completly refuse any connections but the process still running. net-www/apache-2.0.58-r2 USE="apache2 mpm-prefork ssl threads -debug -doc -ldap -mpm-itk -mpm-leader -mpm-peruser -mpm-threadpool -mpm-worker -no-suexec -static-modules" dev-lang/php-4.4.2-r6 USE="apache2 bcmath berkdb bzip2 calendar cdb cgi cli crypt ctype curl dbx discard-path exif expat fastbuild flatfile force-cgi-redirect ftp gd gdbm gmp hardenedphp iconv imap inifile mcal memlimit mhash ming mysql ncurses nls overload pcntl pcre pic posix readline session sharedext sharedmem sockets spell sqlite ssl sysvipc tokenizer truetype unicode wddx xml xmlrpc xsl zip zlib -adabas -apache -birdstep -cjk -concurrentmodphp -db2 -dbase -dbmaker -debug -doc -empress -empress-bcs -esoob -fdftk -filepro -firebird -frontbase -gd-external -hyperwave-api -informix -interbase -iodbc -ipv6 -java-external -java-internal -kerberos -ldap -libedit -mcve -mnogosearch -msql -mssql -oci8 -odbc -oracle7 -ovrimos -pfpro -postgres -recode -sapdb -snmp -solid -sybase -sybase-ct -threads -xpm -yaz" Currently there's no known way for me to reproduce this, just sitting and wait for the next "GoogleBot attack". Any idea how to further debug this? Apache goes down reproducably when this alert is triggered, but triggering it is hard to reproduce.
I think that's a security problem, yes. I would rather have contacted the Apache team before, it's probably not a gentoo-only concern. For the moment, maybe you can use an option limiting the number of requests. Furthermore, AFAIK, USE="hardenedphp" is not a default useflag. Use it at your own risks :) If the server crashes whereas there are not so many requests, that's definitively a Apache problem. I don't know how we could handle that bug...
Okay, probably I should contact the apache team... I will post here if I found out more... If I just had a clue how to reproduce this problem so I could debug and narrow it down... I now which script does trigger the problem, but not which values are passed to the php function leading to apache going down. I currently found working around this problem is by monitoring apache with GET-requests via the monit-package and simply restarting apache in case of problems. Maybe there's a way to take some sort of snapshot of php or apache for debugging at this point.
(In reply to comment #1) > If the server crashes whereas there are not so many requests, that's > definitively a Apache problem. The apache does not really crash... Processes are still running but do not accept connections - just like they are feeling busy doing other jobs. > I don't know how we could handle that bug... I'll try hardening-php and apache teams first - maybe it's just settings related so something could be put into the config files or a warning could be issued during emerge.
> Usually webcrawlers like GoogleBot hit this bug, that shouldn't happen. Note that some hackers sometimes hide their real User-Agent (and use a google or msn-bot one). Or you might have a bug in our php installation that causes these buffer overflows... (!)
> > The apache does not really crash... Processes are still running but do not > accept connections - just like they are feeling busy doing other jobs. are they running ? (have tried a "top"?) > I'll try hardening-php and apache teams first - maybe it's just settings > related so something could be put into the config files or a warning could be > issued during emerge. good idea. I agree.
(In reply to comment #5) > > The apache does not really crash... Processes are still running but do not > > accept connections - just like they are feeling busy doing other jobs. > > are they running ? (have tried a "top"?) ps axuw|grep apache showed them. ;-) Also I didn't notice any sig9 or sig11 warnings in the error_log which usually show up. But I'll have a deeper look again and post here.
(In reply to comment #4) > > Usually webcrawlers like GoogleBot hit this bug, > > that shouldn't happen. > > Note that some hackers sometimes hide their real User-Agent (and use a google > or msn-bot one). Or you might have a bug in our php installation that causes > these buffer overflows... (!) I didn't check the user agent - just the IP is always one of Google's bot ip pool. Hardening-PHP show's only the IP of the "attacker". The problem is triggered by a str_replace() in PHP - I already posted to PhpWiki team (it's the only php project which hits this bug currently). I asked them if unchecked data can be passed to this function and if one could place a guard around this function. I'll try hardening-php team and apache team next. Apache should recover from such situations anyway on it's own.
> > ps axuw|grep apache showed them. ;-) yes but "top" shows if they are consuming CPU or not
(In reply to comment #8) > > > > ps axuw|grep apache showed them. ;-) > > yes but "top" shows if they are consuming CPU or not It's a quad cpu server - so I may not have noticed slowdowns. But I didn't notice a high load, and cpu usage was pretty low. There was just one process (nscd) which eats 100% of one cpu from time to time. But that is related to libnss-mysql as far as I found out yet. I doubt that might be related. It looks like apache's processes just sit there and do nothing (even accept no new connections).
OK. Having a pool of processes waiting for clients is a normal behaviour. Future incoming connections should be catched by the main apache process which will use one of its children. (or start a new one if needed, but it's longer). If a process fails (because of a php segfault or time-exceeded for example), the Apache child process prints it in the logfile, and that's all... I think all you can see here is really normal... let'me close the bug unless you can really exhibe a crash of the Apache server... Feel free to reopen if you disagree.
