Summary: | clamav-0.88.1 segmentation fault on amd64 | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Vieri <rentorbuy> |
Component: | New packages | Assignee: | Antivirus Team <antivirus> |
Status: | RESOLVED FIXED | ||
Severity: | critical | CC: | amd64, gentoo, lool+gentoo |
Priority: | High | ||
Version: | unspecified | ||
Hardware: | AMD64 | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: |
proposed clamav 0.88.1 patch for amd64
Sample infected message as split by amavis + clamav Loic's sample virus |
Description
Vieri
2006-04-12 08:11:07 UTC
Created attachment 84509 [details, diff]
proposed clamav 0.88.1 patch for amd64
Quoting: From: "Damian Menscher" "ClamAV users ML" Subject: Re: [Clamav-users] clamav-0.88.1 segmentation fault on 64bit systems CC: "ClamAV-devel" <clamav-devel@lists.clamav.net> So, this is the third report I've seen of people getting bitten by this bug. I've been warning friends not to upgrade as a result. There needs to be another release to correct this issue. It's silly to argue otherwise (so please don't bother). Damian could you provide us with a sample file that causes clamav to crash? We have reports of it working just fine on olter amd64 kits. Our amd64 is a production system and we only have one for now. :-( When clamd failed I had to de-queue over 1500 e-mails and I can't take the chance of that happening again during Easter vacation. When I go back to work on Tuesday I might emerge the non-patched clamav +debug and post it here. Meanwhile the following message may be useful: From: "Stephen Gran" <steve@lobefin.net> To: clamav-devel@lists.clamav.net, clamav-users@lists.clamav.net (in reply to Damian) No point holding off. This bug has, as far as I can tell from the CVS timestamps, been there since mid-November. I have had several reports of 64 bit issues via the Debian BTS before now (oddly, starting roughly the same time as the CVS changes that added cli_dbgmsg), but hadn't been able to track it down to that. So, anyone running clamav version >=0.88 already has the bug. I am not sure why it's manifesting itself now, but it's not particular to this release. (In reply to comment #4) sorry, I forgot to remove Stephen Gran's e-mail. Can a Gentoo bugzilla maintainer remove it please? Sorry for the inconvenience. We have two EM64T servers running clamav with following details and both are showing randing Seg Faults post upgrade to 0.88.1.
Some times a restart will last for 3 days and sometime it will segfault in like 10 minutes! no more thn just a 'segmentation fault. Bye :-(' in clamd.log
anyone with any suggestions or work arounds?
regards
Shirish
>>
Server 1: ClamAV 0.88.1 + Qmail-Scanner
Portage 2.0.54 (default-linux/amd64/2006.0, gcc-3.4.5, glibc-2.3.5-r2, 2.6.15-gentoo-r7 x86_64)
=================================================================
System uname: 2.6.15-gentoo-r7 x86_64 Intel(R) Xeon(TM) CPU 3.00GHz
Gentoo Base System version 1.6.14
dev-lang/python: 2.4.2
sys-apps/sandbox: 1.2.12
sys-devel/autoconf: 2.13, 2.59-r7
sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils: 2.16.1
sys-devel/libtool: 1.5.22
virtual/os-headers: 2.6.11-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=nocona -O2 -mmmx -msse2 -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/alias /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/eselect/compiler /etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-march=nocona -O2 -mmmx -msse2 -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig distlocks sandbox sfperms strict"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
USE="amd64 acl apache2 berkdb bzlib clamav crypt gdbm gif gmp hardened hardenedphp jpeg logrotate ncurses nls nptl nptlonly pam perl perlsuid php png postgres python qmail readline slang snmp spamassassin ssl tcpd userlocales xml2 zlib userland_GNU kernel_linux elibc_glibc"
Unset: ASFLAGS, CTARGET, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS
Server 2: Squid + ClamAV 0.88.1 + HAVP
System uname: 2.6.15-gentoo-r7 x86_64 Intel(R) Xeon(TM) CPU 3.00GHz
Gentoo Base System version 1.6.14
dev-lang/python: 2.4.2
sys-apps/sandbox: 1.2.12
sys-devel/autoconf: 2.13, 2.59-r7
sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils: 2.16.1
sys-devel/libtool: 1.5.22
virtual/os-headers: 2.6.11-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=nocona -O2 -mmmx -msse2 -mfpmath=sse -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/eselect/compiler /etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-march=nocona -O2 -mmmx -msse2 -mfpmath=sse -fomit-frame-pointer"
DISTDIR="/var/publicfile/file/0/portage/distfiles"
FEATURES="autoconfig distlocks sandbox sfperms strict"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="amd64 acl berkdb clamav crypt gdbm hardened logrotate multipleip nls nptl nptlonly pam perl python readline snmp ssl userlocales zlib userland_GNU kernel_linux elibc_glibc"
update: enclosed patch in #1 fixed the issue and havnet had anymore segfaults on ClamAV 0.88.1 /AMD64. Thanks Viere & OP for the patch. Much much appreciated. I reemerged the non-patched clamav-0.88.1 and am running it with --debug since April 21st 2006 at 11:00 AM GMT+1. "Strangely", no segfaults for now even after heavy e-mail traffic with attached zip files. Will report segfaults if/when they occur. (In reply to comment #8) > I reemerged the non-patched clamav-0.88.1 and am running it with --debug since > April 21st 2006 at 11:00 AM GMT+1. > "Strangely", no segfaults for now even after heavy e-mail traffic with attached > zip files. > Will report segfaults if/when they occur. April 21st 2006 at 14:30 GMT+1: Segmentation fault with the familiar "clamscan: corrupt or unknown ClamAV scanner error or memory/resource/perms" problem. Reverting to the patched clamav 0.88.1 wich works fine. Has someone tried compiling the standard clamav 0.88.1 without -fomit-frame-pointer on amd64? Maybe the gentoo dev team can take into consideration the attached patch and include it into an unstable ~amd64 ebuild? +1 for this patch to go on ~amd64. I havent had a crash since I applied it on my internet facing amd64 servers. (In reply to comment #3) > could you provide us with a sample file that causes clamav to crash? We have > reports of it working just fine on olter amd64 kits. Sorry for the long post but here's the log just when the non-patched 0.88.1 amd64 clamd started segfaulting. (sorry, can't provide a sample file; someone else can?) /var/spool/qmailscan/qmail-queue.log: .. Fri, 21 Apr 2006 14:04:42 CEST:17469: ------ Process 17469 finished. Total of 0.020795 secs Fri, 21 Apr 2006 14:04:43 CEST:17475: +++ starting debugging for process 17475 (ppid=17092) by uid=201 Fri, 21 Apr 2006 14:04:43 CEST:17475: w_c: elapsed time from start 0.00485 secs Fri, 21 Apr 2006 14:04:43 CEST:17475: return-path='mmorales@hphis.com', recips='iharo@mydomain.com' Fri, 21 Apr 2006 14:04:43 CEST:17475: from='=?iso-8859-1?Q?Mar=EDa_Auxiliadora?= <mmorales@hphis.com>', subj='RE: AH! !!', via SMTP from 10.215.144.15 Fri, 21 Apr 2006 14:04:43 CEST:17476: +++ starting debugging for process 17476 (ppid=17093) by uid=201 Fri, 21 Apr 2006 14:04:43 CEST:17476: w_c: elapsed time from start 0.006574 secs Fri, 21 Apr 2006 14:04:43 CEST:17475: clamdscan: finished scan in 0.022941 secs Fri, 21 Apr 2006 14:04:43 CEST:17476: return-path='allompart@conike.com', recips='jmmartorell@mydomain.com' Fri, 21 Apr 2006 14:04:43 CEST:17476: from='"ALLOMPART" <allompart@conike.com>', subj='', via SMTP from 10.215.144.15 Fri, 21 Apr 2006 14:04:43 CEST:17476: clamdscan: finished scan in 0.021432 secs Fri, 21 Apr 2006 14:04:43 CEST:17475: fprot: finished scan in 0.142919 secs Fri, 21 Apr 2006 14:04:43 CEST:17476: fprot: finished scan in 0.17402 secs Fri, 21 Apr 2006 14:04:43 CEST:17369: SA: finished scan in 5.468139 secs - hits=-2.4/5.1 Fri, 21 Apr 2006 14:04:43 CEST:17369: p_s: finished scan in 0.035261 secs Fri, 21 Apr 2006 14:04:43 CEST:17369: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562107772617369". .. Fri, 21 Apr 2006 14:04:43 CEST:17374: SA: yup, this smells like SPAM - hits=12.5/5.1/5.1 - tagging message... Fri, 21 Apr 2006 14:04:43 CEST:17374: SA: finished scan in 5.42983 secs - hits=12.5/5.1 Fri, 21 Apr 2006 14:04:43 CEST:17492: +++ starting debugging for process 17492 (ppid=17096) by uid=201 Fri, 21 Apr 2006 14:04:43 CEST:17492: w_c: elapsed time from start 0.010279 secs Fri, 21 Apr 2006 14:04:43 CEST:17369: ------ Process 17369 finished. Total of 5.903103 secs Fri, 21 Apr 2006 14:04:43 CEST:17492: return-path='sites@hoes.com', recips='ajcorral@mydomain.com' Fri, 21 Apr 2006 14:04:43 CEST:17492: from='sites@hoes.com', subj='delivery failed', via SMTP from 10.215.144.15 Fri, 21 Apr 2006 14:04:43 CEST:17492: clamdscan: there be a virus! (Worm.Mydoom.M) Fri, 21 Apr 2006 14:04:43 CEST:17492: clamdscan: finished scan in 0.02112 secs Fri, 21 Apr 2006 14:04:43 CEST:17492: v_t_d: Virus (Worm.Mydoom.M), dropping Fri, 21 Apr 2006 14:04:43 CEST:17492: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562108372617492". .. Fri, 21 Apr 2006 14:04:43 CEST:17492: ------ Process 17492 finished. Total of 0.08003 secs Fri, 21 Apr 2006 14:04:43 CEST:17374: p_s: finished scan in 0.137891 secs Fri, 21 Apr 2006 14:04:43 CEST:17374: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562107772617374". .. Fri, 21 Apr 2006 14:04:43 CEST:17495: +++ starting debugging for process 17495 (ppid=17087) by uid=201 Fri, 21 Apr 2006 14:04:43 CEST:17495: w_c: elapsed time from start 0.010178 secs Fri, 21 Apr 2006 14:04:43 CEST:17495: return-path='mountjr@sce.com', recips='cramon@mydomain.com' Fri, 21 Apr 2006 14:04:43 CEST:17495: return-path='mountjr@sce.com', recips='cramon@mydomain.com' Fri, 21 Apr 2006 14:04:43 CEST:17495: from='mountjr@sce.com', subj='Delivery reports about your e-mail', via SMTP fro m 10.215.144.15 Fri, 21 Apr 2006 14:04:43 CEST:17495: clamdscan: there be a virus! (Worm.Mydoom.M) Fri, 21 Apr 2006 14:04:43 CEST:17495: clamdscan: finished scan in 0.008197 secs Fri, 21 Apr 2006 14:04:43 CEST:17495: v_t_d: Virus (Worm.Mydoom.M), dropping Fri, 21 Apr 2006 14:04:43 CEST:17495: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562108372617495". .. Fri, 21 Apr 2006 14:04:43 CEST:17495: ------ Process 17495 finished. Total of 0.034417 secs Fri, 21 Apr 2006 14:04:43 CEST:17506: +++ starting debugging for process 17506 (ppid=17090) by uid=201 Fri, 21 Apr 2006 14:04:43 CEST:17506: w_c: elapsed time from start 0.005594 secs Fri, 21 Apr 2006 14:04:43 CEST:17506: return-path='noreply@mydomain.com', recips='jterrassa@mydomain.com' Fri, 21 Apr 2006 14:04:43 CEST:17506: from='"Automatic Email Delivery Software" <noreply@mydomain.com>', subj= '', via SMTP from 10.215.144.15 Fri, 21 Apr 2006 14:04:43 CEST:17506: clamdscan: finished scan in 0.004978 secs Fri, 21 Apr 2006 14:04:43 CEST:17374: ------ Process 17374 finished. Total of 5.966686 secs Fri, 21 Apr 2006 14:04:43 CEST:17506: fprot: there be a virus! (W32/Mydoom.O@mm) Fri, 21 Apr 2006 14:04:43 CEST:17506: fprot: finished scan in 0.1411 secs Fri, 21 Apr 2006 14:04:43 CEST:17506: v_t_d: Virus (W32/Mydoom.O@mm), dropping Fri, 21 Apr 2006 14:04:43 CEST:17506: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562108372617506". .. Fri, 21 Apr 2006 14:04:43 CEST:17506: ------ Process 17506 finished. Total of 0.162207 secs Fri, 21 Apr 2006 14:04:43 CEST:17475: SA: finished scan in 0.754568 secs - hits=0.4/5.1 Fri, 21 Apr 2006 14:04:43 CEST:17475: p_s: finished scan in 0.010189 secs Fri, 21 Apr 2006 14:04:43 CEST:17475: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562108372617475". .. Fri, 21 Apr 2006 14:04:44 CEST:17475: ------ Process 17475 finished. Total of 1.034852 secs Fri, 21 Apr 2006 14:04:44 CEST:17523: +++ starting debugging for process 17523 (ppid=17522) by uid=201 Fri, 21 Apr 2006 14:04:44 CEST:17523: w_c: elapsed time from start 0.003178 secs Fri, 21 Apr 2006 14:04:44 CEST:17523: return-path='', recips='ogdenu@ustv.com.tw' Fri, 21 Apr 2006 14:04:44 CEST:17523: from='System Administrator <postmaster@mydomain.com>', subj='Undeliverab le: **FHM-SPAM** Re: PwvmHARMACY', via SMTP from 10.215.144.20 Fri, 21 Apr 2006 14:04:44 CEST:17523: error_condition: X-Qmail-Scanner-1.25st: clamdscan: corrupt or unknown clamd sc anner error or memory/resource/perms problem - exit status 512/2 Fri, 21 Apr 2006 14:04:44 CEST:17523: ------ Process 17523 finished. Total of 0.016577 secs Fri, 21 Apr 2006 14:04:44 CEST:17400: SA: finished scan in 5.327324 secs - hits=4.2/5.1 Fri, 21 Apr 2006 14:04:44 CEST:17400: p_s: finished scan in 0.010257 secs Fri, 21 Apr 2006 14:04:44 CEST:17400: ini_sc: finished scan of "/var/spool/qmailscan/tmp/INF-BL07114562107972617400". .. Process 17523: email with subject "FHM-SPAM" is supposed to have been generated by spamassassin and contains a zip attachment which in turn has the original SPAM message. (In reply to comment #13) sorry, I must be tired: the attachment is NOT necessarily a zip file. for squid+clamav(0.88.1)+havp (0.79) on Intel EM64T (x86_64) ... following URLs were enough to SegFault ClamAV. http://releases.mozilla.org/pub/mozilla.org/extensions/slashdotter/slashdotter-1.5-fx +fl+mz+zm.xpi http://releases.mozilla.org/pub/mozilla.org/extensions/tabbrowser_preferences/ tabbrowser_preferences-1.2.8.9-fx.xpi Hi there, We've been hit by a segfault in clamd which we use with amavisd-new, threads in the clamav mailing-list pointed to this bug, but even with the proposed patch, the process still dies. A strace reveiled an infected zip file being open()ed prior to the crash, but a clamscan or clamdscan on the same file didn't trigger the bug, so the bug can not be reliably reproduced. We could not get a core nor a backtrace. Bye, comment #16, care to share 'emerge info' of computer in question? Guys, have you tried discussing this on clamav mailinglists? Looks like success of the proposed patch is mixed (or perhaps there are multiple crash issues to be addressed), so I'm waiting for upstream to provide a qualified fix. In the past, they have been decently quick about such issues. Shirish, I'm afraid I'm running Debian. O:-) Andrej: well, threads have been started on the upstream lists already, and they pointed at this patch which was not enough to fix the issue for me, hence my comments here to let you and others know that the patch does not completely fix the problem. (In reply to comment #18) Reproducibility is the problem here. I tried mailing Shirish's mozilla files to a non-patched qmail-clamav system but they did not cause a clamd segfault. I would appreciate it if users could post CFLAG options and GCC version used to compile their clamav (x86_64 users who have both failing and non-failing unpatched 0.88.1 clamd's). My CFLAGS= -march=nocona -O2 -fomit-frame-pointer -pipe gcc 3.4.4 Currently, I'm testing clamav with CFLAGS= -march=nocona -g and gdb /usr/sbin/clamd and am not reporting segfaults for now. Simply "-g -O2" here. (In reply to comment #22) > Simply "-g -O2" here. Thanks. GCC version would be useful too. (In reply to comment #16) > A strace reveiled an infected zip file being open()ed prior to the crash Loic, would you mind providing this zip file? Created attachment 85205 [details]
Sample infected message as split by amavis + clamav
The gcc version is 3.3.5 (Debian's 1:3.3.5-13).
(There was only a single crash since the patch has been applied. Either it's because the patch fixes a frequent issue, but other issues remain, or it's simply by cheer luck.) All seems ok on our mailserver: Portage 2.0.54 (default-linux/amd64/2006.0, gcc-3.4.5, glibc-2.3.5-r2, 2.6.11.8 x86_64) ================================================================= System uname: 2.6.11.8 x86_64 AMD Athlon(tm) 64 Processor 3500+ Gentoo Base System version 1.6.14 dev-lang/python: 2.3.5-r2, 2.4.2 sys-apps/sandbox: 1.2.11 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 CFLAGS="-O2 -march=k8 -pipe" Created attachment 85210 [details]
Loic's sample virus
Loic's sample zip virus (download with caution)
Loic, I've attached your zip file for simplicity. I've tested it with the unpatched clamav but with CFLAGS= -march=nocona -g No segmentation fault. :-( see comment #6 for my cflag details, its gcc 3.4.5 with Hardened flag ... SMP servers with 4 GB RAM each. (ref comment #20) -- my mozilla files failed on clamav+squid not on qmail+clamav. there is a difference, in the way clamav is accessed .. for clamav+squid access is via Unix Socket, however for Clamav+qmail-scanner my server is configured for IP (locahost) based access. not sure if that made a difference. its almost a week now, and both servers (qmail-scanner + clamav) & (squid+havp+clamav) have not shown any segfaults any more since I applied this patch. Apparently, in certain versions of GCC over-optimizations can result in invalid/unstable code. You might find this link useful: http://www.squid-cache.org/mail-archive/squid-users/200512/0212.html Quoting someone on the web: "gcc-3.4 is known to generate incorrect code when optimising on x86_64 (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21804), it may be the cause of the segfault. If removing the optimisations doesn't help you might try gcc-4." This might explain why Andrej is not reporting segfaults but Shirish and I are. Loic, I don't think this may help you. However, you *could* try upgrading GCC. Now I'm running the original clamav with just -march=nocona on a mail server and am not having segmentation faults since Sat. April 22nd at 11:00 am GMT+1 (I know it's too soon to jump to conclusions but my "optimized" clamd usually failed a lot earlier). Hi there, in comment 16, I said that the process would still crash with the patch, however this only happened once after the rebuild, and I don't trust 100% the local admin which might not checked the process has been restarted correctly. Since april the 22th, the process is running fine: amavis 20499 1.0 0.4 61284 17432 ? Ss Apr22 30:07 /usr/sbin/clamd it already did 30 hours of CPU time worth of work. I consider the bug solved on my side, the segfault I experienced after the rebuild was either a very exceptional one triggered by other conditions or a result of a messy restart or no restart at all. At all rates, this fix should be pushed upstream soon as it fixes the most frequently occurring segfault for me. Bye, Andrej et al., as you suggested I dropped a note on the clamav mailing list and got this feedback (FYIO): ===START=== From: "Matt" To: "ClamAV users ML" Subject: Re: [Clamav-users] clamav-0.88.1 segmentation fault on 64bit systems We were having the same problem on our cluster of 64-bit AMD's running gentoo. That patch seems to have resolved the issue. Without the patch during testing it was segfaulting every 5 minutes or so, with the patch it's been running without issue for a few days now. This is where I came across the patch: http://bugs.gentoo.org/show_bug.cgi?id=129702 --Matt Michael wrote: > On Sun, 23 Apr 2006, Vieri wrote: > >> My 64-bit system has clamd compiled with -march=nocona >> -O2 -fomit-frame-pointer -pipe and it segfaults quite >> regularly. Part of a debug session is listed below: > > I cannot now find the site where I found this patch; I think it was a > debian site. In any case, it fixed the segfaults I was seeing on an > AMD64 RHEL4 system. If it is not correct perhaps the clam developers > can comment on it. > > -- Michael ------------------------------------------------------------------------ > > --- clamav-0.88.1/libclamav/zziplib/zzip-zip.c.orig 2006-03-28 00:43:53.000000000 +0100 > +++ clamav-0.88.1/libclamav/zziplib/zzip-zip.c 2006-04-11 18:05:06.275644086 +0100 > @@ -30,6 +30,8 @@ > #include <sys/stat.h> > #include <unistd.h> > > +#include "others.h" > + > /* > #include "__mmap.h" > #include "__debug.h" ===END=== So I guess the clamav dev team is aware of this and will hopefully make fixes upstream so that we don't need to modify gentoo's ebuild. http://www.clamav.net/doc/0.88.2/ChangeLog seems to address this issue. Bumped in CVS. Thanks, everyone! |