I've got an x86_64 server with hardened gentoo (see below for details), using qmail as MTA (see below for details :), and qmail-smtpd sometimes, not always, segfaults.
From the logs I see:
Apr 28 11:13:48 fit qmail-smtpd: segfault at 00000000701b3270 rip 0000003652616904 rsp 00000076b94f6450 error 4
(note: segfault addresses always ends with 0, rip always ends with 904, rsp always starts with 0000007 and always ends with 0)
Apr 28 11:13:48 fit grsec: From W.X.Y.Z: signal 11 sent to /var/mail/qmail/bin/qmail-smtpd[qmail-smtpd:32355] uid/euid:201/201 gid/egid:200/200, parent /usr/bin/tcpserver[tcpserver:694] uid/euid:201/201 gid/egid:200/200
Unfortunately, I had no luck in finding why it segfaults and when. This is a production mail server, and when the traffic was really low I never noticied the problem. I found it now because some mails started taking hours to arrive, and usually a bunch of them arrives all together...
I put a cron job on a remote server sending me a mail with a timestamp every 10 minutes: many of them arrive in just some seconds. Then they stop coming, maybe for an hour, and then the delayed mails arrive all at the same time, when the remote SMTP server tries to reconnect to this, faulty, one.
I don't know if this problem is x86_64-specific, or hardened-specific, or whatever. This is the only x86_64 hardware I own now, and it's a remote server, so I cannot test different kernels on this machine (I've got no serial console to it) or test the same setup on other similar, x86_64, machines.
I'll have a spare computer next week, I'll try to reproduce the setup there but it's gonna be an i386, not an x86_64.
Also I will try a vanilla kernel if I happen to get where the server is colocated.
I really have no other ideas where to look...
Steps to Reproduce:
1. Get a mail domain on my server ;)
2. Send mail to an account on that domain!
Sometimes qmail-smtpd segfaults, so remote SMTP servers delay mail delivery to a later time.
Sometimes it works ok, at least mail arrives sooner or later...
qmail-smtpd shouldn't segfault.
Here are all the gory details...
- Dell SC1425, dual Xeon EM64T HTT with ECC ram and sata disks (no error from hardware)
- using linux raid1 software on two disks
- note: hardware is new and has no other problems. Normal load is near "nothing" (currently it is
serving mail and web for two really low-traffic domains, and nothing else), but the server is stable even
under high load (for testing purposes)
- Hardened Gentoo x86_64
- grsec and PaX active
- pie & ssp active
- kernel 2.6.10-hardened-r3, SMP
- gcc (GCC) 3.4.3 20041125 (Gentoo Hardened Linux 3.4.3-r1, ssp-3.4.3-0, pie-8.7.7)
- mail-mta/qmail-1.03-r15 +noauthcram +notlsbeforeauth (-selinux) +ssl
- mail-filter/qmail-scanner-1.25-r1 -spamassassin
- net-mail/vpopmail-5.4.6-r1 +clearpasswd -ipalias -mysql
emerge info - please note I also tried recompiling qmail with CFLAGS=CXXFLAGS=MAKEOPTS="", and
also turning off ssp:
Portage 22.214.171.124 (hardened/amd64, gcc-3.4.3, glibc-126.96.36.19941102-r1, 2.6.10-hardened-r3-prod
System uname: 2.6.10-hardened-r3-prod x86_64 Intel(R) Xeon(TM) CPU 2.80GHz
Gentoo Base System version 1.4.16
Python: dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 15 2005, 17:17:06)]
sys-devel/autoconf: 2.13, 2.59-r6
sys-devel/automake: 1.5, 1.8.5-r3, 1.6.3, 1.7.9-r1, 1.4_p6, 1.9.4
CFLAGS="-O -march=nocona -pipe"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/
qmail/alias /var/qmail/control /var/vpopmail/domains /var/vpopmail/etc"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O -march=nocona -pipe"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox strict"
USE="amd64 apache2 bash-completion berkdb clearpasswd crypt curl gd gdbm hardened imap innodb
python readline semanticfix snmp ssl tcpd tiff userlocales vhosts xml2 zlib"
Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS
Lowering Severity. amd64 hardened is not even officially supported.
qmail-smtpd works fine under hardened envionments.
enable core dumps and recompile qmail with debugging.
then wait for one of these emails to come in, and generate a backtrace using the core dump.
CFLAGS="-O -march=nocona -pipe -ggdb3"
Can you provide the information, please?