Bug 90694 – qmail-smtpd randomly(?) segfaults, hardened gentoo x86_64

• Product: Gentoo Linux
• Status: VERIFIED NEEDINFO

[Description Luca Lesinigo 2005-04-28 02:44:36 UTC]

I've got an x86_64 server with hardened gentoo (see below for details), using qmail as MTA (see below for details :), and qmail-smtpd sometimes, not always, segfaults.

From the logs I see:
Apr 28 11:13:48 fit qmail-smtpd[32355]: segfault at 00000000701b3270 rip 0000003652616904 rsp 00000076b94f6450 error 4
(note: segfault addresses always ends with 0, rip always ends with 904, rsp always starts with 0000007 and always ends with 0)
Apr 28 11:13:48 fit grsec: From W.X.Y.Z: signal 11 sent to /var/mail/qmail/bin/qmail-smtpd[qmail-smtpd:32355] uid/euid:201/201 gid/egid:200/200, parent /usr/bin/tcpserver[tcpserver:694] uid/euid:201/201 gid/egid:200/200


Unfortunately, I had no luck in finding why it segfaults and when. This is a production mail server, and when the traffic was really low I never noticied the problem. I found it now because some mails started taking hours to arrive, and usually a bunch of them arrives all together...
I put a cron job on a remote server sending me a mail with a timestamp every 10 minutes: many of them arrive in just some seconds. Then they stop coming, maybe for an hour, and then the delayed mails arrive all at the same time, when the remote SMTP server tries to reconnect to this, faulty, one.

I don't know if this problem is x86_64-specific, or hardened-specific, or whatever. This is the only x86_64 hardware I own now, and it's a remote server, so I cannot test different kernels on this machine (I've got no serial console to it) or test the same setup on other similar, x86_64, machines.
I'll have a spare computer next week, I'll try to reproduce the setup there but it's gonna be an i386, not an x86_64.
Also I will try a vanilla kernel if I happen to get where the server is colocated.

I really have no other ideas where to look...

Reproducible: Always
Steps to Reproduce:
1. Get a mail domain on my server ;)
2. Send mail to an account on that domain!

Actual Results:  
Sometimes qmail-smtpd segfaults, so remote SMTP servers delay mail delivery to a later time.
Sometimes it works ok, at least mail arrives sooner or later...

Expected Results:  
qmail-smtpd shouldn't segfault.

Here are all the gory details...

Hardware:
- Dell SC1425, dual Xeon EM64T HTT with ECC ram and sata disks (no error from hardware)
- using linux raid1 software on two disks
- note: hardware is new and has no other problems. Normal load is near "nothing" (currently it is 
serving mail and web for two really low-traffic domains, and nothing else), but the server is stable even 
under high load (for testing purposes)

Operating system:
- Hardened Gentoo x86_64
- grsec and PaX active
- pie & ssp active
- kernel 2.6.10-hardened-r3, SMP
- gcc (GCC) 3.4.3 20041125 (Gentoo Hardened Linux 3.4.3-r1, ssp-3.4.3-0, pie-8.7.7)

Software:
- mail-mta/qmail-1.03-r15  +noauthcram +notlsbeforeauth (-selinux) +ssl
- mail-filter/qmail-scanner-1.25-r1  -spamassassin
- net-mail/relay-ctrl-3.1.1-r2
- net-mail/vpopmail-5.4.6-r1  +clearpasswd -ipalias -mysql

emerge info - please note I also tried recompiling qmail with CFLAGS=CXXFLAGS=MAKEOPTS="", and 
also turning off ssp:

Portage 2.0.51.19 (hardened/amd64, gcc-3.4.3, glibc-2.3.4.20041102-r1, 2.6.10-hardened-r3-prod 
x86_64)
===============================================================
==
System uname: 2.6.10-hardened-r3-prod x86_64 Intel(R) Xeon(TM) CPU 2.80GHz
Gentoo Base System version 1.4.16
Python:              dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 15 2005, 17:17:06)]
dev-lang/python:     2.3.4-r1
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.5, 1.8.5-r3, 1.6.3, 1.7.9-r1, 1.4_p6, 1.9.4
sys-devel/binutils:  2.15.92.0.2-r7
sys-devel/libtool:   1.5.14
virtual/os-headers:  2.6.8.1-r4
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CFLAGS="-O -march=nocona -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/
qmail/alias /var/qmail/control /var/vpopmail/domains /var/vpopmail/etc"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O -march=nocona -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox strict"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/Linux/distributions/
gentoo"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.it.gentoo.org/gentoo-portage"
USE="amd64 apache2 bash-completion berkdb clearpasswd crypt curl gd gdbm hardened imap innodb 
ipv6 javascript jpeg justify libwww mysql ncurses nls noauthcram notlsbeforeauth pam perl pic pie png 
python readline semanticfix snmp ssl tcpd tiff userlocales vhosts xml2 zlib"
Unset:  ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS

[Comment 1 solar (RETIRED) 2005-04-28 05:36:19 UTC]

Lowering Severity. amd64 hardened is not even officially supported. 
qmail-smtpd works fine under hardened envionments.

[Comment 2 Robin Johnson 2005-04-28 12:53:01 UTC]

enable core dumps and recompile qmail with debugging.
then wait for one of these emails to come in, and generate a backtrace using the core dump.

FEATURES="nostrip"
USE="debug"
CFLAGS="-O -march=nocona -pipe -ggdb3"
CXXFLAGS=$CFLAGS

[Comment 3 Michael Hanselmann (hansmi) (RETIRED) 2005-06-05 07:44:44 UTC]

Can you provide the information, please?

[Comment 4 Michael Hanselmann (hansmi) (RETIRED) 2005-08-18 15:58:15 UTC]

No response.