Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 669230

Summary: >=mail-filter/spamassassin-3.4.2: spamd cannot start
Product: Gentoo Linux Reporter: Robin Bankhead <gentoo>
Component: Current packagesAssignee: Philippe Chaintreuil <gentoo_bugs_peep>
Status: RESOLVED CANTFIX    
Severity: normal CC: bug, hydrapolic, jaak, mjo, proxy-maint
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: Syslog output

Description Robin Bankhead 2018-10-21 19:26:19 UTC
Created attachment 552170 [details]
Syslog output

Attaching debug-level syslog output from attempt to /etc/init.d/spamd start.

Previously downgrading to 3.4.1-r21 solved the problem, but on this occasion I ran sa-update after upgrading and now the previously-good version fails to start as well.

Output in terminal is:
hazel ~ # /etc/init.d/spamd start
 * Caching service dependencies ...                                       [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/spamd start
 * Starting spamd ...
 * start-stop-daemon: fopen `/run/spamd.pid': No such file or directory
 * Detaching to start `/usr/sbin/spamd' ...
Oct 21 19:44:21.038 [22389] dbg: logger: adding facilities: all
Oct 21 19:44:21.039 [22389] dbg: logger: logging level is DBG
Oct 21 19:44:21.060 [22389] dbg: logger: calling setlogsock(unix)
Oct 21 19:44:21.060 [22389] dbg: logger: opening syslog with unix socket
Oct 21 19:44:21.061 [22389] dbg: logger: successfully connected to syslog/unix
Oct 21 19:44:21.061 [22389] dbg: logger: successfully added syslog method
Oct 21 19:44:21.061 [22389] dbg: spamd: will perform setuids? 0
Oct 21 19:44:21.063 [22389] dbg: spamd: socket module of choice: IO::Socket::IP 0.38, Socket 2.020_03, have PF_INET, no PF_INET6, no getaddrinfo, using gethostbyname, IPv4-only, AI_ADDRCONFIG not supported
Oct 21 19:44:21.063 [22389] dbg: spamd: socket specification: "localhost", IP address: localhost, port: 783
Oct 21 19:44:21.063 [22389] dbg: spamd: attempting to listen on IP addresses: 127.0.0.1, port 783
Oct 21 19:44:21.064 [22389] dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr: 127.0.0.1, LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 1, V6Only: 1
Oct 21 19:44:21.066 [22389] dbg: spamd: created IO::Socket::IP socket on [127.0.0.1]:783
Oct 21 19:44:21.066 [22389] dbg: spamd: server listen sockets fd bit field: 00000100
Oct 21 19:44:21.067 [22389] dbg: logger: adding facilities: all
Oct 21 19:44:21.067 [22389] dbg: logger: logging level is DBG
Oct 21 19:44:21.067 [22389] dbg: generic: SpamAssassin version 3.4.2
Oct 21 19:44:21.068 [22389] dbg: generic: Perl 5.026002, PREFIX=/usr, DEF_RULES_DIR=/usr/share/spamassassin, LOCAL_RULES_DIR=/etc/mail/spamassassin, LOCAL_STATE_DIR=/var/lib/spamassassin
Oct 21 19:44:21.068 [22389] dbg: config: timing enabled
Oct 21 19:44:21.081 [22389] dbg: config: score set 0 chosen.
child process [22395] exited or timed out without signaling production of a PID file: ABORTED, signal 6 (0006) at /usr/sbin/spamd line 3034.
 *   start-stop-daemon: failed to start `/usr/sbin/spamd'
 * Failed to start spamd                                                  [ !! ]
 * ERROR: spamd failed to start


/etc/conf.d/spamd:
SPAMD_OPTS="-m 5 -x -q -u nobody -D"
SPAMD_NICELEVEL=5
SPAMD_TIMEOUT=15
SPAMD_RUN_AS_ROOT=false


emerge --info mail-filter/spamassassin
Portage 2.3.51 (python 3.6.6-final-0, default/linux/amd64/17.0, gcc-7.3.0, glibc-2.27-r6, 4.18.15-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.18.15-gentoo-x86_64-AMD_Turion-tm-_II_Neo_N40L_Dual-Core_Processor-with-gentoo-2.6
KiB Mem:     3893500 total,    459940 free
KiB Swap:    7577596 total,   7577596 free
Timestamp of repository gentoo: Sat, 20 Oct 2018 19:30:01 +0000
Head commit of repository gentoo: f0af60d27859caa220f06fac51860f48e5e99a9a
sh bash 4.4_p23
ld GNU gold (Gentoo 2.31.1 p3 2.31.1) 1.16
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
ccache version 3.5 [enabled]
app-shells/bash:          4.4_p23::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.26.2::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.6::gentoo
dev-util/ccache:          3.5::gentoo
dev-util/cmake:           3.12.3::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.38.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.13.4-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.31.1-r1::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r5::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.18::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo
Repositories:

gentoo
    location: /home/cache/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-extra-opts: 
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: no

squeezebox
    location: /var/lib/layman/squeezebox
    masters: gentoo
    priority: 0

x-portage
    location: /usr/local/portage
    masters: gentoo
    priority: 1

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/make.conf /usr/bin/vncserver /usr/lib/X11/xdm/Xsetup_0 /usr/share/config /usr/share/easy-rsa /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/hp/hplip.conf /etc/php/apache2-php5.6/ext-active/ /etc/php/apache2-php7.0/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cgi-php7.0/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/php/cli-php7.0/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/home/cache/distfiles"
EMERGE_DEFAULT_OPTS="--nospinner --quiet-build=n --verbose-conflicts"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs ccache distlocks ebuild-locks fail-clean fixlafiles merge-sync multilib-strict news preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.mirrors.ovh.net/gentoo-distfiles/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://mirror.leaseweb.com/gentoo/ http://mirrors.linuxant.fr/distfiles.gentoo.org/ http://mirror.netcologne.de/gentoo/"
LANG="en_GB.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en_GB"
MAKEOPTS="-j10 -l1.2"
PKGDIR="/home/cache/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/tmp"
USE="3dnow 3dnowext X509 a52 aac accessibility acl acpi amd64 apache2 bash-completion berkdb bzip2 clamav cli consolekit crypt curl curlwrappers cxx dbm dbus dbx dri dv dvd enca encode exif fam fastcgi fat ffmpeg flac foomaticdb fortran ftp gd gd-external gdbm gnutls gudev hardened hwdb iconv imagemagick imap imlib innodb java javascript jpeg lcms libkms libtirpc libwww lzo mad matroska mdadm minimal mmx mmxext mp3 multilib mysql ncurses network nls nptl nsplugin ntfs offensive ogg openmp optimized-qmake pam pcre pcre16 php png policykit posix raw readline rtmp samba scanner seccomp session sharedmem snmp soap sse sse2 sse4a ssl startup-notification tcpd theora threads tidy tiff tokenizer tordns udev unicode usb vdpau vhosts win32codecs x264 xattr xinetd xmlrpc xsl xv xvid xvmc xz zip zlib" ABI_X86="64" APACHE2_MODULES="access_compat actions alias auth_basic auth_digest authn_core authn_anon authn_dbd authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgid dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info libass log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif slotmem_shm so socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias" APACHE2_MPMS="worker" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" L10N="en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="AMDGPU" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3 php5-4" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23" SANE_BACKENDS="hp net" USERLAND="GNU" VIDEO_CARDS="radeon r600 vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

mail-filter/spamassassin-3.4.2-r2::gentoo was built with the following:
USE="berkdb mysql ssl -cron -ipv6 -ldap -libressl -postgres -qmail -sqlite -test" ABI_X86="(64)"
Comment 1 Jaak Ristioja 2018-10-21 21:16:36 UTC
Happened to me as well, but I somehow got it to work. I suspect that a simple `sa-update && sa-compile` might help (I downgraded, started the service, stopped the service, ran `sa-update && sa-compile`, upgraded, ran `sa-update && sa-compile` and then started it successfully).
Comment 2 Robin Bankhead 2018-10-21 23:50:58 UTC
(In reply to Jaak Ristioja from comment #1)
> Happened to me as well, but I somehow got it to work. I suspect that a
> simple `sa-update && sa-compile` might help (I downgraded, started the
> service, stopped the service, ran `sa-update && sa-compile`, upgraded, ran
> `sa-update && sa-compile` and then started it successfully).

Thanks, I ran the above line once and it segfaulted, but a second time both commands succeeded and spamd started successfully.

This is the first I've been aware of sa-compile.  Should it always be run after sa-update?
Comment 3 Tomáš Mózes 2018-10-22 03:32:41 UTC
Yes, it's a good practice to run sa-update && sa-compile.
Comment 4 Joseph 2018-10-26 00:17:07 UTC
If it's good practice, it seems to me that it should be in the ebuild. (I was bitten by this as well)
Comment 6 Michael Orlitzky gentoo-dev 2018-10-26 16:28:50 UTC
Are either of you able to reproduce the crash? SpamAssassin simply won't work until you've run sa-update once, but spamd shouldn't segfault under any circumstances. (Does it really segfault, or just exit?)

If it segfaults and is reproducible, we could get a backtrace and report it upstream -- hopefully in time for v3.4.3.

Regarding sa-update and sa-compile in the ebuild: we're generally not allowed to access the network in ebuild phases that can cause a build failure. I guess we could do an sa-update && sa-compile in "emerge --config", but IMO that's more esoteric than sa-update and sa-compile themselves.
Comment 7 Joseph 2018-10-26 23:52:35 UTC
I don't think it crashed as much as it failed to start. I can't find the exact message anymore.
Comment 8 Joseph 2018-10-26 23:53:54 UTC
I can understand not allowing network access during ebuild. Thanks!
Comment 9 Robin Bankhead 2018-10-27 11:52:56 UTC
(In reply to Michael Orlitzky from comment #6)
> Are either of you able to reproduce the crash? SpamAssassin simply won't
> work until you've run sa-update once, but spamd shouldn't segfault under any
> circumstances. (Does it really segfault, or just exit?)
> 
The segfault I mentioned was by sa-compile (during sa-update && sa-compile).  I have this in the log:

Oct 21 22:24:31 hazel kernel: sa-compile[10942]: segfault at d9c876fb98 ip 00007f1f04af2a71 sp 00007ffccd6e26b0 error 4 in libmysqlclient.so.18.4.0[7f1f04ab4000+34c000]
Oct 21 22:24:31 hazel kernel: Code: 00 e8 fb 1b ff ff 48 8b bb 20 04 00 00 e8 ef 1b ff ff 48 8b bb 28 04 00 00 e8 e3 1b ff ff 48 8b 83 88 04 00 00 48 85 c0 74 2c <48> 8b b8 a8 00 00 00 e8 cb 1b ff ff 48 8b 83 88 04 00 00 48 8b 78

I had just (during the same world upgrade) migrated from the libmysqlclient bundled in dev-db/mysql to dev-db/mysql-connector-c - perhaps that's related? (I haven't seen any other issues with packages that use libmysqlclient, and SA is using mysql fine now).

> If it segfaults and is reproducible, we could get a backtrace and report it
> upstream -- hopefully in time for v3.4.3.
> 
I can try running sa-compile again to see if it recurs, when a non-busy time comes.  I suspect it will have been a one-off due to not having run sa-update in some time, though.

As for spamd's failure to start, the output above and in the attached log is all I have. No segfaults mentioned there.

> Regarding sa-update and sa-compile in the ebuild: we're generally not
> allowed to access the network in ebuild phases that can cause a build
> failure. I guess we could do an sa-update && sa-compile in "emerge
> --config", but IMO that's more esoteric than sa-update and sa-compile
> themselves.

I think you have it about right in the ebuild: the einfo is clear that sa-update must be run initially, and it recommends doing it regularly with cron (this is what I failed to do).  There's maybe a case for making it more than a recommendation (in the einfo language, or even defaulting to IUSE=+cron) as I never had any awareness that it might need to be run after an upgrade. (Indeed, if sa-compile was what actually made the difference here, a point I'm still not clear on, then this ought to be mentioned as I've never run it before in >6 years of this install).
Comment 10 Jaak Ristioja 2018-10-27 15:46:49 UTC
(In reply to Robin Bankhead from comment #9)
> I had just (during the same world upgrade) migrated from the libmysqlclient
> bundled in dev-db/mysql to dev-db/mysql-connector-c - perhaps that's
> related?

From my experience and looking at my emerge logs, I think it is not related.

I suspect that sa-compile (and/or sa-update before that) might have to be rerun each time after upgrading but before (re)starting the service, because there might be binary incompatibility between the binaries compiled by the old version of sa-compile and the new version of the spamd service which loads them.
Comment 11 Marcin Mirosław 2018-10-27 17:53:21 UTC
(In reply to Robin Bankhead from comment #9)
> (In reply to Michael Orlitzky from comment #6)
> > Are either of you able to reproduce the crash? SpamAssassin simply won't
> > work until you've run sa-update once, but spamd shouldn't segfault under any
> > circumstances. (Does it really segfault, or just exit?)
> > 
> The segfault I mentioned was by sa-compile (during sa-update && sa-compile).
> I have this in the log:
> 
> Oct 21 22:24:31 hazel kernel: sa-compile[10942]: segfault at d9c876fb98 ip
> 00007f1f04af2a71 sp 00007ffccd6e26b0 error 4 in
> libmysqlclient.so.18.4.0[7f1f04ab4000+34c000]
> Oct 21 22:24:31 hazel kernel: Code: 00 e8 fb 1b ff ff 48 8b bb 20 04 00 00
> e8 ef 1b ff ff 48 8b bb 28 04 00 00 e8 e3 1b ff ff 48 8b 83 88 04 00 00 48
> 85 c0 74 2c <48> 8b b8 a8 00 00 00 e8 cb 1b ff ff 48 8b 83 88 04 00 00 48 8b
> 78
> 
> I had just (during the same world upgrade) migrated from the libmysqlclient
> bundled in dev-db/mysql to dev-db/mysql-connector-c - perhaps that's
> related? (I haven't seen any other issues with packages that use
> libmysqlclient, and SA is using mysql fine now).


Hi,
did you rebuild libmysql linked packages? There were message after installing connector:
Due to ABI changes when switching between different client libraries,
revdep-rebuild must find and rebuild all packages linking to libmysqlclient.
Please run: revdep-rebuild --library libmysqlclient.so.18
Comment 12 Robin Bankhead 2018-10-30 16:52:50 UTC
> Hi,
> did you rebuild libmysql linked packages? There were message after
> installing connector:
> Due to ABI changes when switching between different client libraries,
> revdep-rebuild must find and rebuild all packages linking to libmysqlclient.
> Please run: revdep-rebuild --library libmysqlclient.so.18

Sorry, I missed that. There are some rebuilds needed including dev-perl/DBD-MySQL which I guess is probably what caused the segfault.  I'll do these shortly when the machine is free, but certainly it looks like this is INVALID.
Comment 13 Jaak Ristioja 2018-10-30 19:33:51 UTC
(In reply to Robin Bankhead from comment #12)
> Sorry, I missed that. There are some rebuilds needed including
> dev-perl/DBD-MySQL which I guess is probably what caused the segfault.  I'll
> do these shortly when the machine is free, but certainly it looks like this
> is INVALID.

I disagree. For me, this issue happened on a VM which is updated very often (including @preserved-rebuild and even revdep-rebuild) about every 1-3 days, and restarted every 8-20 days or so.

The VM was restarted on Oct 15, and spamd worker properly after startup. Looking at emerge.log, only some unrelated dev-qt/*, net-irc/* and www-apps/* packages were upgraded after the restart. Nothing else. Nothing from @preserved-rebuild or revdep-rebuild. Then on Oct 21, spamassassin was upgraded from 3.4.1-r21 to 3.4.2-r2, and immediately restarted afterwards, leading to this failure.

So the most reasonable explanation seems to be that spamassassin 3.4.2-r2 is not compatible with sa-update/sa-compile generated by spamassassin 3.4.1-r21.
Comment 14 Michael Orlitzky gentoo-dev 2018-10-30 21:12:57 UTC
(In reply to Jaak Ristioja from comment #13)
> Then on Oct 21, spamassassin was
> upgraded from 3.4.1-r21 to 3.4.2-r2, and immediately restarted afterwards,
> leading to this failure.
> 
> So the most reasonable explanation seems to be that spamassassin 3.4.2-r2 is
> not compatible with sa-update/sa-compile generated by spamassassin 3.4.1-r21.

SpamAssassin v3.4.2 uses an entirely different "rules" directory than v3.4.1. If you don't run the v3.4.2 sa-update before starting v3.4.2, it will just crash because it doesn't see any rules.

I think there are probably two separate problems that are reported here:

  1. The mysql-connector thingy required a rebuild, which made spamd segfault
     until the rebuild happened (nice catch Marcin) 

  2. New versions of SpamAssassin don't work anyway until you've run sa-update
     *after* upgrading.

We have an elog message for the second problem, and while that kinda sucks for an out-of-the-box experience, I don't have a better idea.
Comment 15 Benny Pedersen 2019-11-24 00:31:18 UTC
is spamd seqfaults i solved it with 

perl-cleaner --reallyall

sa-update and sa-compile is not here rellevant

is --reallyall needed on new glibc unsure, but i see logs fails on libnss_db.so
Comment 16 Michael Orlitzky gentoo-dev 2020-01-21 15:50:15 UTC
Presumably no one has had any more problems after revdep-rebuild and perl-cleaner --reallyall?

If so, we can't do anything else within spamassassin to address the problem. The rebuild of DBD-MariaDB should happen automatically next time, because slot-operator dependencies were added:

  RDEPEND="
    >=dev-perl/DBI-1.608.0
    virtual/perl-XSLoader
    mysql? ( dev-db/mysql-connector-c:0= )
    mariadb? ( dev-db/mariadb-connector-c:0= )
  "

And the perl-cleaner thing is just... something you have to live with, with perl. Nothing we can do about it.

Please reopen if you still have problems.
Comment 17 Philippe Chaintreuil 2020-01-21 19:12:56 UTC
Kind of repeating some of the above comments, I had this happen a few times when I was working on v3.4.3 because the rules had been removed/become invalid.

spamd just kind of silently fails when launched from the init script from what I can tell, but running it manually

# sudo /usr/sbin/spamd  --pidfile=/run/spamd.pid --username=spamd --groupname=spamd --nicelevel 0

(...or similar based on if you run as root) outputs the reason.  (To which I facepalm, run the sa-update cron job, and then everything's fine.)