Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 333409

Summary: squid-3.1.6 is unstable, crashing after numerous instances of SIGABRT
Product: Gentoo Linux Reporter: kfm <kfm>
Component: [OLD] ServerAssignee: Alin Năstac (RETIRED) <mrness>
Status: VERIFIED TEST-REQUEST    
Severity: normal CC: net-proxy+disabled, tomek
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description kfm 2010-08-19 09:55:39 UTC
After upgrading to squid-3.1.6, it crashed on account of a segmentation fault within just under an hour's production usage. This is the first incident of its kind I've experienced in all the years that I have used squid; it has hitherto been rock solid.

At of yet, I do not know where the regression stems from and have not yet had an opportunity to investigate further. Here are some relevant messages from the ring buffer (dmesg) with additional information provided by grsecurity's logging mechanism as I use Hardened Gentoo:

[ 7838.347582] squid[31789]: segfault at 29922e45f68 ip 00000045daeb0eec sp 000003f635e9ae40 error 6 in squid[45dadce000+261000]
[ 7838.347598] grsec: From 172.16.0.52: Segmentation fault occurred at 0000029922e45f68 in /usr/sbin/squid[squid:31789] uid/euid:0/31 gid/egid:31/31, parent /usr/sbin/squid[squid:29872] uid/euid:0/0 gid/egid:31/31
[ 7838.347608] grsec: From 172.16.0.52: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/sbin/squid[squid:31789] uid/euid:0/31 gid/egid:31/31, parent /usr/sbin/squid[squid:29872] uid/euid:0/0 gid/egid:31/31

Note that the warning about resource overstepping is quite normal as core dumps are indeed disallowed by default.

From syslog - note that 7:44:48 was the point at which squid-3.1.6 was first started and 10:06:50 is the time of failure):

Aug 19 07:44:48 spoiler squid[29872]: Squid Parent: child process 29874 started
Aug 19 09:17:25 spoiler squid[29872]: Squid Parent: child process 29874 exited due to signal 6 with status 0
Aug 19 09:17:28 spoiler squid[29872]: Squid Parent: child process 31789 started
Aug 19 09:17:29 spoiler squid[29872]: Squid Parent: child process 31789 exited due to signal 11 with status 0
Aug 19 09:17:32 spoiler squid[29872]: Squid Parent: child process 31796 started
Aug 19 09:17:32 spoiler squid[29872]: Squid Parent: child process 31796 exited due to signal 6 with status 0
Aug 19 09:17:35 spoiler squid[29872]: Squid Parent: child process 31816 started
Aug 19 09:17:36 spoiler squid[29872]: Squid Parent: child process 31816 exited due to signal 6 with status 0
Aug 19 09:17:39 spoiler squid[29872]: Squid Parent: child process 31835 started
Aug 19 09:17:50 spoiler squid[29872]: Squid Parent: child process 31835 exited due to signal 6 with status 0
Aug 19 09:17:53 spoiler squid[29872]: Squid Parent: child process 31858 started
Aug 19 09:17:54 spoiler squid[29872]: Squid Parent: child process 31858 exited due to signal 6 with status 0
Aug 19 09:17:57 spoiler squid[29872]: Squid Parent: child process 31878 started
Aug 19 09:17:57 spoiler squid[29872]: Squid Parent: child process 31878 exited due to signal 6 with status 0
Aug 19 09:18:00 spoiler squid[29872]: Squid Parent: child process 31898 started
Aug 19 09:18:01 spoiler squid[29872]: Squid Parent: child process 31898 exited due to signal 6 with status 0
Aug 19 09:18:04 spoiler squid[29872]: Squid Parent: child process 31919 started
Aug 19 09:39:31 spoiler squid[29872]: Squid Parent: child process 31919 exited due to signal 6 with status 0
Aug 19 09:39:34 spoiler squid[29872]: Squid Parent: child process 570 started
Aug 19 09:39:35 spoiler squid[29872]: Squid Parent: child process 570 exited due to signal 6 with status 0
Aug 19 09:39:38 spoiler squid[29872]: Squid Parent: child process 590 started
Aug 19 09:47:42 spoiler squid[29872]: Squid Parent: child process 590 exited due to signal 6 with status 0
Aug 19 09:47:45 spoiler squid[29872]: Squid Parent: child process 939 started
Aug 19 09:53:27 spoiler squid[29872]: Squid Parent: child process 939 exited due to signal 6 with status 0
Aug 19 09:53:30 spoiler squid[29872]: Squid Parent: child process 1204 started
Aug 19 09:54:31 spoiler squid[29872]: Squid Parent: child process 1204 exited due to signal 6 with status 0
Aug 19 09:54:34 spoiler squid[29872]: Squid Parent: child process 1245 started
Aug 19 09:55:26 spoiler squid[29872]: Squid Parent: child process 1245 exited due to signal 6 with status 0
Aug 19 09:55:29 spoiler squid[29872]: Squid Parent: child process 1312 started
Aug 19 09:55:34 spoiler squid[29872]: Squid Parent: child process 1312 exited due to signal 6 with status 0
Aug 19 09:55:37 spoiler squid[29872]: Squid Parent: child process 1333 started
Aug 19 10:06:14 spoiler squid[29872]: Squid Parent: child process 1333 exited due to signal 6 with status 0
Aug 19 10:06:17 spoiler squid[29872]: Squid Parent: child process 1709 started
Aug 19 10:06:18 spoiler squid[29872]: Squid Parent: child process 1709 exited due to signal 6 with status 0
Aug 19 10:06:21 spoiler squid[29872]: Squid Parent: child process 1729 started
Aug 19 10:06:29 spoiler squid[29872]: Squid Parent: child process 1729 exited due to signal 6 with status 0
Aug 19 10:06:32 spoiler squid[29872]: Squid Parent: child process 1750 started
Aug 19 10:06:36 spoiler squid[29872]: Squid Parent: child process 1750 exited due to signal 6 with status 0
Aug 19 10:06:39 spoiler squid[29872]: Squid Parent: child process 1770 started
Aug 19 10:06:41 spoiler squid[29872]: Squid Parent: child process 1770 exited due to signal 6 with status 0
Aug 19 10:06:44 spoiler squid[29872]: Squid Parent: child process 1791 started
Aug 19 10:06:50 spoiler squid[29872]: Squid Parent: child process 1791 exited due to signal 6 with status 0
Aug 19 10:06:50 spoiler squid[29872]: Exiting due to repeated, frequent failures

In my case, squid is built with only USE="epoll pam" effective. Here is some background information regarding the host:

System uname: Linux-2.6.32.18-grsec-r201008112233-x86_64-Intel-R-_Xeon-R-_CPU_E3113_@_3.00GHz-with-gentoo-2.0.0
Timestamp of tree: Thu, 19 Aug 2010 02:00:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
app-shells/bash:     4.0_p37
dev-java/java-config: 2.1.11
dev-lang/python:     2.6.5-r3
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 2.0.0
sys-apps/openrc:     0.6.1-r1
sys-apps/sandbox:    1.6-r2
sys-devel/autoconf:  2.65
sys-devel/automake:  1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.4-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6b
virtual/os-headers:  2.6.32
ACCEPT_KEYWORDS="amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -g -pipe"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="-march=native -O2 -g -pipe"
FEATURES="assume-digests distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-logs unmerge-orphans userfetch"
LDFLAGS="-Wl,-O1,--hash-style=gnu -Wl,--as-needed"

For the time being, I have downgraded to the squid-3.1.5 package that I attached in bug 311927 so as to placate my users.
Comment 1 kfm 2010-08-19 10:14:20 UTC
I missed the first line from --info:

Portage 2.1.8.3 (hardened/linux/amd64/10.0/no-multilib, gcc-4.4.4, glibc-2.11.2-r0, 2.6.32.18-grsec-r201008112233 x86_64)
Comment 2 Alin Năstac (RETIRED) gentoo-dev 2010-08-22 06:48:40 UTC
Version 3.1.6-r1 works as expected on one of my production servers and it's rather difficult for me to fix something I cannot reproduce. I'm afraid you'll have to investigate it further.

Closed as WORKSFORME, please reopen it if you have a patch or enough debug information for creating such patch.
Comment 3 Tomek L 2010-08-31 13:22:54 UTC
please reopen this bug, I have exactly the same issue on two productions servers using Squid 3.1.6-r1. Please don't leave it as resolved and be more precise of what kind of "debug information" you want and how to obtain it. Below my emerge --info.

Portage 2.1.8.3 (default/linux/amd64/10.0/no-multilib, gcc-4.4.3, glibc-2.11.2-r0, 2.6.35.1 x86_64)
=================================================================
System uname: Linux-2.6.35.1-x86_64-Intel-R-_Xeon-R-_CPU_X5550_@_2.67GHz-with-gentoo-1.12.13
Timestamp of tree: Tue, 31 Aug 2010 10:00:20 +0000
app-shells/bash:     4.0_p37
dev-java/java-config: 2.1.11
dev-lang/python:     2.6.5-r3, 3.1.2-r4
sys-apps/baselayout: 1.12.13
sys-apps/sandbox:    1.6-r2
sys-devel/autoconf:  2.65
sys-devel/automake:  1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.3-r2
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6b
sys-devel/make:      3.81-r2
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -mtune=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -mtune=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch userpriv"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="pl_PL.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en"
MAKEOPTS="-j16"
PKGDIR="/usr/portage/packages"
PORTAGE_COMPRESS="bzip2"
PORTAGE_COMPRESS_FLAGS="-9"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://213.25.35.137/gentoo-portage"
USE="acl amavis amd64 apache2 auto-hinter bash-completion berkdb bzip2 charconv clamav clamdtop cli cracklib crypt cxx diskio dkim dri extensible gdbm geoip gocr gpm iconv idn inode jpeg mmx modules mudflap ncurses nls nptl nptlonly ntlm ocrad openmp pam pcre perl png postgres pppd rar razor readline reflection samba session slang spamassassin spl sse sse2 ssl suhosin sysfs threads unicode vim-syntax xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="alias autoindex auth_basic authn_default authn_file authz_default authz_host authz_user cache filter deflate dir disk_cache file_cache info log_config mem_cache mime negotiation rewrite env setenvif so status vhost_alias dav dav_fs expires headers" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware voodoo" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTDIR_OVERLAY
Comment 4 kfm 2010-09-27 10:01:44 UTC
It seems that this bug may have been resolved as of squid-3.1.8. Tomek, can you confirm whether this is the case in your environment?
Comment 5 kfm 2010-10-14 10:51:46 UTC
I'm fairly certain that this is has been fixed. Closing, as nothing has been heard to the contrary.