apache2 worker process segfaults on any https connection with any mpm module. Reproducible for all apache versions in portage (i.e. 2.2.14 -2.2.16) Reproducible: Always Steps to Reproduce: 1. make fresh apache install with ssl 2. make https connection on localhost Actual Results: worker process segfaults noname cores # emerge --info Portage 2.1.9.39 (default/linux/x86/10.0/desktop, gcc-4.5.2, glibc-2.13-r1, 2.6.36-gentoo-r7 i686) ================================================================= System uname: Linux-2.6.36-gentoo-r7-i686-Intel-R-_Core-TM-2_Duo_CPU_E8400_@_3.00GHz-with-gentoo-2.0.1 Timestamp of tree: Mon, 14 Feb 2011 19:00:01 +0000 ccache version 3.1.2 [enabled] app-shells/bash: 4.1_p9 dev-java/java-config: 1.3.7, 2.1.11-r3 dev-lang/python: 2.6.6-r1, 2.7.1, 3.1.3 dev-util/ccache: 3.1.2 dev-util/cmake: 2.8.3-r1 sys-apps/baselayout: 2.0.1-r1 sys-apps/openrc: 0.7.0 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13::<unknown repository>, 2.68 sys-devel/automake: 1.4_p6::<unknown repository>, 1.5::<unknown repository>, 1.6.3::<unknown repository>, 1.7.9-r1::<unknown repository>, 1.8.5-r3::<unknown repository>, 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.21 sys-devel/gcc: 4.4.5, 4.5.2 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) ACCEPT_KEYWORDS="x86 ~x86" ACCEPT_LICENSE="* -@EULA dlj-1.1 PUEL" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=core2 -O2 -pipe -g0" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=core2 -O2 -pipe -g0" DISTDIR="/var/db/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv" FFLAGS="" GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles/" LANG="ru_RU.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en" MAKEOPTS="-j3" PKGDIR="/var/db/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/var/db/portage" PORTDIR_OVERLAY="/usr/local/portage/layman/java-overlay /usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi aio alsa avi bash-completion berkdb bluetooth branding bzip2 cairo caps cdr chroot cli consolekit cracklib crypt cups cxx dba dbus divx4linux dri dts dvd dvdr emboss enca encode exif fbcon firefox flac fontconfig fortran freetype gconf gdbm gdu gif gnome gnome-keyring gnutls gphoto2 gtk gtk2 iconv idn ipv6 jbig jpeg jpeg2k lcms libnotify lzma mad matroska memlimit mikmod mng modules mp3 mp4 mpeg mudflap nautilus ncurses nptl nptlonly nvidia ogg openexr opengl openmp pam pango pch pcre pdf png policykit ppds pppd python qt3support qt4 readline rtc sasl schroedinger sdl session speex spell sse sse2 ssl ssse3 startup-notification svg sysfs threads tiff tracker truetype udev unicode usb v4l v4l2 vdpau vim-syntax vorbis x264 x86 xattr xcb xml xmp xorg xscreensaver xulrunner xv xvid xvmc zlib" ALSA_CARDS="emu10k1" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authz_groupfile authz_user authz_owner authn_file auth_basic authz_svn_module filter unique_id access authz_host auth auth_dbm auth_anon auth_digest alias file_cache echo charset_lite ext_filter case_filter case_filter_in mime_magic expires headers usertrack proxy proxy_connect proxy_http info include dav dav_fs dav_lock vhost_alias speling rewrite log_config logio env setenvif mime status autoindex asis negotiation dir actions so" APACHE2_MPMS="prefork" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="v4l fbcon vesa nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
backtrace for worker mpm: #0 0x00000058 in ?? () #1 0xb74024ea in brigade_consume (bio=0x8538188, in=0x853f926 "", inlen=151) at ssl_engine_io.c:412 #2 bio_filter_in_read (bio=0x8538188, in=0x853f926 "", inlen=151) at ssl_engine_io.c:527 #3 0xb72a486e in BIO_read (b=0x8538188, out=0x853f926, outl=151) at bio_lib.c:212 #4 0xb738fb24 in ssl3_read_n (s=0x8531ac8, n=157, max=157, extend=1) at s3_pkt.c:238 #5 0xb7390d39 in ssl3_get_record (s=0x8531ac8, type=22, buf=0x853a370 "", len=4, peek=0) at s3_pkt.c:369 #6 ssl3_read_bytes (s=0x8531ac8, type=22, buf=0x853a370 "", len=4, peek=0) at s3_pkt.c:959 #7 0xb739299e in ssl3_get_message (s=0x8531ac8, st1=8465, stn=8466, mt=1, max=16384, ok=0xb50cbf0c) at s3_both.c:426 #8 0xb7381a5d in ssl3_get_client_hello (s=0x8531ac8) at s3_srvr.c:810 #9 0xb73869f9 in ssl3_accept (s=0x8531ac8) at s3_srvr.c:316 #10 0xb73a758b in SSL_accept (s=0x8531ac8) at ssl_lib.c:924 #11 0xb739307a in ssl23_get_client_hello (s=0x8531ac8) at s23_srvr.c:590 #12 0xb73939f4 in ssl23_accept (s=0x8531ac8) at s23_srvr.c:203 #13 0xb73a758b in SSL_accept (s=0x8531ac8) at ssl_lib.c:924 #14 0xb74017da in ssl_io_filter_connect (filter_ctx=0x852e200) at ssl_engine_io.c:1104 #15 0xb7401e57 in ssl_io_filter_input (f=0x85371c8, bb=0x8539010, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at ssl_engine_io.c:1350 #16 0x0806d07b in ap_rgetline_core (s=0x8538240, n=8192, read=0xb50cc23c, r=0x8538228, fold=0, bb=0x8539010) at protocol.c:231 #17 0x0806ee36 in read_request_line (conn=0x852dcc0) at protocol.c:596 #18 ap_read_request (conn=0x852dcc0) at protocol.c:891 #19 0x08086045 in ap_process_http_connection (c=0x852dcc0) at http_core.c:183 #20 0x080821e4 in ap_run_process_connection (c=0x852dcc0) at connection.c:43 #21 0x0808ecce in process_socket (thd=0x84d1720, dummy=0x852c738) at worker.c:544 #22 worker_thread (thd=0x84d1720, dummy=0x852c738) at worker.c:894 #23 0xb765e856 in dummy_worker (opaque=0x84d1720) at threadproc/unix/thread.c:142 #24 0xb75e5e93 in start_thread () from /lib/libpthread.so.0 #25 0xb754ceee in clone () from /lib/libc.so.6
Most likely a dupe of bug 354297.
(In reply to comment #2) > Most likely a dupe of bug 354297. > Perhaps. I believe this bug come with glibc upgrade to 2.13. I've rollbacked everything else to older versions, but without success.
IIRC, you can't downgrade glibc. Simply try those commits from the upstream bug.
(In reply to comment #4) > IIRC, you can't downgrade glibc. > > Simply try those commits from the upstream bug. > I've just tried 2.2.17 version suggested in the bug 354297, it seems this version has all of those patches. The result is the same, segfault still appears on the same place.
It seems this bug depends on new gcc-4.5.2, with gcc-4.4.5 there is no SIGSEGV. SIGSEV is caused by invalid buckets ring links around lines 281 - 304 of server/core_filters.с: /* Must do move before CONCAT */ brigade_move(ctx->b, ctx->tmpbb, e); if (mode == AP_MODE_READBYTES) { APR_BRIGADE_CONCAT(b, ctx->b); } else if (mode == AP_MODE_SPECULATIVE) { apr_bucket *copy_bucket; for (e = APR_BRIGADE_FIRST(ctx->b); e != APR_BRIGADE_SENTINEL(ctx->b); e = APR_BUCKET_NEXT(e)) { rv = apr_bucket_copy(e, ©_bucket); if (rv != APR_SUCCESS) { return rv; } APR_BRIGADE_INSERT_TAIL(b, copy_bucket); } } /* Take what was originally there and place it back on ctx->b */ APR_BRIGADE_CONCAT(ctx->b, ctx->tmpbb); Probably APR macros in brigade_move(...) are optimized in an unexpected way with new gcc so it damage buckets rings linkage for brigates ctx->b and ctx->tmpbb and later inconsistent brigade b, one of argument of function ap_core_input_filter(), cause the error in mod_ssl given above.
Created attachment 262655 [details] bug testsuite
I've attached testsuite for bug reproduction. This suite emulates execution of ap_core_input_filter() on https request, reduced lines 281 - 304 of server/core_filters.с to the minimal code: brigade_move(a, b, e); APR_BRIGADE_CONCAT(c, a); APR_BRIGADE_CONCAT(a, b); Correct test pass: $ CFLAGS="-O2" make && ./hope brigade C: (0x804a080, 0x804a080), head=0xbfaeba24 next: (0xbfaeba24,0xbfaeba24) Brigade C is the output bgirage b after ap_core_input_filter(...) execution. On live apache request there is one bucket in the output brigade, 0x804a080 is the pointer to that bucket. The next string is links from bucket to the ring head and both values (next and prev) have to be pointed to the ring head=0xbfaeba24. Bug reproductive result: $ CFLAGS="-O2" make && ./hope brigade C: (0x804a080, 0x804a080), head=0xbf83a404 next: (0xbf83a424,0xbf83a404) Pay attention to the second line, next link pointer from bucket 0xbf83a424 is invalid, have to be head value 0xbf83a404. This error cause SIGSEV in mod_ssl while brigade processing. This bug reproducible on ( -O2 flag is required) 1. Gentoo 4.5.2 p1.0, pie-0.4.5 2. gcc version 4.5.1 20100924 (Red Hat 4.5.1-4) Not affected versions: 1. gcc version 4.4.5 (Gentoo 4.4.5 p1.0, pie-0.4.5)
Maybe it's implicit for all of you but I prefer to make it explicit : it seems to occur on x86, not amd64.
It seems there is error in apr_ring.h file in definition of . There is macro of declaration of ring item: #define APR_RING_ENTRY(elem) \ struct { \ struct elem * volatile next; \ struct elem * volatile prev; \ } and similar for declaration a ring head #define APR_RING_HEAD(head, elem) \ struct head { \ struct elem *next; \ struct elem *prev; \ } but should be #define APR_RING_HEAD(head, elem) \ struct head { \ struct elem * volatile next; \ struct elem * volatile prev; \ } and this fix the bug.
Created attachment 262693 [details] patch for gentoo-apache-2.2.xx-2011XXYY.tar.bz2 Patch against httpd fixing volatile modifiers in APR_RING_HEAD() macro. Gentoo also needs the same patch for apr library as apache on compiling gets apr_ring.h from system installed apr library.
should be fixed with apr-1.4.2-r1
