Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 666406

Summary: >=www-servers/apache-2.4.33 segfaults with ssl enabled and apr-util with mysql
Product: Gentoo Linux Reporter: Manuel Schmitt <manuel.schmitt>
Component: Current packagesAssignee: Apache Team - Bugzilla Reports <apache-bugs>
Status: CONFIRMED ---    
Severity: critical CC: apache-bugs, jstein
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bz.apache.org/bugzilla/show_bug.cgi?id=62552
Whiteboard:
Package list:
Runtime testing required: ---

Description Manuel Schmitt 2018-09-17 11:55:01 UTC
When configuring www-servers/apache >= 2.4.33 with
* openssl-1.0.2o
* apr
* apr-util

it segefaults when having SSL enabled and using "DBDriver mysql".

Reproducible: Always

Steps to Reproduce:
1. Use www-servers/apache >= 2.4.33, latest openssl, apr and apr-util (with USE=+mysql)

2. Enable SSL in conf.d/apache2, e.g.

	APACHE2_OPTS="-D INFO -D STATUS -D SSL -D LANGUAGE -D PHP"

3. Use a mysql based auth query in vhost:

	DBDriver mysql
	DBDParams "host=HOSTNAME_HERE port=3306 user=USERNAME_HERE pass=PASS_HERE dbname=DB_HERE"
	DBDMin 4
	DBDKeep 8
	DBDMax 20
	DBDExptime 300
	AuthType Basic
	AuthName "foo"
	AuthBasicProvider dbd
	AuthDBDUserPWQuery "SELECT user_passwd FROM users WHERE user_name = %s"
	AuthzDBDQuery "SELECT user_group FROM groups WHERE user_name = %s"
Actual Results:  
Apache segfaults.



Expected Results:  
Should not segfault

The segfault does NOT happen if disabling SSL or the DBDriver* stuff. So one of them alone works perfectly.

We can reproduce this with apache 2.4.33 and 2.4.34 (all releases). 

2.4.27 works as expected with NO segfault.


We did made a strace on the apache process on startup. The difference is:

(1) with either ssl or mysql stuff disabled:

	stat("/usr/bin/suexec", 0x7ffd3365f320) = -1 ENOENT (No such file or directory)
	getpid()                                = 15418
	open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 7
	fstat(7, {st_mode=S_IFREG|0644, st_size=1510, ...}) = 0
	read(7, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1510
	close(7)                                = 0
	open("/etc/group", O_RDONLY|O_CLOEXEC)  = 7
	fstat(7, {st_mode=S_IFREG|0644, st_size=657, ...}) = 0
	read(7, "root:x:0:root\nbin:x:1:root,bin,d"..., 4096) = 657
	close(7)                                = 0
	
	
(2) with both ssl and mysql stuff enabled:
	
	stat("/usr/bin/suexec", 0x7ffd48bde690) = -1 ENOENT (No such file or directory)
	getpid()                                = 15443
	--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7fdaa3585610} ---
	+++ killed by SIGSEGV +++


We generated a core dump:

	(gdb) bt
	#0  0x00007f55c47cd610 in ?? ()
	#1  0x00007f55c420ce6c in ERR_get_state () from /usr/lib64/libcrypto.so.1.0.0
	#2  0x00007f55c420d09f in ERR_clear_error () from /usr/lib64/libcrypto.so.1.0.0
	#3  0x00007f55c41fce43 in ENGINE_load_builtin_engines () from /usr/lib64/libcrypto.so.1.0.0
	#4  0x00007f55c3ec4bde in ?? () from /usr/lib64/apache2/modules/mod_ssl.so
	#5  0x00005606741eb7ae in ap_run_pre_config ()
	#6  0x00005606741c7af3 in main ()
Comment 1 Manuel Schmitt 2018-09-17 11:57:18 UTC
# emerge --info
Portage 2.3.49 (python 3.6.5-final-0, features/web/apache, gcc-7.3.0, glibc-2.26-r7, 4.14.69 x86_64)
=================================================================
System uname: Linux-4.14.69-x86_64-AMD_FX-tm-8320_Eight-Core_Processor-with-gentoo-2.4.1
KiB Mem:    32959568 total,  29819972 free
KiB Swap:    8388540 total,   8388540 free
Timestamp of repository gentoo: Mon, 17 Sep 2018 09:15:01 +0000
Head commit of repository gentoo: 31de42637219644118151c72f274710a2a48fea0
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p2) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r2::gentoo
sys-devel/gcc:            6.4.0-r1::gentoo, 7.3.0-r4::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14::gentoo (virtual/os-headers)
sys-libs/glibc:           2.26-r7::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://mirror.manitu.net/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-metamanifest: no
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts:

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA intel-ucode-20180807"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/apache2-php7.1/ext-active/ /etc/php/apache2-php7.2/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS=" --usepkg-exclude='sys-apps/shadow'"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles getbinpkg merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp.uni-erlangen.de/gentoo"
LANG="C"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="de en us en_GB en_US"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl amd64 berkdb bzip2 cli crypt cxx dri fortran gdbm iconv idn ipv6 jpeg libtirpc modules multilib ncurses nls nptl openmp openssl pam pcre png python qemu readline seccomp ssl tcpd tiff unicode vhost-net vim-syntax virt-network xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="prefork" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" L10N="de-DE en-US en-GB en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 2 Manuel Schmitt 2018-09-18 06:10:05 UTC
Sorry, I meant

    openssl-1.0.2p

(not o), but it happens also on "o" release, too.
Comment 3 damage 2018-09-18 18:37:31 UTC
Hi,
I can confirm the segfault on version 2.4.34-r2. My core dump shows the same stack trace.

Is masking

        >=www-servers/apache-2.4.33

a workaround?

regards
Daniel
Comment 4 Manuel Schmitt 2018-09-19 02:42:31 UTC
Currently it is the only solution as we need the mysql based authentication.

We already did this masking - thank you :-)
Comment 5 Manuel Schmitt 2018-09-19 02:57:59 UTC
Maybe it's a upstream issue:

     https://bz.apache.org/bugzilla/show_bug.cgi?id=62552
Comment 6 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2018-09-24 11:37:39 UTC
Does this still happen with =www-servers/apache-2.4.35 version?
Comment 7 Manuel Schmitt 2018-09-25 04:53:57 UTC
Yes :( Also happens with 2.4.35
Comment 8 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2018-10-23 09:06:49 UTC
Still happening with 2.4.37?
Comment 9 S. Rothe 2018-12-04 20:15:48 UTC
I'm experiencing the same problem (mysql + apache, same stack trace in core dump).

More info:
https://bz.apache.org/bugzilla/show_bug.cgi?id=60947
https://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=2325

As far as I understood the issue: It's a problem of apache interacting with openssl

Workaround (from openssl bugtracker): 
- start program with "LD_PRELOAD=/lib/libssl.so.0.9.8"
- in my case: LD_PRELOAD=/usr/lib64/libcrypto.so.1.0.0 /usr/sbin/apache2 ... (works with apache 2.4.37)
Comment 10 Jonas Stein gentoo-dev 2021-06-12 21:21:06 UTC
Unfortunately no one could help here yet. Is this problem still present in the latest stable version (2.4.48)? The ticket upstream is also still open without any good news.