I upgraded apache from 2.4.48 to 2.4.49 and I've noticed massive segfaults in log. I have coredump, here is backtrace: Core was generated by `/usr/sbin/apache2 -D DEFAULT_VHOST -D LANGUAGE -D PHP -D RPAF -D NAGIOS -D DAV'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3b732112f3 in apr_socket_close (thesocket=0x0) at network_io/unix/sockets.c:213 213 return apr_pool_cleanup_run(thesocket->pool, thesocket, socket_cleanup); (gdb) bt #0 0x00007f3b732112f3 in apr_socket_close (thesocket=0x0) at network_io/unix/sockets.c:213 #1 0x0000564c3d48e844 in ap_lingering_close (c=<optimized out>) at connection.c:159 #2 0x0000564c3d4a3fed in child_main (child_num_arg=child_num_arg@entry=3, child_bucket=child_bucket@entry=0) at prefork.c:655 #3 0x0000564c3d4a43de in make_child (s=<optimized out>, slot=3) at prefork.c:756 #4 0x0000564c3d4a4cfb in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:860 #5 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1053 #6 0x0000564c3d4627af in ap_run_mpm (pconf=pconf@entry=0x564c3e8083c8, plog=0x564c3e835608, s=0x564c3e831888) at mpm_common.c:95 #7 0x0000564c3d459fda in main (argc=<optimized out>, argv=<optimized out>) at main.c:819 (gdb) bt full #0 0x00007f3b732112f3 in apr_socket_close (thesocket=0x0) at network_io/unix/sockets.c:213 No locals. #1 0x0000564c3d48e844 in ap_lingering_close (c=<optimized out>) at connection.c:159 dummybuf = '\000' <repeats 176 times>, ".\000\000\000\000\000\000\000\250 \302>LV", '\000' <repeats 11 times>... nbytes = 0 now = <optimized out> timeup = 0 csd = <optimized out> #2 0x0000564c3d4a3fed in child_main (child_num_arg=child_num_arg@entry=3, child_bucket=child_bucket@entry=0) at prefork.c:655 current_conn = 0x564c3ec21ca0 csd = 0x564c3ec21ab0 thd = 0x564c3ec1faa0 osthd = 139893308863808 sig_mask = {__val = {0, 0, 94885466533848, 140730829455136, 94885445784016, 67108864, 139893310788592, 0, 0, 0, 0, 0, 0, 0, 0, 13480424970716487680}} ptrans = 0x564c3ec21a38 allocator = 0x564c3ec1f930 status = <optimized out> i = <optimized out> lr = <optimized out> pollset = 0x564c3ec1fb68 sbh = 0x564c3ec1fb60 bucket_alloc = 0x564c3ec26ae8 last_poll_idx = 0 lockfile = <optimized out> #3 0x0000564c3d4a43de in make_child (s=<optimized out>, slot=3) at prefork.c:756 bucket = 0 pid = <optimized out> #4 0x0000564c3d4a4cfb in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:860 i = 1 idle_count = <optimized out> ws = <optimized out> free_length = <optimized out> free_slots = {2, 3, 4, 5, 0, 0, 14, -1156311274, 37, 0, -1461936128, -1156311274, 1048778888, 22092, 0, 0, 1048786392, 22092, 1931528481, 32571, 1048778888, 22092, 1048609736, 22092, 1912150992, 32571, 1931477261, 32571, 0, 0, 26, 0} last_non_dead = <optimized out> total_non_dead = <optimized out> i = <optimized out> idle_count = <optimized out> ws = <optimized out> free_length = <optimized out> free_slots = {<optimized out> <repeats 32 times>} last_non_dead = <optimized out> total_non_dead = <optimized out> status = <optimized out> bucket_kill_child_record = -1 sr__ = <optimized out> sr__ = <optimized out> #5 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1053 status = 11 pid = {pid = -1, in = 0x7f3b71ed1c69 <__zend_malloc+9>, out = 0x7f3b72879e40 <php_post_entries+32>, err = 0x7f3b71ea1822 <sapi_register_post_entry+450>} child_slot = <optimized out> exitwhy = (APR_PROC_SIGNAL | APR_PROC_SIGNAL_CORE) processed_status = <optimized out> index = <optimized out> remaining_children_to_start = 0 i = <optimized out> #6 0x0000564c3d4627af in ap_run_mpm (pconf=pconf@entry=0x564c3e8083c8, plog=0x564c3e835608, s=0x564c3e831888) at mpm_common.c:95 pHook = <optimized out> --Type <RET> for more, q to quit, c to continue without paging--c n = 0 rv = -1 #7 0x0000564c3d459fda in main (argc=<optimized out>, argv=<optimized out>) at main.c:819 c = 102 'f' showcompile = <optimized out> showdirectives = 0 confname = 0x7ffe73195a7d "/etc/apache2/httpd.conf" def_server_root = 0x7ffe73195a67 "/usr/lib64/apache2" temp_error_log = <optimized out> error = <optimized out> process = 0x564c3e8064a8 pconf = 0x564c3e8083c8 plog = 0x564c3e835608 ptemp = 0x564c3e8335d8 pcommands = 0x564c3e82a4d8 opt = 0x564c3e82a5c8 rv = <optimized out> mod = <optimized out> opt_arg = 0x7ffe73195a7d "/etc/apache2/httpd.conf" signal_server = <optimized out> rc = <optimized out> I recompiled apr, apr-tools, apache-tools but it nothing changed. Downgrade "fixes" problem. Reproducible: Always Portage 3.0.20 (python 3.9.6-final-0, default/linux/amd64/17.1/no-multilib/hardened, gcc-10.3.0, glibc-2.33-r1, 5.13.0-00915-gbd6ed9fb42c0 x86_64) ================================================================= System uname: Linux-5.13.0-00915-gbd6ed9fb42c0-x86_64-Intel-R-_Xeon-R-_CPU_E3-1230_v5_@_3.40GHz-with-glibc2.33 KiB Mem: 9044116 total, 1155372 free KiB Swap: 1060856 total, 1056760 free Timestamp of repository gentoo: Mon, 04 Oct 2021 08:53:02 +0000 Head commit of repository gentoo: c27097e8f99d3d5d8899facfd6595564b951ffdd sh bash 5.1_p8 ld GNU ld (Gentoo 2.36.1 p5) 2.36.1 ccache version 4.3 [enabled] app-shells/bash: 5.1_p8::gentoo dev-lang/perl: 5.34.0-r2::gentoo dev-lang/python: 3.9.6_p2::gentoo dev-util/ccache: 4.3-r3::gentoo dev-util/cmake: 3.20.5::gentoo sys-apps/baselayout: 2.7::gentoo sys-apps/openrc: 0.43.5::gentoo sys-apps/sandbox: 2.24::gentoo sys-devel/autoconf: 2.69-r5::gentoo, 2.71-r1::gentoo sys-devel/automake: 1.16.4::gentoo sys-devel/binutils: 2.36.1-r2::gentoo, 2.37_p1::gentoo sys-devel/gcc: 10.3.0-r2::gentoo sys-devel/gcc-config: 2.4::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.10::gentoo (virtual/os-headers) sys-libs/glibc: 2.33-r1::gentoo Repositories: gentoo location: /usr/portage sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo.git sync-user: portage:portage priority: -1000 Installed sets: @masscheck, @nagios-plugins, @rblowanie, @recoll ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=westmere -mtune=native -s -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/easy-rsa /usr/share/gnupg/qualified.txt /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.4/ext-active/ /etc/php/apache2-php8.0/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cgi-php8.0/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/php/cli-php8.0/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=westmere -mtune=native -s -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -march=westmere -mtune=native -s -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs ccache cgroup collision-protect compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -march=westmere -mtune=native -s -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="pl_PL.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--sort-common" LINGUAS="en" MAKEOPTS="-j5 -l6" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="acl aio amd64 apache2 bash-completion bzip2 caps crypt hardened iconv idn ipv6 jit libglvnd libtirpc lto modules nano-syntax ncurses nls nptl openmp pcre pie readline seccomp smp split-usr ssl ssp threads unicode vhosts vim-syntax xattr xtpax zlib" ABI_X86="64" ADA_TARGET="gnat_2019" APACHE2_MODULES="alias authn_core access_compat auth_basic authz_core authn_alias authn_anon auth_digest authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cgi dav dav_fs dav_lock dir env expires ext_filter filter headers hugepages include info log_config logio mime mime_magic negotiation remoteip rewrite setenvif status unique_id unixd socache_shmcb usertrack vhost_alias" APACHE2_MPMS="itk" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="apache aggregation bind cgroups conntrack contextswitch cpu curl curl_json df disk dns email entropy ethstat exec filecount hugepages interface iptables irq lvm match_regex mysql netlink load memory network nginx notify_email ntpd openvpn ping postgresql processes protocols redis rrdcached rrdtool snmp statsd swap syslog tail tail_csv tcpconns unixsock uptime vmem" CPU_FLAGS_X86="aes mmx mmxext sse sse2 sse3 ssse3" ELIBC="glibc" GRUB_PLATFORMS="pc" KERNEL="linux" L10N="en" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" NGINX_MODULES_HTTP="access autoindex browser charset fastcgi gzip map limit_zone proxy rewrite http2 stub_status gzip_static" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" RUBY_TARGETS="ruby26" USERLAND="GNU" XTABLES_ADDONS="fuzzy geoip lscan psd tarpit" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RUSTFLAGS
APACHE2_OPTS="-D DEFAULT_VHOST -D LANGUAGE -D PHP -D RPAF -D NAGIOS -D DAV -D SVN -D SVN_AUTHZ -D PERL -D MPM_ITK"
This is likely going to have to be reported upstream. No issues here with: -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D USERDIR -D HEADERS
Can you try with .50 please? Thanks!
(In reply to Sam James from comment #3) > Can you try with .50 please? Thanks! Nothing changed. But it looks that mod_itk is triggering problem. When I run .50 without -D MPM_ITK segfaults gone away.
Created attachment 744189 [details, diff] segfault fix With the traceback provided I was able to isolate the problem to code changes in httpd-2.4.51/server/connection.c in which ap_lingering_close() attempts to pass a NULL pointer to apr_socket_close(). The attached patch prevents that and thereby resolves the segfault issue.
(In reply to acmondor from comment #5) > Created attachment 744189 [details, diff] [details, diff] > segfault fix > > With the traceback provided I was able to isolate the problem to code > changes in httpd-2.4.51/server/connection.c in which ap_lingering_close() > attempts to pass a NULL pointer to apr_socket_close(). The attached patch > prevents that and thereby resolves the segfault issue. Thanks. Is this reported upstream?
With the attached patch I don't see segfaults anymore.
I just reported this upstream as: https://bz.apache.org/bugzilla/show_bug.cgi?id=65627
What do you think about adding this patch to ebuild and don't wait for upstream new release?
ping
Adding the patch to the ebuild might be a good idea since it's not clear when a new upstream release will be provided. However, I'm not a Gentoo developer so there may be issues I'm not aware of. If a patch is added to the ebuild, it might make sense to use the upstream version rather than the one I originally provided. The patches are different, but the net effect is the same.
My servers are extra secure now that they can't process any requests and the last working version of apache has been removed from the tree.
(In reply to Michael Orlitzky from comment #12) > My servers are extra secure now that they can't process any requests and the > last working version of apache has been removed from the tree. Yes, I think it should just be added.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6b014110c2dafa2f293384be74a40e9cb01e1924 commit 6b014110c2dafa2f293384be74a40e9cb01e1924 Author: Michael Orlitzky <mjo@gentoo.org> AuthorDate: 2021-11-16 04:02:09 +0000 Commit: Michael Orlitzky <mjo@gentoo.org> CommitDate: 2021-11-16 04:02:23 +0000 www-servers/apache: new revision to unbreak mpm-itk. This latest version of apache was stabilized with bug 816258 open, affecting everyone who uses the itk MPM. Worse, all of the old ebuilds were removed, leaving no working versions of apache in the tree for anyone with an itk-based configuration. I've added an emergency patch, backported from upstream trunk, to fix the issue, and will ask for stabilization immediately. Bug: https://bugs.gentoo.org/816258 Package-Manager: Portage-3.0.28, Repoman-3.0.3 Signed-off-by: Michael Orlitzky <mjo@gentoo.org> www-servers/apache/apache-2.4.51-r1.ebuild | 264 +++++++++++++++++++++ .../apache/files/apache-2.4.51-mpm-itk.patch | 34 +++ 2 files changed, 298 insertions(+)
