Bug 308797 – Segmentation fault from large post to www-apache/libapreq2 based application

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Alex Masidlover 2010-03-10 08:41:01 UTC]

When I submit a large post to my mod_perl/Apache::Request (libapreq2) application I get a Segmentation Fault (11) from the Apache Child and a blank screen on the browser. I have a number of virtual (openVZ) machines running the same application and this only seems to happen on one of them with, as best as I can achieve, the same post parameters. The machines have just had emerge -uav world run to eliminate any 'old' packages causing the issue.

The application has extensive debugging, but Apache seg faults after the debug file is created but before any debugs are written i.e. file resulting is empty...

Reproducible: Always

Steps to Reproduce:
1. Install an application that is a mod_perl 'Handler'
2. (optional) run apache from gdb:
gdb /usr/sbin/apache2
run -X -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D LDAP -D AUTHNZ_LDAP -D PERL -D ZYMONIC -D APREQ -d /usr/lib64/apache2 -f /etc/apache2/httpd.conf
3. Make the following POST:
ZZpageid=3&ZZsystem=shop.edinburgh-printmakers.co.uk&142_prints_filter_results_per_page=1666&177_print_maintenance_print_maintenance_form_28_edition_code=EPRGFHC&177_print_maintenance_print_maintenance_form_28_pp_pandi_artist=66&177_print_maintenance_print_maintenance_form_28_pandi_title=Hotel+California&177_print_maintenance_print_maintenance_form_28_pandi_description=&177_print_maintenance_print_maintenance_form_28_pp_pandi_subject=&177_print_maintenance_print_maintenance_form_28_pp_pandi_medium=3&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_54_ipsf_edition_code=EPRGFHC&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_54_ipsf_impression_number=&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_54_ipsf_print_code=&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_ZZNEW_ipsf_edition_code=EPRGFHC&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_ZZNEW_ipsf_impression_number=&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_ZZNEW_ipsf_print_code=&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_Form_records_per_page=1&177_print_maintenance_print_maintenance_form_28_pmf_impressionsimpressions_print_sub_form_Formcurrent_fieldgroup=&177_print_maintenance_print_maintenance_form_28_pp_pandi_internet=&177_print_maintenance_print_maintenance_form_28_pp_pandi_sell_price=0.00&177_print_maintenance_print_maintenance_form_28_pp_pandi_short_code=1&177_print_maintenance_print_maintenance_form_28_pp_pandi_dept_id=01&177_print_maintenance_print_maintenance_form_28_pp_pandi_sub_dept_id=01&177_print_maintenance_print_maintenance_form_28_pp_pandi_additional2=Artist%3A+%3Cbr%2F%3ESubject%3A+%3Cbr%2F%3EMedium%3A+Lithograph&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_67_01_f_category_id=67&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_67_01_ppc_cc=EPRGFHC&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_67_01_ppc_di=01&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_67_01_ppc_sc=1&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_67_01_ppc_sdi=01&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_71_01_f_category_id=71&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_71_01_ppc_cc=EPRGFHC&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_71_01_ppc_di=01&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_71_01_ppc_sc=1&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_01_1_EPRGFHC_71_01_ppc_sdi=01&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_f_category_id=&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_cc=EPRGFHC&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_di=01&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_sc=1&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_sdi=01&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_Form_records_per_page=10&177_print_maintenance_print_maintenance_form_28_pmf_categoriespandi_print_categories_Formcurrent_fieldgroup=&177_print_maintenance_print_maintenance_form_28_print_id=28&177_print_maintenance_print_maintenance_form_Formcurrent_fieldgroup=&181_print_maintenance_print_maintenance_form_68_edition_code=EPRGFAS&181_print_maintenance_print_maintenance_form_68_pp_pandi_artist=66&181_print_maintenance_print_maintenance_form_68_pandi_title=Arthurs+Seat&181_print_maintenance_print_maintenance_form_68_pandi_description=&181_print_maintenance_print_maintenance_form_68_pp_pandi_subject_expand=Expand&181_print_maintenance_print_maintenance_form_68_pp_pandi_subject=&181_print_maintenance_print_maintenance_form_68_pp_pandi_medium=2&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_94_ipsf_edition_code=EPRGFAS&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_94_ipsf_impression_number=&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_94_ipsf_print_code=&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_ZZNEW_ipsf_edition_code=EPRGFAS&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_ZZNEW_ipsf_impression_number=&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_ZZNEW_ipsf_print_code=&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_Form_records_per_page=1&181_print_maintenance_print_maintenance_form_68_pmf_impressionsimpressions_print_sub_form_Formcurrent_fieldgroup=&181_print_maintenance_print_maintenance_form_68_pp_pandi_internet=&181_print_maintenance_print_maintenance_form_68_pp_pandi_sell_price=0.00&181_print_maintenance_print_maintenance_form_68_pp_pandi_short_code=1&181_print_maintenance_print_maintenance_form_68_pp_pandi_dept_id=01&181_print_maintenance_print_maintenance_form_68_pp_pandi_sub_dept_id=01&181_print_maintenance_print_maintenance_form_68_pp_pandi_additional2=Artist%3A+%3Cbr%2F%3ESubject%3A+%3Cbr%2F%3EMedium%3A+Etching&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_10_01_f_category_id=10&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_10_01_ppc_cc=EPRGFAS&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_10_01_ppc_di=01&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_10_01_ppc_sc=1&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_10_01_ppc_sdi=01&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_67_01_f_category_id=67&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_67_01_ppc_cc=EPRGFAS&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_67_01_ppc_di=01&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_67_01_ppc_sc=1&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_01_1_EPRGFAS_67_01_ppc_sdi=01&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_f_category_id=&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_cc=EPRGFAS&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_di=01&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_sc=1&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ZZNEW_ppc_sdi=01&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_Form_records_per_page=10&181_print_maintenance_print_maintenance_form_68_pmf_categoriespandi_print_categories_Formcurrent_fieldgroup=&181_print_maintenance_print_maintenance_form_68_print_id=68&181_print_maintenance_print_maintenance_form_Formcurrent_fieldgroup=

Actual Results:  
0b response to the browser and the following in the apache log:
child pid 30602 exit signal Segmentation fault (11)

Expected Results:  
Returned a page of XML.

Given that I have one machine with the same version of the application and, as far as I can tell, the same versions of apache, libapreq2 and mod_perl and the working machine was actually cloned (several months ago - so different packages have been installed since) from the non-working one, I'm a bit lost now. I've included emerge --info and a backtrace below. If necessary I can provide shell access to the machine in question.

(gdb) bt
#0  0x00007ffff602eef0 in strcasecmp () from /lib/libc.so.6
#1  0x00007fffef4caaa2 in apreq_hook_find_param (hook=0x1bdeb88, param=0x1bdeba8, bb=0x7ffff6303580) at parser.c:350
#2  0x00007fffef4cb6ac in apreq_hook_run (parser=0x1be58d0, t=0x1be59d0, bb=<value optimized out>) at ../include/apreq_parser.h:141
#3  apreq_parse_urlencoded (parser=0x1be58d0, t=0x1be59d0, bb=<value optimized out>) at parser_urlencoded.c:248
#4  0x00007fffef6d63c9 in apreq_filter_prefetch () from /usr/lib64/apache2/modules/mod_apreq2.so
#5  0x00007fffef6d53ad in ?? () from /usr/lib64/apache2/modules/mod_apreq2.so
#6  0x00007fffeca36ea5 in ?? () from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux/auto/APR/Request/Request.so
#7  0x00007fffefd81265 in Perl_pp_entersub () from /usr/lib/libperl.so.1
#8  0x00007fffefd7f9d6 in Perl_runops_standard () from /usr/lib/libperl.so.1
#9  0x00007fffefd29453 in Perl_call_sv () from /usr/lib/libperl.so.1
#10 0x00007ffff003ce45 in modperl_callback () from /usr/lib64/apache2/modules/mod_perl.so
#11 0x00007ffff003d75b in modperl_callback_run_handlers () from /usr/lib64/apache2/modules/mod_perl.so
#12 0x00007ffff003dc56 in modperl_callback_per_dir () from /usr/lib64/apache2/modules/mod_perl.so
#13 0x00007ffff003879b in ?? () from /usr/lib64/apache2/modules/mod_perl.so
#14 0x00007ffff0038a52 in modperl_response_handler_cgi () from /usr/lib64/apache2/modules/mod_perl.so
#15 0x0000000000439d43 in ap_run_handler ()
#16 0x000000000043d26f in ap_invoke_handler ()
#17 0x00000000004482b8 in ap_process_request ()
#18 0x0000000000445388 in ?? ()
#19 0x00000000004411f3 in ap_run_process_connection ()
#20 0x000000000044c8fb in ?? ()
#21 0x000000000044cb88 in ?? ()
#22 0x000000000044d183 in ap_mpm_run ()
#23 0x00000000004268c5 in main ()

Portage 2.1.7.17 (default/linux/amd64/10.0, gcc-4.3.4, glibc-2.10.1-r1, 2.6.27-openvz-briullov.1-r2 x86_64)
=================================================================
System uname: Linux-2.6.27-openvz-briullov.1-r2-x86_64-Intel-R-_Xeon-R-_CPU_E5405_@_2.00GHz-with-gentoo-1.12.13
Timestamp of tree: Tue, 09 Mar 2010 16:00:05 +0000
app-shells/bash:     4.0_p35
dev-java/java-config: 2.1.9-r1
dev-lang/python:     2.5.4-r2, 2.6.4-r1
sys-apps/baselayout: 1.12.13
sys-apps/sandbox:    1.6-r2
sys-devel/autoconf:  2.63-r1
sys-devel/automake:  1.7.9-r1, 1.10.3
sys-devel/binutils:  2.18-r3
sys-devel/gcc:       4.3.4
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6b
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /sbin/rc /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/gconf /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="acl amd64 berkdb bzip2 cli cracklib crypt cups cxx dri fortran gdbm gpm iconv ipv6 ldap mmx modules mudflap multilib ncurses nls nptl nptlonly openmp pam pcre perl pppd python readline reflection session spl sse sse2 ssl sysfs tcpd unicode xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nv r128 radeon savage sis tdfx trident vesa via vmware voodoo" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, MAKEOPTS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY

[Comment 1 Alex Masidlover 2010-04-06 09:26:46 UTC]

Searching the changelogs for parser.c showed that the method in question has been described as 'buggy' and version 2.12 of libapreq2 has fixes in it. I updated to this unstable (~amd64) version and the problem disappears...

[Comment 2 David Abbott (RETIRED) 2010-04-08 00:23:47 UTC]

reassigning to apache-bugs as www-apache/libapreq2 is in the apache herd

[Comment 3 Peter Volkov (RETIRED) 2011-05-23 15:09:31 UTC]

*** Bug 306145 has been marked as a duplicate of this bug. ***

[Comment 4 Peter Volkov (RETIRED) 2011-05-23 15:37:17 UTC]

Thank you for report. Real bugs in our bugzilla reflects ~arch tree state. For arch tree we open stabilization bugs and I've opened bug 368469 for this issue.