When I try to do subversion operations with binary files, the http connection drops and the svn client breaks with "connection truncated" or similar errors. Some files work, others do not. Http-Compression=no and lots of other workarounds I found in other forums did not help. It happens with "svn checkout" , "svn update", "svn copy" and maybe with others. When I disable mod_auth_pam everything works just fine. There is also a workaround with loss of security: If I have one vhost doing the auth_pam stuff and proxying to another vhost without authentication, then everything works. Workaround Example: <VirtualHost localhost:80> ServerName localhost <Location /repos> DAV svn SVNParentPath /svn Allow from all AuthType None </Location> </VirtualHost> <VirtualHost external.host.name:443> ServerName svn.entropia.biz # SSL Configuration not shown <Location /repos> AuthType Basic AuthPAM_Enabled on AuthName "Some" require user memyself ProxyPass https://localhost/repos/ ProxyPassReverse https://localhost/repos/ </Location> </VirtualHost> Reproducible: Always Steps to Reproduce: Install apache2, mod_auth_pam and subversion. (I tried with some versions around apache-2.0.52-r1 subversion-1.1.3 mod_auth_pam-1.1.1 on x86 and amd64) Apache conf: <Location /repos> DAV svn SVNParentPath /svn AuthPAM_Enabled on AuthType Basic AuthName "Some" require user myself </Location> Put some larger binary files in the repository and check out: svn checkout https://..../repository /tmp/tmpco Actual Results: # svn checkout https://..../repository /tmp/tmpco svn: REPORT request failed on '/repository/!svn/vcc/default' svn: REPORT of '/repository/!svn/vcc/default': Could not read chunk delimiter: Secure connection truncated (https://....) Apache error log shows apache2: dl-close.c:365: _dl_close: Assertion `imap->l_type == lt_loaded || imap->l_opencount > 0' failed. I use mod_dav_svn, apache2 and mod_auth_pam. Emerge-Info: Portage 2.0.51.19 (default-linux/x86/2004.3, gcc-3.3.5, glibc-2.3.4.20041102-r1, 2.6.10-gentoo-r6 i686) ================================================================= System uname: 2.6.10-gentoo-r6 i686 Intel(R) Pentium(R) 4 CPU 2.66GHz Gentoo Base System version 1.4.16 Python: dev-lang/python-2.3.4-r1 [2.3.4 (#1, Mar 9 2005, 18:26:02)] ccache version 2.3 [enabled] dev-lang/python: 2.3.4-r1 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.9.4, 1.5, 1.8.5-r3, 1.6.3, 1.7.9-r1, 1.4_p6 sys-devel/binutils: 2.15.92.0.2-r1 sys-devel/libtool: 1.5.10-r4 virtual/os-headers: 2.6.8.1-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-Os -march=pentium4 -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-Os -march=pentium4 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms" GENTOO_MIRRORS="http://ftp.uni-erlangen.de/pub/mirrors/gentoo http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ ftp://ftp.gentoo.mesh-solutions.com/gentoo/ ftp://pandemonium.tiscali.de/pub/gentoo/" LANG="de_DE@euro" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 3dnow X acl apache2 avi bash-completion berkdb bitmap-fonts cdr crypt dvd dvdr flash gd gdb gdbm gkrellm gnome gnomedb gtk gtk2 gtkhtml imap java jpeg jpeg2k kde ldap mmx mmx2 mozilla mozp3p mozsvg mozxmlterm mp3 mysql ncurses nls nptl opengl pam perl php png postgres postgresql python readline samba slang sse sse2 ssl tiff truetype truetype-fonts unicode xfs xinerama xpm xscreensaver xsl xslt xv zlib linguas_de" Unset: ASFLAGS, CBUILD, CTARGET, LC_ALL, LDFLAGS, PORTDIR_OVERLAY
If you want to use the workaround I suggested, please remove the / at the end of the ProxyPass and ProxyPassReverse statement. Like this: ProxyPass https://localhost/repos ProxyPassReverse https://localhost/repos
Forget my workaround. The bug still happens... Not as often as before but it still happens. I think subversion with mod_auth_pam is a key feature so I increased severity to critical.
If I understand you correctly this is a problem specifically connected to mod_auth_pam with subversion. Do you/have you used the mod_auth_pam module with other web resources? If not, could you test that? I'm cc'ing the apache team on this.
mod_auth_pam alone seems to be fine. mod_dav_svn alone seems to be fine too. If I split them on two seperate vhosts, the error occurs, but only 10% of the time. I'm currently trying lots of different ebuild-versions with and without nptl/threads to find out what caused the bug. What I've found out so far is, that mod_auth_pam does not call dlclose, but apache does. The error is then reported by the glibc function dl-close, as the subversion client does a lot of subsequent http connections. I currently try it with an older version of glibc. If I can do anything to help (e.g. provide root access to the server with the problem) I will do that. Just write a mail. Aron
This is maybe another useful information: I disabled the pam_ldap module, then I can use mod_auth_pam for subversion. I'm digging into that now.
If you're using ldap anyway you might consider using mod_auth_ldap. In any case it seems to be some strange interaction between the modules.
I will try to move to mod_auth_ldap. Exactly that configuration now works for about one year. It broke after a reboot, so I guess it's a package update that causes the error. (or is it a race condition?) BTW I tracked it down to this line in /etc/pam.d/system-auth password sufficient /lib/security/pam_ldap.so use_authtok If it's commented out, everything works, if not it breaks.
I switched the production system (x86) to mod_auth_ldap. No more problems, thanks for your help! This will hopefully work also on a 64bit dual-opteron machine. (The replacement for the production system) Thanks a lot! Bug state -> Worksforme
Probably some kind of strange bug /interaction in either mod_auth_pam, pam, padl_pam (or whatever the pam module is called), apache2 or subversion. It's probably easier to find a working configuration than to find out what exactly is going wrong.
The problem persists with mod_auth_ldap :((( So maybe it's an openldap bug? apache2: dl-close.c:365: _dl_close: Assertion `imap->l_type == lt_loaded || imap->l_opencount > 0' failed.
I downgraded to apache-2.0.51-r1, and for now it works. But I fear that's not the end yet :(
That's all kinda strange. Okay, I'm not using mod_ldap, nor ldap at all. But I'm using mod_auth_pam as well as mod_dav_svn/subversion extensivily (w/o any errors like this). It might be interesting to debug this piece of code that throws this assertion, anyway. Although, another question is, how safe is using mod_auth_{ldap,pam} at the same time (If I understood you right).
It's now running about one day with the usual load, and it's perfectly stable with apache-2.0.51-r1 or without ldap. On apache2.0.52* neither mod_auth_ldap nor mod_auth_pam with ldap authentication is working. Surprisingly it's enough to enable mod_auth_pam/mod_auth_ldap, even if you login with a local (passwd) user. The error seems to be an old glibc bug in dlclose (for unloading a library I guess). Most of dlclose calls in apache are commented out, so I guess the call that leads to the assertion is located in a ldap library that is used by mod_auth_ldap and pam_ldap. Subversion opens a lot of subsequent connections, so maybe it is a threading/locking issue or a race condition. But there was no difference in the behaviour, with threading enabled or without. (I also tried with and without nptl glibc) For now I'm out of ideas and open for suggestions.
we are having the exact problem. Using Mod_auth_pam along with winbind, and Subversion dies intermittently with following in apache error_log. apache2: dl-close.c:365: _dl_close: Assertion `imap->l_type == lt_loaded || imap->l_opencount > 0' failed. I doubt its a subversion issue, as i can use svn+ssh:// method fine. all repos verify fine. [Fri Apr 08 23:23:54 2005] [notice] child pid 23991 exit signal Aborted (6) apache2: dl-close.c:365: _dl_close: Assertion `imap->l_type == lt_loaded || imap->l_opencount > 0' failed. [Fri Apr 08 23:23:57 2005] [notice] child pid 24047 exit signal Aborted (6) >> This happens on commands such as "svn update" or svn checkout etc. Typically happens about 10 or 11th files during checkout. SVN fails saying svn: REPORT request failed on '/testing/!svn/vcc/default' svn: REPORT of '/testing/!svn/vcc/default': Could not read chunk delimiter: Secure connection truncated (https://svn.example.com) >> Its fairly critical as its on a live server, with a fairly large user base. Let me know if any other information is required. i have other sites on the same apache server using mod_auth_pam + winbind + mod_php (viewcvs, websvn etc) and they all work fine. Strange to see SVN opening (flooding) Mod_auth_pam (& winbind) about 60 times with same user access granted message, b4 process dies out with apache * client side error above. I have tried it on Apache 2.0.53-r1 as we require version svn 1.1.4. as well as bug seems to remain even on Apache 2.0.52-r1. We do not have mod_auth_ldap installed. [ebuild R ] net-www/apache-2.0.53 +apache2 -debug -doc +ldap -mpm-leader -mpm-peruser -mpm-prefork -mpm-threadpool -mpm-worker -no-suexec +ssl -static-modules +threads 0 kB [ebuild R ] dev-util/subversion-1.1.4 +apache2 +bash-completion -berkdb -debug -emacs -java +nls +perl +python +ssl 0 kB As original poster explained, the bug goes away when we disable mod_auth_pam. Emerge info == Portage 2.0.51.19 (default-linux/x86/2005.0, gcc-3.3.5-20050130, glibc-2.3.4.20041102-r1, 2.6.11-gentoo-r5 i686) ================================================================= System uname: 2.6.11-gentoo-r5 i686 Pentium III (Katmai) Gentoo Base System version 1.4.16 Python: dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 18 2005, 14:13:23)] distcc 2.16 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] dev-lang/python: 2.3.4-r1 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.5, 1.8.5-r3, 1.7.9-r1, 1.6.3, 1.9.4, 1.4_p6 sys-devel/binutils: 2.15.92.0.2-r7 sys-devel/libtool: 1.5.10-r4 virtual/os-headers: 2.6.8.1-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O2 -march=pentium3 -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=pentium3 -fomit-frame-pointer" DISTDIR="/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms" GENTOO_MIRRORS="http://mirror.pacific.net.au/gentoo" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" USE="acl apache2 bash-completion crypt dlloader erandom gd gdbm hardened hardenedphp kerberos ldap nls nptl nptlonly pam perl php python readline samba sse ssl threads userlocales winbind x86 xml2 zlib" Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, PORTDIR_OVERLAY
hmm... this bug is getting worse. In short: can you provide a backtrace (callstack)? looks like we've to track into the sources a bit more (upstream bug?)
I tried to, but I was unable to find the line with the assertion. It's not in mod_auth_ldap/pam nor in apache itself. Turning on debug information in http.conf did not improve log output, could you give me a clue on how to trace this? I can setup a vmware with the bug if you want to take a closer look at this problem. Just tell me how to transmit login credentials to you.
Created attachment 55966 [details] Valgrind output-1
Created attachment 55967 [details] valgrind output -2
I have tried various combinations. Demoted to 2.0.51 but still bug persists. Moved to 2.0.53 finally, as we need some features from SVN1.1.4. here is the summary, let me know if more information is required. I have tested it on a few other servers i have, using similar config all have similar results. I am not sure if GDB output is what/how u expected, i havent used GDB, so if i have made any mistakes, then please do let me know the instructions. (i have basically followed http://dev.gentoo.org/~tigger/blog/guidexml/auditing-system.html APache, etc have been compiled with [ebuild R ] net-www/apache-2.0.53 +apache2 +debug -doc -ldap -mpm-leader -mpm-peruser -mpm-prefork -mpm-threadpool -mpm-worker -no-suexec +ssl -static-modules -threads 0 kB [ebuild R ] dev-util/subversion-1.1.4-r1 +apache2 -bash-completion -berkdb +debug -emacs -java +nls +perl +python +ssl +webdav 0 kB [ebuild R ] net-www/mod_auth_pam-1.1.1-r1 +apache2 0 kB [ebuild R ] net-www/mod_auth_kerb-5.0_rc6 +apache2 0 kB [ebuild R ] net-fs/samba-3.0.10 +acl -cups -debug -doc +kerberos +ldap -libclamav -mysql -oav +pam -postgres -python -quotas +readline (-selinux) +winbind -xml +xml2 0 kB [ebuild R ] dev-libs/apr-0.9.6-r1 -debug -ipv6 0 kB [ebuild R ] net-misc/neon-0.24.7 -expat +ssl +zlib 0 kB [ebuild R ] dev-libs/apr-util-0.9.6 -berkdb +gdbm -ldap 0 kB >> Portage 2.0.51.19 (default-linux/x86/2005.0, gcc-3.3.5-20050130, glibc-2.3.4.20041102-r1, 2.6.11-gentoo-r5 i686) ================================================================= System uname: 2.6.11-gentoo-r5 i686 Pentium III (Katmai) Gentoo Base System version 1.4.16 Python: dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 18 2005, 14:13:23)] distcc 2.16 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] dev-lang/python: 2.3.4-r1 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.5, 1.8.5-r3, 1.7.9-r1, 1.6.3, 1.9.4, 1.4_p6 sys-devel/binutils: 2.15.92.0.2-r7 sys-devel/libtool: 1.5.10-r4 virtual/os-headers: 2.6.8.1-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O2 -march=pentium3 -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=pentium3 -fomit-frame-pointer" DISTDIR="/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms" GENTOO_MIRRORS="http://mirror.pacific.net.au/gentoo" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" USE="acl apache2 bash-completion crypt gd gdbm hardened hardenedphp kerberos nls pam perl php python readline samba sse ssl userlocales webdav winbind x86 xml2 zlib" Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, PORTDIR_OVERLAY >> When used under GDB, output is == GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu". (gdb) set args -D AUTH_PAM -d AUTH_KERB -D SSL -D DAV -D SVN -D SVN_AUTHZ -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start -X (gdb) file /usr/sbin/apache2.prefork Reading symbols from /usr/sbin/apache2.prefork...done. Using host libthread_db library "/lib/libthread_db.so.1". (gdb) run Starting program: /usr/sbin/apache2.prefork -D AUTH_PAM -d AUTH_KERB -D SSL -D DAV -D SVN -D SVN_AUTHZ -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start -X Detaching after fork from child process 9477. Program received signal SIGABRT, Aborted. 0xb7b5c971 in ?? () (gdb) bt #0 0xb7b5c971 in ?? () #1 0xb7c5c271 in ?? () #2 0x00002502 in ?? () #3 0x00000006 in ?? () #4 0xb7c62de0 in ?? () #5 0x00000006 in ?? () #6 0x00000002 in ?? () #7 0xbfffe8d4 in ?? () #8 0xb7c5c5eb in ?? () #9 0x00004000 in ?? () #10 0x00000006 in ?? () #11 0xb7c4ad4c in ?? () #12 0x00000006 in ?? () #13 0xbfffe8ec in ?? () #14 0xb7b5c704 in ?? () #15 0x00000006 in ?? () #16 0xb7c5a3e2 in ?? () #17 0xb7c4ad4c in ?? () #18 0xb7c3bdaf in ?? () #19 0xbfffea18 in ?? () #20 0xb7b5dc2d in ?? () #21 0x00000006 in ?? () #22 0xbfffe900 in ?? () >> from apache error-log == [Mon Apr 11 19:10:19 2005] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec2) [Mon Apr 11 19:10:19 2005] [info] PAM: mod_auth_pam/2.0-1.1 [Mon Apr 11 19:10:19 2005] [info] Init: Initializing OpenSSL library [Mon Apr 11 19:10:19 2005] [info] Init: Seeding PRNG with 136 bytes of entropy [Mon Apr 11 19:10:19 2005] [info] Init: Generating temporary RSA private keys (512/1024 bits) [Mon Apr 11 19:10:19 2005] [info] Init: Generating temporary DH parameters (512/1024 bits) [Mon Apr 11 19:10:19 2005] [info] Init: Initializing (virtual) servers for SSL [Mon Apr 11 19:10:19 2005] [info] Server: Apache/2.0.53, Interface: mod_ssl/2.0.53, Library: OpenSSL/0.9.7e [Mon Apr 11 19:10:19 2005] [notice] Digest: generating secret for digest authentication ... [Mon Apr 11 19:10:19 2005] [notice] Digest: done [Mon Apr 11 19:10:19 2005] [info] PAM: mod_auth_pam/2.0-1.1 [Mon Apr 11 19:10:19 2005] [info] Init: Initializing OpenSSL library [Mon Apr 11 19:10:19 2005] [info] Init: Seeding PRNG with 136 bytes of entropy [Mon Apr 11 19:10:19 2005] [info] Init: Generating temporary RSA private keys (512/1024 bits) [Mon Apr 11 19:10:20 2005] [info] Init: Generating temporary DH parameters (512/1024 bits) [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(365): shmcb_init allocated 128000 bytes of shared memory [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(568): entered shmcb_init_memory() [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(590): for 128000 bytes, recommending 1066 indexes [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(633): shmcb_init_memory choices follow [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(635): division_mask = 0x0F [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(637): division_offset = 64 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(639): division_size = 7996 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(641): queue_size = 800 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(643): index_num = 66 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(645): index_offset = 8 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(647): index_size = 12 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(649): cache_data_offset = 8 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(651): cache_data_size = 7188 [Mon Apr 11 19:10:20 2005] [debug] ssl_scache_shmcb.c(664): leaving shmcb_init_memory() [Mon Apr 11 19:10:20 2005] [info] Shared memory session cache initialised [Mon Apr 11 19:10:20 2005] [info] Init: Initializing (virtual) servers for SSL [Mon Apr 11 19:10:20 2005] [info] Server: Apache/2.0.53, Interface: mod_ssl/2.0.53, Library: OpenSSL/0.9.7e apache2.prefork: dl-close.c:365: _dl_close: Assertion `imap->l_type == lt_loaded || imap->l_opencount > 0' failed. >> Valgrind output as attached.
Shirish Jain: the valgrind output: all "conditional jump or move" messages belong to glibc and are raised because of how glibc has been compiled. Safely ignore those ones. The SIGABRT action lets me fear. Usually, for C++, this action comes when a native language exception gets thrown but not catched. Maybe one of the apache modules you've loaded in is written in C++ (we don't know - yet). However, in C (and in general) SIGABRT gets also thrown on broken pipes e.g. But according to the stacktrace given this does (yet again) not belong to apache itself, rather to valgrind. Would be interesting wether valgrind 2.4.0 is unmasked already (and ready to use on gentoo). Furthermore, It is adviceable to run apache with runtime argument "-X" (you didn't pass it, obviousely, did you?) that lets apache *NOT* fork into background nor spawn any other children, instead, it's keeping in foreground and processes just one request at a time (for exactly *those* debugging times). The backtrace you already reported does not show up any symbols nor file name to line relation. You did not compile with debugging enabled (instruction below) Secondly, did you already try running apache-2.0.53 (please lets don't talk about older versions of apache for now) without any other additional modules loaded except those one enabled by upstream? How I do track bugs within apache [modules]: * set C[XX]FLAGS in /etc/make.conf to "-O0 -ggdb3 -pipe" - nothing more. * add to /etc/make.conf: FEATURES="${FEATURES} nostrip keeptemp keepwork" * emerge -C apache apr apr-util # and all related. (apr and apr-util added here for safety, just to make sure) * mv /etc/apache2 /etc/apache2.mine * emerge -u apache # and optional module to test with * $EDITOR /etc/apache2/httpd.conf # and related, as needed * gdb /usr/sbin/apache2 | set args -X | run # now apache runs until you press CTRL-C or it hits an error (SEGFAULT) | backtrace # paste me exactly *that* output you get by this "backtrace" command | backtrace full # optionally paste me *this* output as well (as attachement, not inline). the "full" parameter to "backtrace" gives local stack frame infos as well. | quit. I modified this list a bit to fit our current needs. btw, the "connection truncated" messages may (maybe) exactly be these "SIGABRT" abort()'s. Lets see :)
Hi, I did use -X option, but apache still forks out under GDB. Allright this is what i did a) setup a fresh server. b) clean install of the software c) compiled apache etc as per ur directions. net outcome, same. Although Apache is compiled with debug option, still not real info in GDB! Now where do we go from here. >> root # gdb /usr/sbin/apache2 GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) set args -X -D AUTH_PAM -D SSL -D DAV -D SVN -d /usr/lib/apache2 -f /etc/apache2/httpd.conf (gdb) run Starting program: /usr/sbin/apache2 -X -D AUTH_PAM -D SSL -D DAV -D SVN -d /usr/lib/apache2 -f /etc/apache2/httpd.conf Detaching after fork from child process 7333. Program received signal SIGABRT, Aborted. 0xb7b4c971 in ?? () (gdb) backtrace #0 0xb7b4c971 in ?? () #1 0xb7c4c271 in ?? () #2 0x00001ca2 in ?? () #3 0x00000006 in ?? () #4 0xb7c52de0 in ?? () #5 0x00000006 in ?? () #6 0x00000002 in ?? () #7 0xbfffec34 in ?? () #8 0xb7c4c5eb in ?? () #9 0x00004000 in ?? () #10 0x00000006 in ?? () #11 0xb7c3ad4c in ?? () #12 0x00000006 in ?? () #13 0xbfffec4c in ?? () #14 0xb7b4c704 in ?? () #15 0x00000006 in ?? () #16 0xb7c4a3e2 in ?? () #17 0xb7c3ad4c in ?? () #18 0xb7c2bdaf in ?? () #19 0xbfffed78 in ?? () #20 0xb7b4dc2d in ?? () #21 0x00000006 in ?? () #22 0xbfffec60 in ?? () (gdb) emerge info Portage 2.0.51.19 (default-linux/x86/2005.0, gcc-3.3.5-20050130, glibc-2.3.4.20041102-r1, 2.6.10-gentoo-r6 i686) ================================================================= System uname: 2.6.10-gentoo-r6 i686 Pentium III (Coppermine) Gentoo Base System version 1.4.16 Python: dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 8 2005, 19:13:09)] dev-lang/python: 2.3.4-r1 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.8.5-r3, 1.5, 1.6.3, 1.7.9-r1, 1.4_p6, 1.9.4 sys-devel/binutils: 2.15.92.0.2-r7 sys-devel/libtool: 1.5.14 virtual/os-headers: 2.6.8.1-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O2 -mcpu=i686 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS=" -ggdb3" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks keeptemp keepwork nostrip sandbox sfperms" GENTOO_MIRRORS="http://portage.syd.ite.com.au/portage http://mirror.pacific.net.au/gentoo" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://portage.syd.ite.com.au/gentoo-portage" USE="x86 apache2 berkdb curl hardednedphp hardened kerberos ldap mmx ncurses nls pam perl python readline sse ssl tcpd winbind xml2 zlib" Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTDIR_OVERLAY emerge apache -vp These are the packages that I would merge, in order: Calculating dependencies ...done! [ebuild R ] net-www/apache-2.0.53 +apache2 +debug -doc +ldap -mpm-leader -mpm-peruser -mpm-prefork -mpm-threadpool -mpm-worker -no-suexec +ssl -static-modules -threads 0 kB Total size of downloads: 0 kB
All right, looked around somemore between different scenarios a) accessing repository using a Web Browser http://svn.example.com/testing -- 1 access request is made to the web browser for /testing -- 2 user Authetication attempts are recorded for it b) accessing repository using SVN client (tested with svn, tsvn) http://svn.exmaple.com/testing -- 5 PROPFIND attempts are recorded in the access_log -- about 60 user Authentication attempts are recorder /var/log/messages b4 this error surfaces. not sure if its worth any! *still very lost*
update: given this whole exercise with Mod_Auth_PAM + SVN has some what hit a dead end for me, had no other chice but Alternate Auth schemes. In my limited testing, Mod_Auth_Kerb+SVN+SSL appear to work fine, for instances where Mod_Auth_PAM+SVN+SSL use to fail. (Apache 2.0.53) Over next few days/week, I shall be stress testing the Mod_Auth_Kerb+SSL+SVN setup with SVN (Linux) / tSVN (Win32) Clients, and see if it fails. I still have alternate test environment available for Mod_Auth_pam+SSL+SVN, let me know if u folks need any more info. many thanks, for your time and effort. Shirish
this is really not funny. the valgrind logs, pastes and others are not helpfull. I am bit out of /dev/brain when being unable to locate that damn error, especially where I'm having the same config on my hosts (mod_auth_pam+subversion1.1.4+apache2.0.54) and svn/svk/tsvn/ankhsvn on the other end(s). Seriousely, I do not believe, that we get anything useful when we're having no real verbose - I mean, backtraces showing at least symbol names, and better fileName:filePosition location - output. Has anyone ever tried pinging upstream about this? (e.g. svn, mod auth modules, or glibc?)
Will it help, if i send you (.gz) my /usr/lib/apache2 folder along with apache binaries!? (i will need to sanitise some config files though). or files in /var/tmp/portage for apache and mod_auth_pam, any others? I can bzip them and send u a link to download from! I did follow the suggested steps to gather symbol information, but nothing came up in backtrace! in fact using -X switch under gdb still forks Apache, baffles me!!! yes, we had mod_auth_pam work with SVN for over 8 months, and still works fine with other websites (but SVN). Server got rebooted due to power outage and mod_auth_pam threw in this tantrum! Since, this is a live setup i had to look for plan-b, and mod_auth_kerb seems ok so far! As i said, i still have a test environment in place for Mod_auth_pam + Winbind + Apache + SVN ... feel free to suggest any further tests, etc.
Hi, made changes in make.conf as per http://gentoo-wiki.com/SECURITY_Debugging_with_Hardened_Gentoo I unmerged and emerge apr, apache, subversion, mod_auth_pam. gdb atleast turned in some more information this time, but doesnt look like its what u expect! any pointers folks!? regards Shirish >> (gdb) set args -X (gdb) run Starting program: /usr/sbin/apache2 -X [Thread debugging using libthread_db enabled] [New Thread -1213188432 (LWP 10559)] Detaching after fork from child process 10562. Program received signal SIGABRT, Aborted. [Switching to Thread -1213188432 (LWP 10559)] 0xffffe410 in ?? () (gdb) bt #0 0xffffe410 in ?? () #1 0xbfffeeac in ?? () #2 0x00000006 in ?? () #3 0x0000293f in ?? () #4 0xb7b2dbc1 in raise () from /lib/libc.so.6 #5 0xb7b2f3c7 in abort () from /lib/libc.so.6 #6 0xb7b271c8 in __assert_fail () from /lib/libc.so.6 #7 0xb7bf324a in _dl_close () from /lib/libc.so.6 #8 0xb7c12a88 in __libc_ptyname2 () from /lib/libc.so.6 #9 0x0000016d in ?? () #10 0xb7c12b33 in __libc_ptyname2 () from /lib/libc.so.6
Try to build apache and subversion with debugging symbols. Emerge them the following way: 'FEATURES=nostrip USE=debug CFLAGS=-g CXXFLAGS=-g emerge apache subversion' Then try again. Without debug symbols these backtraces are just gibberish.
Hey, folks, I'm seeing a very similar problem on my system, though not with Apache. If these comments are neither helpful nor appropriate, please ignore them with my apologies. I'm new to Gentoo and lost my USE flags during an 'emerge sync' last week (because I had put them in the wrong file). I set about recreating them (in the correct place this time). I added "ntpl" to the new USE flags which caused glibc to be recompiled. Now, when I run an app named amarok, which uses qt, I get the same error (comes from qt): amarokapp: dl-close.c:365: _dl_close: Assertion `imap->l_type == lt_loaded || imap->l_opencount > 0' failed. I tried recompiling glibc w/o ntpl, but the problem persists. Here's my system info: Portage 2.0.51.19 (default-linux/x86/2005.0, gcc-3.3.5-20050130, glibc-2.3.4.20041102-r1, 2.6.11.6 i686) ================================================================= System uname: 2.6.11.6 i686 AMD Athlon(tm) XP 2600+ Gentoo Base System version 1.4.16 Python: dev-lang/python-2.3.4-r1 [2.3.4 (#1, Apr 16 2005, 22:04:21)] dev-lang/python: 2.3.4-r1 sys-devel/autoconf: 2.59-r6, 2.13 sys-devel/automake: 1.7.9-r1, 1.8.5-r3, 1.5, 1.4_p6, 1.6.3, 1.9.4 sys-devel/binutils: 2.15.92.0.2-r7 sys-devel/libtool: 1.5.14 virtual/os-headers: 2.6.8.1-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O2 -march=athlon-xp -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=athlon-xp -fomit-frame-pointer -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms strict" GENTOO_MIRRORS="ftp://ftp.ussg.iu.edu/pub/linux/gentoo ftp://ftp.ucsb.edu/pub/mirrors/linux/gentoo/" LC_ALL="en_US.utf8" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.us.gentoo.org/gentoo-portage" USE="3dfx 3dnow X Xaw3d a52 aac aalib acpi aim alsa apm arts avi bash-completion berkdb bitmap-fonts bzip2 bzlib cdparanoia cdr crypt cups curl curlwrappers dga dio divx4linux dv dvd dvdr dvdread emacs emacs-w3 emboss encode esd ethereal fam firebird flac foomaticdb fortran ftp gdbm gif gnome gphoto2 gpm gtk gtk2 imap imlib java jpeg junit kde ldap libcaca libedit libg++ libwww mad mikmod mime mmx motif mozilla mp3 mpeg mule ncurses nls nptl ogg oggvorbis opengl oscar oss pam pda pdflib perl png posix postgres python qt quicktime readline real samba sasl scanner sdl sockets spell ssl svga szip tcpd threads tiff toolbar truetype truetype-fonts type1-fonts usb v4l vcd visualizations vorbis win32codecs wmf x86 xface xine xinerama xml2 xmms xpm xprint xscreensaver xv xvid yahoo zlib" Unset: ASFLAGS, CBUILD, CTARGET, LANG, LDFLAGS, LINGUAS, PORTDIR_OVERLAY Anyway, hope this helps.
All right, I rebuilt the environment and this time GDB does capture lot more information. (Thanks Paul comment #27) For what its worth, BT output is all below. If u need more details, feel free to ask! >> gdb /usr/sbin/apache2 GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) set args -X (gdb) run Starting program: /usr/sbin/apache2 -X [Thread debugging using libthread_db enabled] [New Thread -1212889424 (LWP 21292)] Program received signal SIGABRT, Aborted. [Switching to Thread -1212889424 (LWP 21292)] 0xffffe410 in __kernel_vsyscall () (gdb) (gdb) bt #0 0xffffe410 in __kernel_vsyscall () #1 0xb7b76bc1 in *__GI_raise (sig=6) at raise.c:64 #2 0xb7b78467 in *__GI_abort () at abort.c:88 #3 0xb7b701c8 in *__GI___assert_fail (assertion=0x6 <Address 0x6 out of bounds>, file=0x6 <Address 0x6 out of bounds>, line=6, function=0xb7c5bbd3 "_dl_close") at assert.c:83 #4 0xb7c3c2ea in *__GI__dl_close (_map=0x0) at dl-close.c:504 #5 0xb7c6ddcc in dlclose_doit (handle=0x0) at dlclose.c:36 #6 0xb7ff5c46 in _dl_catch_error (objname=0x80ea178, errstring=0x80ea17c, operate=0xb7c6ddb0 <dlclose_doit>, args=0x8194c48) at dl-error.c:161 #7 0xb7c6e2b1 in _dlerror_run (operate=0xb7c6ddb0 <dlclose_doit>, args=0x8194c48) at dlerror.c:162 #8 0xb7c6de0c in __dlclose (handle=0x6) at dlclose.c:47 #9 0xb7a95c96 in _pam_free_handlers () from /lib/libpam.so.0 #10 0xb799c4d1 in _pam_stack_cleanup () from /lib/security/pam_stack.so #11 0xb7a93d9b in _pam_free_data () from /lib/libpam.so.0 #12 0xb7a93226 in pam_end () from /lib/libpam.so.0 #13 0xb7aa0f59 in pam_auth_basic_user (r=0x81aa2b0) at mod_auth_pam.c:393 #14 0x08086b91 in ap_run_check_user_id (r=0x81aa2b0) at request.c:69 #15 0x0808764a in ap_process_request_internal (r=0x81aa2b0) at request.c:193 #16 0x08089949 in ap_sub_req_method_uri (method=0x809ed5e "GET", new_uri=0x81bb6a0 "/testing/!svn/bc/29/", r=0x8191330, next_filter=0x0) at request.c:1608 #17 0x08089991 in ap_sub_req_lookup_uri (new_uri=0x81bb6a0 "/testing/!svn/bc/29/", r=0x8191330, next_filter=0x0) at request.c:1620 #18 0xb7a83cfe in dav_do_prop_subreq (propdb=0x81bb6c8) at props.c:326 #19 0xb7a83e84 in dav_insert_coreprop (propdb=0x81bb6c8, propid=10000, name=0xb7a8c6e0 "getcontenttype", what=DAV_PROP_INSERT_VALUE, phdr=0xbffff580, inserted=0xbffff564) at props.c:386 #20 0xb7a84595 in dav_get_allprops (propdb=0x81bb6c8, what=DAV_PROP_INSERT_VALUE) at props.c:663 #21 0xb7a7d8ae in dav_propfind_walker (wres=0xbffff684, calltype=2) at mod_dav.c:1899 #22 0xb7a68264 in dav_svn_do_walk (ctx=0xbffff680, depth=1) at /var/tmp/portage/subversion-1.1.4/work/subversion-1.1.4/subversion/mod_dav_svn/repos.c:2615 #23 0xb7a687ce in dav_svn_walk (params=0xbffff750, depth=1, response=0xbffff74c) at /var/tmp/portage/subversion-1.1.4/work/subversion-1.1.4/subversion/mod_dav_svn/repos.c:2789 #24 0xb7a7dd17 in dav_method_propfind (r=0x8191330) at mod_dav.c:2031 #25 0xb7a832c8 in dav_handler (r=0x8191330) at mod_dav.c:4613 #26 0x0806adef in ap_run_handler (r=0x8191330) at config.c:152 #27 0x0806b57d in ap_invoke_handler (r=0x8191330) at config.c:364 #28 0x08067691 in ap_process_request (r=0x8191330) at http_request.c:249 #29 0x08060f73 in ap_process_http_connection (c=0x8182ec8) at http_core.c:251 #30 0x080776f4 in ap_run_process_connection (c=0x8182ec8) at connection.c:43 #31 0x08077b2c in ap_process_connection (c=0x8182ec8, csd=0x8182e10) at connection.c:176 #32 0x0806939c in child_main (child_num_arg=0) at prefork.c:610 #33 0x08069476 in make_child (s=0x80afe58, slot=0) at prefork.c:650 #34 0x080695d9 in startup_children (number_to_start=1) at prefork.c:722 #35 0x08069a5e in ap_mpm_run (_pconf=0x80ae0a8, plog=0x80d8150, s=0x80afe58) at prefork.c:941 #36 0x08071a7b in main (argc=2, argv=0xbffffb44) at main.c:618
That's weird, could you do the same debug output with authldap enabled and without authpam? This are my authldap lines in vhosts.conf: AuthLDAPEnabled on AuthLDAPURL ldap://10.0.0.2:389/dc=somecompany,dc=com?uid?sub?(objectClass=posixAccount)
I haven't had any errors yet with Auth_LDAP (authpam disabled). Its a test setup with a very small repo, and I am able to consistently reproduce error with mod_auth_pam + subversion. No errors with say mod_auth_pam+gforge or mod_auth_pam+websvn or mod_auth_pam+bugzilla . My LDAP config snippet (its against MS Active directory) AuthLDAPURL ldap://10.0.0.1:389/dc=example,dc=com,dc=au?sAMAccountName?sub AuthLDAPBindDN someuser@example.com.au AuthLDAPBindPassword ********** I cannot test it against big repos, as they all use mod_auth_kerb! sorry. ...hth.
I seems from the backtrace that the error is based in pam or libdl. In any case the backtrace is situated in the pam module unloading code.
I experienced exactly the same bug as described above with: apache-2.0.52-r1 mod_auth_pam-1.1.1-r1 openldap-2.1.30-r4 subversion-1.1.3 glibc-2.3.4.20041102-r1 Subversion is accessed via http+webdav and authenticates against ldap with mod_auth_pam. After tracking this down to be (probably) a glibc issue i installed glibc-2.3.3.20040420-r2. The server is running under heavy load now for some hours and so far it's working fine. Maybe it works with some newer versions of glibs too, but i have'nt tested this yet.
Same problem here: apache-2.0.54-r11 glibc-2.3.4.20041102-r1 subversion-1.2.0 mod_auth_pam-1.1.1 hardened-2.6-profile no ldap, however. Going back to glibc-2.3.3.20040420-r2 and recompiling apache and mod_auth_pam fixed it for me... On http://svn.haxx.se/users/archive-2005-04/0886.shtml this is considered a Gentoo problem only!? Regards, Alex!!!
I'm able to generate this same error using a pure Windows XP environment (server and client), so it's not a gentoo only situation. In my case it seems to be related to storing large .exe files in the repository. Our solution was just to remove those files--not a great workaround.
(In reply to comment #35) > I'm able to generate this same error using a pure Windows XP environment > (server and client), so it's not a gentoo only situation. > I believe the original error (in the #1 etc) was due to "glibc+pam" and not a subversion problem/bug. Please refer to the thread, #32 confirms it as well. Also, you should post to Subversion Mailing List (http://subversion.tigris.org) since your problem is stated to be related to Subversion's Win32 version.
Well, some test w/ subversion-1.3.2-r4 and 1.4.4-r2. Also, as noted above, bugs reproducible on Windows are definitely not Gentoo-specific and belong upstream. http://subversion.tigris.org/project_issues.html
