The client programs from net-nds/openldap-2.4.24 on hardened x86 are crashing. It only seems to be the client programs like ldapwhoami, ldapmodify and ldapsearch. IIRC the server runs okay. Note that the programs successfully execute, but crash towards the end. The programs crash in different ways depending on which PaX flags I enable. Reproducible: Always Steps to Reproduce: 1. Use fully hardened kernel and toolchain. 2. Emerge =net-nds/openldap-2.4.24 3. Run ldapwhoami, ldapmodify or ldapsearch. Actual Results: Programs run fine but crash near the end. Expected Results: Programs run fine but don't crash near the end. Portage 2.1.9.42 (hardened/linux/x86, gcc-4.4.5, glibc-2.11.3-r0, 2.6.36-hardened-r9 i686) ================================================================= System uname: Linux-2.6.36-hardened-r9-i686-AMD_Athlon-tm-_XP_1800+-with-gentoo-2.0.1 Timestamp of tree: Sat, 19 Mar 2011 14:15:01 +0000 app-shells/bash: 4.1_p9 dev-lang/python: 2.6.6-r2, 3.1.3-r1 sys-apps/baselayout: 2.0.1-r1 sys-apps/openrc: 0.7.0 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.65-r1 sys-devel/automake: 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.81-r2 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA PUEL" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=athlon-xp -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/local /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa" CONFIG_PROTECT_MASK="" CXXFLAGS="-march=athlon-xp -O2 -pipe" DISTDIR="/mnt/diamond/usr/portage-distfiles" FEATURES="binpkg-logs collision-protect distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_GB en_US de fr he ja ru zh_CN zh_TW" MAKEOPTS="-j1" PKGDIR="/mnt/diamond/usr/portage-packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.au.gentoo.org/gentoo-portage" USE="3dnow 3dnowext acl acpi alsa apache2 audit bash-completion berkdb boehm-gc boost bzip2 caps ccache cdda chroot cli cracklib crypt curl cxx directfb emacs expat fbdev finger gcrypt gdbm gmp gopher gpg gpm hardened iconv icu idn imap iproute2 ipv6 javascript justify kerberos ldap libssh2 lua mbox mmx mmxext mng modules mudflap ncurses nfs nls nptl nptlonly ntp pam pam_krb5 pcre perl pic pop postgres pppd python readline samba sasl savedconfig secure-delete session smime smtp sockets socks5 sql sqlite sqlite3 sse ssl sysfs syslog system-sqlite tcl tcpd threads threadsafe tproxy unicode urandom wcwidth x86 xattr xft xinetd zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias asis auth_basic auth_digest authn_alias authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cern_meta cgi charset_lite dav dav_fs dav_lock dbd deflate dir disk_cache dumpio env expires ext_filter file_cache filter headers ident imagemap include info log_config log_forensic logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_ftp proxy_http rewrite setenvif speling status substitute unique_id userdir usertrack version vhost_alias" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="mouse keyboard evdev" KERNEL="linux" LCD_DEVICES="ncurses text" LINGUAS="en en_GB en_US de fr he ja ru zh_CN zh_TW" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="vga vesa nv fbdev" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
Created attachment 266673 [details] core file
Created attachment 268987 [details] ldapwhoami binary
Created attachment 268991 [details] kernel config
Created attachment 268995 [details] ldapwhoami error
Here are some more details.
Sorry for the delay in responding, but I had to set up an ldap server with sasl and get my system as close to yours as possible. I'm sorry but I just can't hit this. I tried all three functions and nothing. I even tried gcc-4.5.2. Here's my system. hard-thirtytwo ~ # emerge --info openldap Portage 2.1.9.42 (hardened/linux/x86, gcc-4.4.5, glibc-2.11.3-r0, 2.6.36-hardened-r9 i686) ================================================================= System Settings ================================================================= System uname: Linux-2.6.36-hardened-r9-i686-Intel-R-_Core-TM-_i7_CPU_920_@_2.67GHz-with-gentoo-2.0.2 Timestamp of tree: Thu, 07 Apr 2011 07:00:01 +0000 app-shells/bash: 4.1_p9 dev-lang/python: 2.6.6-r2, 3.1.3-r1 dev-util/cmake: 2.8.1-r2 sys-apps/baselayout: 2.0.2 sys-apps/openrc: 0.8.0 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.65-r1 sys-devel/automake: 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5, 4.5.2 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.81-r2 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=i686 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=i686 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="ftp://192.168.100.9/pub/gentoo" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/blueness /var/lib/layman/hardened-development" SYNC="rsync://192.168.100.7/portage" USE="acl berkdb bzip2 cli cracklib crypt cups cxx dri gdbm gpm hardened iconv modules mudflap ncurses nls nptl nptlonly openmp pam pcre perl pic pppd python readline session ssl sysfs tcpd urandom x86 xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS ================================================================= Package Settings ================================================================= net-nds/openldap-2.4.24 was built with the following: USE="berkdb crypt perl ssl tcpd -cxx -debug -experimental -gnutls -icu -iodbc -ipv6 -kerberos -minimal -odbc -overlays -samba -sasl (-selinux) -slp -smbkrb5passwd -syslog" CFLAGS="-O2 -march=i686 -pipe -D_GNU_SOURCE" CXXFLAGS="-O2 -march=i686 -pipe -D_GNU_SOURCE"
@reporter. Can you give me the last few lines of emerge --info openldap so I can see what USE flags you used.
(client system) net-nds/openldap-2.4.23 was built with the following: USE="berkdb crypt icu ipv6 kerberos minimal perl sasl ssl syslog tcpd -cxx -debug -experimental -gnutls -iodbc -odbc -overlays -samba (-selinux) -slp -smbkrb5passwd" CFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" CXXFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" (server system) net-nds/openldap-2.4.23 was built with the following: USE="berkdb crypt icu ipv6 kerberos odbc perl samba sasl ssl syslog tcpd -cxx -debug -experimental -gnutls -iodbc -minimal -overlays (-selinux) -slp -smbkrb5passwd" CFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" CXXFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" (note that I'm using a new kernel since 2011-04-08) System uname: Linux-2.6.38-hardened-i686-AMD_Athlon-tm-_XP_1800+-with-gentoo-2.0.2
(In reply to comment #8) > (client system) > net-nds/openldap-2.4.23 was built with the following: > USE="berkdb crypt icu ipv6 kerberos minimal perl sasl ssl syslog tcpd -cxx > -debug -experimental -gnutls -iodbc -odbc -overlays -samba (-selinux) -slp > -smbkrb5passwd" > CFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" > CXXFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" > > (server system) > net-nds/openldap-2.4.23 was built with the following: > USE="berkdb crypt icu ipv6 kerberos odbc perl samba sasl ssl syslog tcpd -cxx > -debug -experimental -gnutls -iodbc -minimal -overlays (-selinux) -slp > -smbkrb5passwd" > CFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" > CXXFLAGS="-march=athlon-xp -O2 -pipe -D_GNU_SOURCE" > > (note that I'm using a new kernel since 2011-04-08) > System uname: > Linux-2.6.38-hardened-i686-AMD_Athlon-tm-_XP_1800+-with-gentoo-2.0.2 I *still* can't hit this even with your USE flags. Sorry to ask you to do the following, but I'm stuck. From easiest to hardest: 0) Post the crash at the end of, say ldapsearch. Post more information about your kernel-config. Post any other system info that may be relevant. 1) Hit the crash and look at the tail of your dmesg. See if there's a clue. Post. 2) Run ldapsearch -d 255. See if there's a clue and post your debug messages. 3) Get a full strace of the crash: strace -f ldapsearch 4) Recompile with CFLAGS+="-ggdb", run ldapsearch within gdb, hit the fault, and provide a backtrace with bt. 5) Generate a core file and pass that along.
Sure thing and thanks for taking the time to look into this. My kernel config is still the same as this attachment https://bugs.gentoo.org/attachment.cgi?id=268991 so I won't upload it again. Other relevant information concerning my LDAP set up is: authenticate through SASL with GSSAPI mechanism and using start TLS encryption on the connections. "dmesg" shows absolutely no errors, kernel.grsecurity.dmesg and other Grsec logging options are all on. I've previously turned on all the OpenLDAP system log levels and there doesn't seem to be anything seriously wrong there. Nonetheless I have provided the details you've asked for, particularly the strace which I never thought to do for some reason. The GDB and Valgrind tests (which pipacs suggested) I will look into later today, I'll have to build those packages first.
Created attachment 270443 [details] ldapwhoami strace log Strace log from "ldapwhoami" run as root.
Created attachment 270447 [details] stdout/stderr from ldapwhoami. Results printed to stdout/stderr from running the command "ldapwhoami -d 255" as root.
The dumped core file is too big to upload (1.1MB) sorry.
Backtrace from the core file. We're missing some symbols but the badness appears to happen in ber_memrealloc_x () which is defined in libraries/liblber/memory.c around line 304. #0 0x50331422 in __kernel_vsyscall () (gdb) bt #0 0x50331422 in __kernel_vsyscall () #1 0x5019c441 in raise () from /lib/libc.so.6 #2 0x5019db82 in abort () from /lib/libc.so.6 #3 0x501d7a7d in ?? () from /lib/libc.so.6 #4 0x501dd931 in ?? () from /lib/libc.so.6 #5 0x501debbe in ?? () from /lib/libc.so.6 #6 0x502dc008 in ber_memrealloc_x () from /usr/lib/liblber-2.4.so.2 #7 0x5030a52e in ldap_create_page_control_value () from /usr/lib/libldap-2.4.so.2 #8 0x5030a5d2 in ldap_create_page_control () from /usr/lib/libldap-2.4.so.2 #9 0x4fcef2b2 in ?? () from /usr/lib/libldap_r-2.4.so.2 #10 0x4fccb7fe in ldap_pvt_thread_rmutex_unlock () from /usr/lib/libldap_r-2.4.so.2 #11 0x4fd01b30 in ?? () from /usr/lib/libldap_r-2.4.so.2 #12 0x5034527e in ?? () from /lib/ld-linux.so.2 #13 0x50345d47 in ?? () from /lib/ld-linux.so.2 #14 0x4ff70cc4 in ?? () from /lib/libdl.so.2 #15 0x503402e6 in ?? () from /lib/ld-linux.so.2 #16 0x4ff710bc in ?? () from /lib/libdl.so.2 #17 0x4ff70cfa in dlclose () from /lib/libdl.so.2 #18 0x502ccf91 in sasl_seterror () from /usr/lib/libsasl2.so.2 #19 0x502c5030 in sasl_done () from /usr/lib/libsasl2.so.2 #20 0x11f49a92 in ?? () #21 0x11f45bfb in main ()
The crash disappears when I build cyrus-sasl without the postgres USE flag.
You need to to have only one of the libldap/libldap_r stuff linked in. Trace where the libldap link is coming from, and fix that package (I suspect postgres based on what you've said). #8 0x5030a5d2 in ldap_create_page_control () from /usr/lib/libldap-2.4.so.2 #9 0x4fcef2b2 in ?? () from /usr/lib/libldap_r-2.4.so.2
I'm testing 2.4.28-r1 now (SASL built without PostgreSQL support) and it's working fine on both x86 and amd64. Do you want me to rebuild Cyrus-SASL with the postgres USE flag and see if that bug still occurs?
(In reply to comment #17) > I'm testing 2.4.28-r1 now (SASL built without PostgreSQL support) and it's > working fine on both x86 and amd64. Do you want me to rebuild Cyrus-SASL > with the postgres USE flag and see if that bug still occurs? I am already running openldap w/ sasl w/ postgres use flag and am having no issues on amd64 (no-multilib).
closing old bugs, reopen if it still breaks on an up to date system
