I cannot ssh onto my Gentoo box from external computers, all incoming connections are terminated. For every attempt I find the following line in dmesg: sshd[14000]: segfault at b77373f5 ip b771dcc7 sp bfb63a50 error 7 in libcrypto.so.1.0.0[b7682000+141000] I am using net-misc/openssh-5.5_p1_r2 with libcrypto coming from dev-libs/openssl-1.0.0a-r3 (both are the stable versions). I have the same problem when I downgrade to openssl-0.9.8o (the are no older versions in the portage tree). I also tried openssh versions 5.3_p1-r1 and ~5.6_p1-r2, with the same symptoms. (This only happens when using interactive password authentication, with RSA public key authentication there is no crash, but I got a "Received disconnect from 9.56.46.207: 2: Invalid ssh2 packet type: 192" error on the client side.) Thanks for looking into this, let me know if you need any more info. Reproducible: Always Steps to Reproduce: 1. start sshd on local Gentoo box 2. ssh from external machine to Gentoo box (using keyboard-interactive login) 3. Type in password when prompted on external machine Actual Results: Server disconnects, crash in libcrypto.so Expected Results: Ssh session is established Here is the sshd log (obtained by manually running /usr/bin/sshd -D -d -d -d instead of starting the /etc/init.d/sshd service): ------------------ debug2: load_server_config: filename /etc/ssh/sshd_config debug2: load_server_config: done config len = 238 debug2: parse_server_config: config /etc/ssh/sshd_config len 238 debug3: /etc/ssh/sshd_config:59 setting PasswordAuthentication no debug3: /etc/ssh/sshd_config:84 setting UsePAM yes debug3: /etc/ssh/sshd_config:90 setting X11Forwarding yes debug3: /etc/ssh/sshd_config:93 setting PrintMotd no debug3: /etc/ssh/sshd_config:94 setting PrintLastLog no debug3: /etc/ssh/sshd_config:112 setting Subsystem sftp /usr/lib/misc/sftp-server debug1: sshd version OpenSSH_5.5p1 debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key. debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key. debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: rexec_argv[0]='/usr/sbin/sshd' debug1: rexec_argv[1]='-D' debug1: rexec_argv[2]='-d' debug1: rexec_argv[3]='-d' debug1: rexec_argv[4]='-d' debug3: oom_adjust_setup Set /proc/self/oom_adj from 0 to -17 debug2: fd 3 setting O_NONBLOCK debug1: Bind to port 22 on 0.0.0.0. Server listening on 0.0.0.0 port 22. socket: Address family not supported by protocol debug3: fd 4 is not O_NONBLOCK debug1: Server will not fork when running in debugging mode. debug3: send_rexec_state: entering fd = 7 config len 238 debug3: ssh_msg_send: type 0 debug3: send_rexec_state: done debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7 debug1: inetd sockets after dupping: 3, 3 Connection from 9.56.165.36 port 33053 debug1: Client protocol version 2.0; client software version OpenSSH_5.4 debug1: match: OpenSSH_5.4 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.5 debug2: fd 3 setting O_NONBLOCK debug2: Network child is on pid 14647 debug3: preauth child monitor started debug3: mm_request_receive entering debug3: privsep user:group 22:22 debug1: permanently_set_uid: 22/22 debug1: list_hostkey_types: ssh-rsa,ssh-dss debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-md5 debug1: kex: client->server aes128-ctr hmac-md5 none debug2: mac_setup: found hmac-md5 debug1: kex: server->client aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received debug3: mm_request_send entering: type 0 debug3: monitor_read: checking request 0 debug3: mm_answer_moduli: got parameters: 1024 1024 8192 debug3: mm_request_send entering: type 1 debug2: monitor_read: 0 used once, disabling now debug3: mm_request_receive entering debug3: mm_choose_dh: waiting for MONITOR_ANS_MODULI debug3: mm_request_receive_expect entering: type 1 debug3: mm_request_receive entering debug3: mm_choose_dh: remaining 0 debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent debug2: dh_gen_key: priv key bits set: 126/256 debug2: bits set: 495/1024 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT debug2: bits set: 535/1024 debug3: mm_key_sign entering debug3: mm_request_send entering: type 4 debug3: monitor_read: checking request 4 debug3: mm_answer_sign debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN debug3: mm_request_receive_expect entering: type 5 debug3: mm_request_receive entering debug3: mm_answer_sign: signature 0x80c2a50(271) debug3: mm_request_send entering: type 5 debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: monitor_read: 4 used once, disabling now debug3: mm_request_receive entering debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: KEX done debug1: userauth-request for user drazse service ssh-connection method none debug1: attempt 0 failures 0 debug3: mm_getpwnamallow entering debug3: mm_request_send entering: type 6 debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM debug3: mm_request_receive_expect entering: type 7 debug3: mm_request_receive entering debug3: monitor_read: checking request 6 debug3: mm_answer_pwnamallow debug3: Trying to reverse map address 9.56.165.36. debug2: parse_server_config: config reprocess config len 238 debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1 debug3: mm_request_send entering: type 7 debug2: input_userauth_request: setting up authctxt for drazse debug2: monitor_read: 6 used once, disabling now debug3: mm_start_pam entering debug3: mm_request_send entering: type 45 debug3: mm_inform_authserv entering debug3: mm_request_send entering: type 3 debug2: input_userauth_request: try method none debug3: mm_request_receive entering debug3: monitor_read: checking request 45 debug1: PAM: initializing for "drazse" debug1: userauth-request for user drazse service ssh-connection method publickey debug1: attempt 1 failures 0 debug2: input_userauth_request: try method publickey debug1: test whether pkalg/pkblob are acceptable debug3: mm_key_allowed entering debug3: mm_request_send entering: type 20 debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED debug3: mm_request_receive_expect entering: type 21 debug3: mm_request_receive entering debug1: PAM: setting PAM_RHOST to "pacers.pok.ibm.com" debug1: PAM: setting PAM_TTY to "ssh" debug2: monitor_read: 45 used once, disabling now debug3: mm_request_receive entering debug3: monitor_read: checking request 3 debug3: mm_answer_authserv: service=ssh-connection, style= debug2: monitor_read: 3 used once, disabling now debug3: mm_request_receive entering debug3: monitor_read: checking request 20 debug3: mm_answer_keyallowed entering debug3: mm_answer_keyallowed: key_from_blob: 0x80c50f8 debug1: temporarily_use_uid: 1000/1000 (e=0/0) debug1: trying public key file /home/drazse/.ssh/authorized_keys debug1: fd 4 clearing O_NONBLOCK debug3: secure_filename: checking '/home/drazse/.ssh' debug3: secure_filename: checking '/home/drazse' debug3: secure_filename: terminating check at '/home/drazse' debug1: restore_uid: 0/0 debug2: key not found debug1: temporarily_use_uid: 1000/1000 (e=0/0) debug1: trying public key file /home/drazse/.ssh/authorized_keys2 debug1: restore_uid: 0/0 Failed publickey for drazse from 9.56.165.36 port 33053 ssh2 debug3: mm_answer_keyallowed: key 0x80c50f8 is not allowed debug3: mm_request_send entering: type 21 debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa debug3: mm_request_receive entering debug1: userauth-request for user drazse service ssh-connection method keyboard-interactive debug1: attempt 2 failures 1 debug2: input_userauth_request: try method keyboard-interactive debug1: keyboard-interactive devs debug1: auth2_challenge: user=drazse devs= debug1: kbdint_alloc: devices 'pam' debug2: auth2_challenge_start: devices pam debug2: kbdint_next_device: devices <empty> debug1: auth2_challenge_start: trying authentication method 'pam' debug3: mm_sshpam_init_ctx debug3: mm_request_send entering: type 48 debug3: mm_sshpam_init_ctx: waiting for MONITOR_ANS_PAM_INIT_CTX debug3: mm_request_receive_expect entering: type 49 debug3: mm_request_receive entering debug3: monitor_read: checking request 48 debug3: mm_answer_pam_init_ctx debug3: PAM: sshpam_init_ctx entering debug3: mm_request_send entering: type 49 debug3: mm_sshpam_query debug3: mm_request_send entering: type 50 debug3: mm_sshpam_query: waiting for MONITOR_ANS_PAM_QUERY debug3: mm_request_receive_expect entering: type 51 debug3: mm_request_receive entering debug3: mm_request_receive entering debug3: monitor_read: checking request 50 debug3: mm_answer_pam_query debug3: PAM: sshpam_query entering debug3: ssh_msg_recv entering debug3: PAM: sshpam_thread_conv entering, 1 messages debug3: ssh_msg_send: type 1 debug3: ssh_msg_recv entering debug3: mm_request_send entering: type 51 debug3: mm_sshpam_query: pam_query returned 0 Postponed keyboard-interactive for drazse from 9.56.165.36 port 33053 ssh2 debug3: mm_request_receive entering debug3: mm_sshpam_respond debug3: mm_request_send entering: type 52 debug3: mm_sshpam_respond: waiting for MONITOR_ANS_PAM_RESPOND debug3: mm_request_receive_expect entering: type 53 debug3: mm_request_receive entering debug3: monitor_read: checking request 52 debug3: mm_answer_pam_respond debug2: PAM: sshpam_respond entering, 1 responses debug3: ssh_msg_send: type 6 debug3: mm_request_send entering: type 53 debug3: mm_sshpam_respond: pam_respond returned 1 debug3: mm_sshpam_query debug3: mm_request_send entering: type 50 debug3: mm_sshpam_query: waiting for MONITOR_ANS_PAM_QUERY debug3: mm_request_receive_expect entering: type 51 debug3: mm_request_receive entering debug3: mm_request_receive entering debug3: monitor_read: checking request 50 debug3: mm_answer_pam_query debug3: PAM: sshpam_query entering debug3: ssh_msg_recv entering debug1: do_cleanup debug3: PAM: sshpam_thread_cleanup entering ---------------------- Here is the tail end of an strace log for the same sshd session (password sanitized) --------------- read(4, "\1\0\0\0\nPassword: ", 15) = 15 write(2, "debug3: mm_request_send entering"..., 43) = 43 write(5, "\0\0\0#3", 5) = 5 write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\nPassword: \0\0"..., 34) = 34 write(2, "debug3: mm_request_receive enter"..., 37) = 37 read(5, "\0\0\0\17", 4) = 4 read(5, "4\0\0\0\1\0\0\0\6******", 15) = 15 write(2, "debug3: monitor_read: checking r"..., 43) = 43 write(2, "debug3: mm_answer_pam_respond\r\n", 31) = 31 write(2, "debug2: PAM: sshpam_respond ente"..., 51) = 51 write(2, "debug3: ssh_msg_send: type 6\r\n", 30) = 30 write(4, "\0\0\0\v\6", 5) = 5 write(4, "\0\0\0\6******", 10) = 10 write(2, "debug3: mm_request_send entering"..., 43) = 43 write(5, "\0\0\0\0055", 5) = 5 write(5, "\0\0\0\1", 4) = 4 write(2, "debug3: mm_request_receive enter"..., 37) = 37 read(5, "\0\0\0\1", 4) = 4 read(5, "2", 1) = 1 write(2, "debug3: monitor_read: checking r"..., 43) = 43 write(2, "debug3: mm_answer_pam_query\r\n", 29) = 29 write(2, "debug3: PAM: sshpam_query enteri"..., 36) = 36 write(2, "debug3: ssh_msg_recv entering\r\n", 31) = 31 read(4, 0xbf9bfcd8, 4) = ? ERESTARTSYS (To be restarted) --- SIGCHLD (Child exited) @ 0 (0) --- rt_sigaction(SIGCHLD, NULL, {0x806e551, [], 0}, 8) = 0 rt_sigaction(SIGCHLD, {SIG_DFL, [], 0}, NULL, 8) = 0 waitpid(14648, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV}], WNOHANG) = 14648 exit_group(1) = ? -------------
Did you forget to run `revdep-rebuild --library libcrypto.so.0.9.8` and `revdep-rebuild --library libssl.so.0.9.8` after openssl-1.0.0a-r3 upgrade? If yes, please try that first. If no, please attach the missing `emerge --info` here.
No, I ran the two revdep-rebuild --library commands before, and removed the older .so files, so I currently have only the 1.0.0 version for libssl.so and libcrypto.so. Running again the two commands yielded: # revdep-rebuild --library libcrypto.so.0.9.8 ... There are no dynamic links to libcrypto.so.0.9.8. and # revdep-rebuild --library libssl.so.0.9.8 ... There are no dynamic links to libssl.so.0.9.8. Here is the missing emerge --info --------- $ emerge --info Portage 2.1.8.3 (default/linux/x86/10.0/desktop/kde, gcc-4.4.4, glibc-2.11.2-r0, 2.6.34-gentoo-r6 i686) ================================================================= System uname: Linux-2.6.34-gentoo-r6-i686-Intel-R-_Core-TM-2_Duo_CPU_T7700_@_2.40GHz-with-gentoo-1.12.13 Timestamp of tree: Thu, 28 Oct 2010 13:30:01 +0000 ccache version 2.4 [disabled] app-shells/bash: 4.1_p7 dev-java/java-config: 2.1.11 dev-lang/python: 2.6.5-r3, 3.1.2-r4 dev-util/ccache: 2.4-r7 dev-util/cmake: 2.8.1-r2 sys-apps/baselayout: 1.12.13 sys-apps/sandbox: 2.3-r1 sys-devel/autoconf: 2.13, 2.65-r1 sys-devel/automake: 1.7.9-r1, 1.8.5-r3, 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 3.4.6-r2, 4.1.2, 4.3.4, 4.4.4-r2 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.81-r2 virtual/os-headers: 2.6.30-r1 ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="*" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=nocona -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=nocona -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LC_ALL="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/ibm-internal-cvs /usr/portage/local" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi afs alsa audiofile avi bash-completion berkdb bluetooth branding bzip2 cairo cdda cddb cdr cli consolekit cracklib crypt cups cxx dbus dirac dri dts dvd dvdr dvdread emboss encode esd exif fam ffmpeg firefox flac font-server fontconfig foomaticdb fortran freetype gd gdbm gif gmp gnutls gphoto2 gpm gtk hal iconv imlib ipv6 java jpeg jpeg2k kde kdehiddenvisibility kerberos kpathsea lcms libnotify mad matroska mikmod mmx mmxext mng modules motif mozbranding mp3 mp4 mpeg mudflap ncurses networkmanager nls nodrm nptl nptlonly ogg oggvorbis opengl openmp oss pam pango pcre pdds pdf perl png policykit ppds pppd python qt3support qt4 readline sasl sdl session speex spell sse sse2 ssl ssse3 startup-notification svg sysfs tcpd tetex theora tiff truetype unicode usb v4l v4l2 vcd vidix vlm vorbis win32codecs wmp wxwindows x264 x86 xcb xml xorg xosd xulrunner xv xvid xvmc zlib" ALSA_CARDS="snd_hda_intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-2" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nv nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Ok, I think I found the cause of the segfault. After some more digging I found the following lines in /var/log/auth.log: Nov 1 09:55:13 viktor authdaemond: PAM adding faulty module: /lib/security/pam_gsa.so Nov 1 09:55:53 viktor authdaemond: PAM unable to dlopen(/lib/security/pam_gsa.so): /lib/security/pam_gsa.so: undefined symbol: pam_get_item pam_gsa.so belongs to the net-fs/ibm-gsa-client-2.7.3-r1 package installed from an portage overlay tree. After commenting out the following line from /etc/pam.d/system-auth auth sufficient /lib/security/pam_gsa.so I do not get the segfault anymore, so I think the problem is in the ibm-gsa-client package (it's a binary package, probably uses an older PAM API). I think this can be closed now, not sure if resolution should be FIXED or INVALID.
