Bug 369025 – =sys-kernel/vanilla-sources-2.6.39: ssh through an IPSEC vpn fails at expecting SSH2_MSG_KEX_DH_GEX_GROUP

• Status: RESOLVED UPSTREAM
• Product: Gentoo Linux

[Description Anthony Basile 2011-05-28 15:03:28 UTC]

When trying to ssh from a box on one private IP subnet to another connected by a IPSEC VPN, ssh fails if the box is running vanilla 2.6.39, but it succeeds if it is running <2.6.39.  This does not happen to the 2.6.39 box when ssh-ing to hosts not connected through the IPSEC VPN --- that is to hosts on the same private IP subnet, or on public IP addrs.

Eg. My box has IP address 192.168.3.7.  Running vanilla 2.6.39 I try to ssh to 192.168.100.193.  It freezes with only the following output.  This same test succeed when running a 2.6.38 kernel.

blueness@yellowness ~ $ ssh -vvv 192.168.100.193
OpenSSH_5.8p1-hpn13v10lpk, OpenSSL 1.0.0d 8 Feb 2011
debug1: Reading configuration data /home/blueness/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.100.193 [192.168.100.193] port 22.
debug1: Connection established.
debug1: identity file /home/blueness/.ssh/id_rsa type -1
debug1: identity file /home/blueness/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/blueness/.ssh/id_dsa" as a RSA1 public key
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/blueness/.ssh/id_dsa type 2
debug1: identity file /home/blueness/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8p1-hpn13v10
debug1: match: OpenSSH_5.8p1-hpn13v10 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1-hpn13v10lpk
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.168.100.193" from file "/home/blueness/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/blueness/.ssh/known_hosts:221
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [e-mail],[e-mail],ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: AUTH STATE IS 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [e-mail],[e-mail],ssh-rsa,[e-mail],[e-mail],ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[e-mail],zlib
debug2: kex_parse_kexinit: none,[e-mail],zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[e-mail]
debug2: kex_parse_kexinit: none,[e-mail]
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP



Reproducible: Always

Steps to Reproduce:
1. Set up two private subnets, A and B, initially isolated.
2. Set up two IPSec routers on each subnet, configured as usual.  These are linux boxes running <2.6.39
3. On any box on private subnet A running 2.6.39, try to ssh a box on subnet B.  It fails.  (Similar failure when ssh-ing from B to A when running 2.6.39).
4. On the same box, boot into a pre 2.6.39, try to ssh to the same remote box, it succeeds.

[Comment 1 Anthony Basile 2011-05-28 15:05:12 UTC]

This happens on both amd64 and i686.  I'm doing the git bisect now and will submit upstream.

[Comment 2 Anthony Basile 2011-05-28 21:09:18 UTC]

The culprit is commit 2c8cec5c10bced2408082a6656170e74ac17231c.  Its summary says:

    ipv4: Cache learned PMTU information in inetpeer.

So messing with the MTU info messes up when tunneling via an IPSEC vpn or possibly other tunnels.  This is consistent with the misbehavior of ssh because it is known that mismatched MTU's which cause fragmentation further cause ssh to freeze at "expecting SSH2_MSG_KEX_DH_GEX_GROUP".  For reference see

    http://www.snailbook.com/faq/mtu-mismatch.auto.html

I'm working on a reverse patch right now.  Simple revert doesn't apply cleanly.  I'll bug upstream.

[Comment 3 Anthony Basile 2011-05-28 22:00:54 UTC]

Okay reading the code carefully, I see that

   echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc

fixes the problem.  So I'm not sure this is a "bug" or a new behavior that ipsec users should be aware of.  Let's let upstream decide.

[Comment 4 Anthony Basile 2011-05-29 15:52:26 UTC]

(In reply to comment #3)
> Okay reading the code carefully, I see that
> 
>    echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc
> 
> fixes the problem.  So I'm not sure this is a "bug" or a new behavior that
> ipsec users should be aware of.  Let's let upstream decide.

And it breaks other things, such as browsing some web sites.

[Comment 5 Anthony Basile 2011-06-05 14:23:29 UTC]

(In reply to comment #4)
> (In reply to comment #3)
> > Okay reading the code carefully, I see that
> > 
> >    echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc
> > 
> > fixes the problem.  So I'm not sure this is a "bug" or a new behavior that
> > ipsec users should be aware of.  Let's let upstream decide.
> 
> And it breaks other things, such as browsing some web sites.

echo 1 > tcp_mtu_probing

Fixes the ssh via ipsec problem and does not break other things.  At this point I'm not sure if this is a "bug" or a new behavior that people setting up ipsec tunnels should be aware of?

Any feelings from the kernel@ people?

[Comment 6 Mike Pagano 2011-06-07 18:06:19 UTC]

We have a workaround, we can watch the upstream bug and determine if something comes from that where we need to do some work