Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 664384 - net-misc/openssh-7.7p1: openssh private key decryption breakage
Summary: net-misc/openssh-7.7p1: openssh private key decryption breakage
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo's Team for Core System packages
URL: https://bugzilla.mindrot.org/show_bug...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-08-23 19:44 UTC by Alexander Sergeyev
Modified: 2018-08-29 19:05 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
an example of failing key (testkey_gceSwXMT_iter00000652,444 bytes, text/plain)
2018-08-26 09:39 UTC, Alexander Sergeyev
Details
find_bad_key.sh (find_bad_key.sh,641 bytes, text/plain)
2018-08-26 14:37 UTC, Alexander Sergeyev
Details
find_bad_key.sh (v2) (find_bad_key.sh,835 bytes, text/plain)
2018-08-26 16:43 UTC, Alexander Sergeyev
Details
find_bad_key.sh (v3) (find_bad_key.sh,749 bytes, text/plain)
2018-08-26 17:47 UTC, Alexander Sergeyev
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Sergeyev 2018-08-23 19:44:52 UTC
I've just been hit by not being unable to connect to a server via ssh due to:

$ ssh ...
Load key "/redacted/path": invalid format
johndoe@1.2.3.4: Permission denied (publickey).

The private key is encrypted with a passphrase, other openssh utilities fail similarly:

$ ssh-keygen -p -f ./key
Failed to load key ./key: invalid format

The bug is somewhat subtle: it's always reproducible on the offending key, but not on every encrypted key.
Downgrading to dev-libs/openssl-1.0.2o-r6 fixes the problem.

Versions and use-flags:
net-misc/openssh-7.7_p1-r6:0 hpn pam pie ssl -X -X509 -audit -bindist -debug -kerberos -ldap -ldns -libedit -libressl -livecd -sctp -selinux -skey -static -test ABI_MIPS="-n32" KERNEL="linux"
dev-libs/openssl-1.0.2p:0 asm zlib -bindist -gmp -kerberos -rfc3779 -sctp -sslv2 -sslv3 -static-libs -test -tls-heartbeat -vanilla ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="64 -32 -x32" CPU_FLAGS_X86="sse2"
Comment 1 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 07:37:38 UTC
> it's always reproducible on the offending key, but not on every encrypted key

We need details about the non-working key. Can you provide details how we can generate such a key?
Comment 2 Alexander Sergeyev 2018-08-26 08:36:24 UTC
> Can you provide details how we can generate such a key?

It's a ecdsa-sha2-nistp521 key with additional passphrase encryption (AES-128-CBC).

I cannot share the original material without implied access revokation, etc, which I would prefer to avoid. Currently, I'm trying to re-create behaviour with no sensitive data in play.
Comment 3 Alexander Sergeyev 2018-08-26 09:39:41 UTC
Created attachment 545034 [details]
an example of failing key
Comment 4 Alexander Sergeyev 2018-08-26 09:41:57 UTC
I've added an example key that fails to load. The passphrase is in between underscores [of the filename]. But it's not really needed to observe the failure:

$ ssh-keygen -y -f ./testkey_gceSwXMT_iter00000652 </dev/null
Load key "./testkey_gceSwXMT_iter00000652": invalid format
Comment 5 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 11:25:54 UTC
I am only asking for the command to re-create such a key.
Comment 6 Alexander Sergeyev 2018-08-26 12:09:09 UTC
(In reply to Thomas Deutschmann from comment #5)
> I am only asking for the command to re-create such a key.

Here it is:

$ ssh-keygen -t ecdsa -b 521
Comment 7 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 13:20:07 UTC
I cannot reproduce:

I downgraded a test box to openssl-1.0.2o-r6.
I have rebuilt openssh-7.7_p1-r9 on that box, too (just to be sure):

> openssl-1-0-2-o-r6 ~ # openssl version
> OpenSSL 1.0.2o  27 Mar 2018
> openssl-1-0-2-o-r6 ~ # mkdir /tmp/ssh-test
> openssl-1-0-2-o-r6 ~ # cd /tmp/ssh-test/
> openssl-1-0-2-o-r6 /tmp/ssh-test # ssh-keygen -t ecdsa -b 521 -f id_test
> Generating public/private ecdsa key pair.
> Enter passphrase (empty for no passphrase):
> Enter same passphrase again:
> Your identification has been saved in id_test.
> Your public key has been saved in id_test.pub.
> The key fingerprint is:
> SHA256:fz7vI/LtbUAuU5XaIE+k2kpcoFiPTiJNEo3E/iN7OL8 root@openssl-1-0-2-o-r6
> The key's randomart image is:
> +---[ECDSA 521]---+
> |  o++. . .  ..  .|
> |   o+.o + ...o ..|
> |  .. + + . o+ +. |
> |   .. + . +  oo. |
> |    .  .S+ . +   |
> |   . o  ... o o  |
> |    + .  .. .o . |
> |   + .    .oo....|
> |    +E.    oo*=o.|
> +----[SHA256]-----+
> 
> 
> openssl-1-0-2-p ~ # openssl version
> OpenSSL 1.0.2p  14 Aug 2018
> openssl-1-0-2-p ~ # rsync -avP root@openssl-1-0-2-o-r6:/tmp/ssh-test /tmp/
> receiving incremental file list
> ssh-test/
> ssh-test/id_test
>             444 100%  433.59kB/s    0:00:00 (xfr#1, to-chk=1/3)
> ssh-test/id_test.pub
>             264 100%  257.81kB/s    0:00:00 (xfr#2, to-chk=0/3)
> 
> sent 66 bytes  received 901 bytes  1,934.00 bytes/sec
> total size is 708  speedup is 0.73
> openssl-1-0-2-p ~ # cd /tmp/ssh-test
> openssl-1-0-2-p /tmp/ssh-test # ssh-keygen -p -f id_test
> Enter old passphrase:
> Enter new passphrase (empty for no passphrase):
> Enter same passphrase again:
> Your identification has been saved with the new passphrase.
> openssl-1-0-2-p /tmp/ssh-test # cat id_test
> -----BEGIN EC PRIVATE KEY-----
> Proc-Type: 4,ENCRYPTED
> DEK-Info: AES-128-CBC,C7D921E20752EA2DFEE098CDCB7CBD39
> 
> lJfv9zMofyYhBBMU0c1Pn1jHLdC3+LNOppZ2cwYREpRP64gN7etWmy8dYlvtyzRx
> z6UJ0scbv6dN+3Xbds85rT+DKKrpyQtFSzEtIUBNBTLrSD+v1mufmhns8d5TxrtA
> N1oI5u+HxrxNjdt4Uw6ZrqfXP6jX7Sc7kVlx/tgG4AQRQ1Lp9w3bX689YdBVjPJR
> exhLLxhbI4K1Y/svTEPIVf8qXL9aE3nkkjgetfNyachhYw7zJlH0ETtgJlDwkv/y
> 7LuYcw38FKf9bIqt9QNIhv82Og5p1ERB1vbGMfAPtsw=
> -----END EC PRIVATE KEY-----
>
Comment 8 Alexander Sergeyev 2018-08-26 14:23:16 UTC
(In reply to Thomas Deutschmann from comment #7)
> I cannot reproduce:

It's not about a key being generated on an old version of openssh+openssl while being load on a newer version -- all repro could be done without downgrading to openssl-1.0.2o. The point was that 1.0.2o works but 1.0.2p does not.  

> The bug is somewhat subtle: it's always reproducible on the offending key, but not on every encrypted key.

I should have been more accurate here. The problem doesn't consistently manifest itself even when the key type is fixed. I had to spin on key generation & loading to finally catch some "unlucky" keys:

> $ openssl version
> OpenSSL 1.0.2p  14 Aug 2018
> $ cat ./find_bad_key.sh
> #!/bin/bash
> set -o errexit
>
> readonly tmp_path=$(mktemp --directory)
> function cleanup { [[ ! -z "$tmp_path" ]] && rm -r "$tmp_path"; }
> trap cleanup EXIT
>
> cd "${tmp_path}"
> for ((i=1;i<10000;++i)); do
>     ssh-keygen -t ecdsa -b 521 -N "12345678" -f testkey 1>/dev/null
>     if ssh-keygen -y -f testkey 0<&- 2>&1 | grep -q invalid; then
>         echo "[attempt #$i] found a broken key (passphrase = 12345678):"
>         echo; cat testkey; echo
>         exit 0
>     fi
>     rm ./testkey*
> done
> echo "No broken keys"
>
> $ ./find_bad_key.sh
> [attempt #82] found a broken key (passphrase = 12345678):
>
> -----BEGIN EC PRIVATE KEY-----
> Proc-Type: 4,ENCRYPTED
> DEK-Info: AES-128-CBC,24814239DFAA8BBAC60FAED983AD45DF
>
> I9KKWekBxm3q2lcVve0T/6f57Dm0WYPVQIiJg/qnKeICk8iGHt+gnWUBTD2Xu4ea
> 6vutwkOYjRSfTmL+gT2IeIefIkw9Ibs9YD/Q5pH8bT2hdt2NdqI5WPkQMjyBHCtr
> 4oCZq/m3CY3ckPoot98cTincBws4w+/hYCFHupKV3Dp1LPieZd80UfkiUVRxsXyB
> 94u2XdNw49j3PXKyNfjrtfvWXINWHKYzeBo0Qvhfo9Pj4AfLqNlz4cf3zRlqHx4S
> 5RDkIyJ2J4xXKx+btl0D9QqtgqCQb3MiL+nSdaPAtvs=
> -----END EC PRIVATE KEY-----
Comment 9 Alexander Sergeyev 2018-08-26 14:37:07 UTC
Created attachment 545120 [details]
find_bad_key.sh
Comment 10 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 15:03:36 UTC
For yet unknown reason, your script is still prompting for a passphrase so I cannot really run it.
Comment 11 Alexander Sergeyev 2018-08-26 16:38:29 UTC
(In reply to Thomas Deutschmann from comment #10)
> For yet unknown reason, your script is still prompting for a passphrase so I
> cannot really run it.

It appears to be some tricky behaviour of openssh: it might go for opening /dev/tty (ie controlling terminal) and not touching stdin. But in X environment it delegates the task to /usr/lib64/misc/ssh-askpass, which I didn't configure:

> # env DISPLAY= ssh-keygen -y -f testkey 0<&-
> ssh_askpass: exec(/usr/lib64/misc/ssh-askpass): No such file or directory
> Load key "testkey": incorrect passphrase supplied to decrypt private key

In case askpass is configured, there is a way to work around that:

> # env DISPLAY= SSH_ASKPASS=/bin/false ssh-keygen -y -f testkey 0<&-                                                                                                                                                                 
> Load key "testkey": incorrect passphrase supplied to decrypt private key

To avoid these hacks [1] it's easier just to use a command which wouldn't try to ask the passphrase -- passphrase changing is a good candidate. I will update the script in a minute.

[1] might be it'll get better someday: https://bugzilla.mindrot.org/show_bug.cgi?id=69
Comment 12 Alexander Sergeyev 2018-08-26 16:43:14 UTC
Created attachment 545134 [details]
find_bad_key.sh (v2)
Comment 13 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 17:29:40 UTC
Thanks, v2 works. However, I am getting invalid keys also with dev-libs/openssl-1.0.2o-r6. So this doesn't look like an dev-libs/openssl-1.0.2p problem for me.
Comment 14 Alexander Sergeyev 2018-08-26 17:40:53 UTC
(In reply to Thomas Deutschmann from comment #13)
> Thanks, v2 works. However, I am getting invalid keys also with
> dev-libs/openssl-1.0.2o-r6. So this doesn't look like an
> dev-libs/openssl-1.0.2p problem for me.

Oh, it's a minefield. Some new observations:

1) error message could be misleading in passphrase change mode -- supplying a wrong passphrase might lead to the "invalid format" message [1]:

> $ openssl version
> OpenSSL 1.0.2o  27 Mar 2018
> $ ssh-keygen -p -P x -N '' -f testkey_rsa 0<&- 2>&1
> Failed to load key testkey_rsa: invalid format
> $ ssh-keygen -p -P wrongpassphrase -N '' -f testkey_rsa 0<&- 2>&1
> Failed to load key testkey_rsa: incorrect passphrase supplied to decrypt private key
> $ ssh-keygen -p -P 12345678 -N '' -f testkey_rsa 0<&- 2>&1
> Your identification has been saved with the new passphrase.

2) openssh + openssl-1.0.2p might blow up with RSA keys as well (change appropriate options for ssh-keygen):

> [attempt #220] found a broken key (passphrase = 12345678):
>·
> # $ env DISPLAY= SSH_ASKPASS=/bin/false ssh-keygen -y -f testkey 0<&- 2>&1
> # Load key "testkey": invalid format
>·
> -----BEGIN RSA PRIVATE KEY-----
> Proc-Type: 4,ENCRYPTED
> DEK-Info: AES-128-CBC,4555D8E950EE897B40CC4073C0F7EFAE
>·
> RnK1jTUJJJZTR+OeyNRqg+ptcdjmYAZ9PPelnQpYMt7v918GkH497AdDuxCvaMoV
> lj7xK2L04Ry6+ToWpWnrvljPAz6y8WinD2VGkYpsZ+HNazr3ey7c31m/Yv4tsAgJ
> ToydA3Y5lA2EsEdpIPOr/nb5QILdPHmEAhuHLFkoVuwwpLH9YBFgk2Wd5r4JI91q
> FaQjXpaYSOm4X8i6h48TN/XQJtGXcRRqiLhQCltld4ednh2N1JGhTUuGCAJIpg7L
> eecaNY4Jrd1lrUfGsdSPvSwmup60NT0wwVLTS4apJhUtlgDCah0bqdrJGOftWWWf
> JQxH5b/hLT8TJHqJFWkCA5HcLf++WwTILPXCIE5hwC+v6RezmRrKO7uTahFby3Xp
> yJNFbgXv0+05rPfiBcgoDxzSk2MIf4mPJVSMlYS61HXBi1J+NHOEH/by5gHeAvi3
> 3UjRQhEGMjBycu4EK+J61Da5YeZUri5TjyxKGItar+zk8dOAPufnaXHqIS7ih+g8
> bELT6nwFekKE30dXJgWi3CpwHVgIrpw+LIkKS3ZweNBV+WSydwj7ZeSvbSuaunMg
> OjeJOOtQURyJ/gkhWJGkF23QkDUrDBh0S9FZSqERwZxUEtUQKuWQ2xBBRsLBPmLR
> y1pwPvecFyk0VEK/6se5mIL6B2ZexSDh7gpEwZ7dB90qbntEZqVwEEoJhUkKBLLW
> Ze6/TkTtAl4l2EsfBD0oNXuDBdzQi7dmXgguHKaX03C1GZhn5JgJB/3Latt7ReHq
> KuWIqFG+cNnrnXUvdWFZX+itg8Xz0ytWbgF6UhKB7K3fkRFJp8GxK6x1jUmuThpq
> oZgLYdkMd2pPMXX6gSRbjsTpmuuWB4ZOUFJeKCKBl/o9q7KfVvptYvikcIEaRqlh
> TKcw3/tdk3oYvOFCvH+x+uHIIDevY1264xLwP9UmT9rExGJkE/v7cVRuhURIW8uN
> Z3/AVd8zxCJK4u6UqZCRaKWbtoytNea8ikffdqBBvLxQaAcP1kyTYOxG35tD0hPu
> dGNuuLKkl6ulLgaRlq3IbB22B42Wr9QxGxJ9jM33OhbRovC/hHuAwcWvwFPs7NP+
> fIO4cQNSubiltPI5k60lDzP0pd992InfzJCbjMD240Mi73yr7tvWGI4ErJKlEcn4
> WrP2ACbgYrHP2mxuQFNgHW0bGLXgTJZtHRp0N8RuUCW/K2HI/DY8zMWQMiD91h9K
> qYoK4/bSCSyGRqAlt3MoFrXeCkjag94VTtDRzgiJCP1u0XfIYvA86T+mr8q7dYC6
> 6yhdfT1WlOd9f4khsh8a7JQXEMOblZ0BhUCaCTB5T2F/MQQe0ZH8FfUATijTAT6l
> Dq/ZhCbJpQYC4UvZCLYiTi97sAYUPz8aqk+04quo4Kqy0abhu2nGoimuywJB6OtS
> ZA+ij9HwUYhRLRVh55IghWqVSgOaV0JOXFq6cevuepWqI1jiIDpyR+50NDIX9ASf
> W0kcW8XtU8kaVJhiPTwl+Y/3kiJATWmvuBzUYW3otnkwJ/YqFPCainMOkSJmeZ9a
> GWdB7+AxIcoQsml45uo/1naCmm/OO+3QPce83h6by5EZ0cGcpry+Hw+nfeN7lgYY
> -----END RSA PRIVATE KEY-----

[1] this means the script must be changed back to the previously described hacks (to avoid false positives).
Comment 15 Alexander Sergeyev 2018-08-26 17:47:32 UTC
Created attachment 545156 [details]
find_bad_key.sh (v3)
Comment 16 Alexander Sergeyev 2018-08-26 17:52:59 UTC
> So this doesn't look like an dev-libs/openssl-1.0.2p problem for me.

Could you try the third version? It reverted back to `ssh-keygen -y` because the output of `ssh-keygen -p` is unreliable.
Comment 17 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 18:16:34 UTC
This is an OpenSSH 7.7 bug. I was able to reproduce this on Debian Stretch with vanilla OpenSSH built from source.
Comment 18 Thomas Deutschmann (RETIRED) gentoo-dev 2018-08-26 19:44:00 UTC
I filed a bug upstream: The error disappears in 7.8 once the new key format is used (https://github.com/openssh/openssh-portable/commit/ed7bd5d93fe14c7bd90febd29b858ea985d14d45). However, I guess that this is just hiding the problem but I am unable at the moment to modify the script to produce keys in old format (just adding "-pm PEM" doesn't work).
Comment 19 Alexander Sergeyev 2018-08-26 21:20:23 UTC
(In reply to Thomas Deutschmann from comment #18)

Thank you! I will try to play with v7.8 some time tomorrow.
Comment 20 Alexander Sergeyev 2018-08-29 19:05:37 UTC
(In reply to Thomas Deutschmann from comment #18)
> However, I guess that this is just hiding the problem but I am unable at the moment to modify the script to produce keys in old format (just adding "-pm PEM" doesn't work).

You're right, the problem is still there once PEM format is used:
https://github.com/sergeev917/openssh-openssl-bug-repro/tree/openssh-7.8p1