I've just been hit by not being unable to connect to a server via ssh due to: $ ssh ... Load key "/redacted/path": invalid format johndoe@1.2.3.4: Permission denied (publickey). The private key is encrypted with a passphrase, other openssh utilities fail similarly: $ ssh-keygen -p -f ./key Failed to load key ./key: invalid format The bug is somewhat subtle: it's always reproducible on the offending key, but not on every encrypted key. Downgrading to dev-libs/openssl-1.0.2o-r6 fixes the problem. Versions and use-flags: net-misc/openssh-7.7_p1-r6:0 hpn pam pie ssl -X -X509 -audit -bindist -debug -kerberos -ldap -ldns -libedit -libressl -livecd -sctp -selinux -skey -static -test ABI_MIPS="-n32" KERNEL="linux" dev-libs/openssl-1.0.2p:0 asm zlib -bindist -gmp -kerberos -rfc3779 -sctp -sslv2 -sslv3 -static-libs -test -tls-heartbeat -vanilla ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="64 -32 -x32" CPU_FLAGS_X86="sse2"
> it's always reproducible on the offending key, but not on every encrypted key We need details about the non-working key. Can you provide details how we can generate such a key?
> Can you provide details how we can generate such a key? It's a ecdsa-sha2-nistp521 key with additional passphrase encryption (AES-128-CBC). I cannot share the original material without implied access revokation, etc, which I would prefer to avoid. Currently, I'm trying to re-create behaviour with no sensitive data in play.
Created attachment 545034 [details] an example of failing key
I've added an example key that fails to load. The passphrase is in between underscores [of the filename]. But it's not really needed to observe the failure: $ ssh-keygen -y -f ./testkey_gceSwXMT_iter00000652 </dev/null Load key "./testkey_gceSwXMT_iter00000652": invalid format
I am only asking for the command to re-create such a key.
(In reply to Thomas Deutschmann from comment #5) > I am only asking for the command to re-create such a key. Here it is: $ ssh-keygen -t ecdsa -b 521
I cannot reproduce: I downgraded a test box to openssl-1.0.2o-r6. I have rebuilt openssh-7.7_p1-r9 on that box, too (just to be sure): > openssl-1-0-2-o-r6 ~ # openssl version > OpenSSL 1.0.2o 27 Mar 2018 > openssl-1-0-2-o-r6 ~ # mkdir /tmp/ssh-test > openssl-1-0-2-o-r6 ~ # cd /tmp/ssh-test/ > openssl-1-0-2-o-r6 /tmp/ssh-test # ssh-keygen -t ecdsa -b 521 -f id_test > Generating public/private ecdsa key pair. > Enter passphrase (empty for no passphrase): > Enter same passphrase again: > Your identification has been saved in id_test. > Your public key has been saved in id_test.pub. > The key fingerprint is: > SHA256:fz7vI/LtbUAuU5XaIE+k2kpcoFiPTiJNEo3E/iN7OL8 root@openssl-1-0-2-o-r6 > The key's randomart image is: > +---[ECDSA 521]---+ > | o++. . . .. .| > | o+.o + ...o ..| > | .. + + . o+ +. | > | .. + . + oo. | > | . .S+ . + | > | . o ... o o | > | + . .. .o . | > | + . .oo....| > | +E. oo*=o.| > +----[SHA256]-----+ > > > openssl-1-0-2-p ~ # openssl version > OpenSSL 1.0.2p 14 Aug 2018 > openssl-1-0-2-p ~ # rsync -avP root@openssl-1-0-2-o-r6:/tmp/ssh-test /tmp/ > receiving incremental file list > ssh-test/ > ssh-test/id_test > 444 100% 433.59kB/s 0:00:00 (xfr#1, to-chk=1/3) > ssh-test/id_test.pub > 264 100% 257.81kB/s 0:00:00 (xfr#2, to-chk=0/3) > > sent 66 bytes received 901 bytes 1,934.00 bytes/sec > total size is 708 speedup is 0.73 > openssl-1-0-2-p ~ # cd /tmp/ssh-test > openssl-1-0-2-p /tmp/ssh-test # ssh-keygen -p -f id_test > Enter old passphrase: > Enter new passphrase (empty for no passphrase): > Enter same passphrase again: > Your identification has been saved with the new passphrase. > openssl-1-0-2-p /tmp/ssh-test # cat id_test > -----BEGIN EC PRIVATE KEY----- > Proc-Type: 4,ENCRYPTED > DEK-Info: AES-128-CBC,C7D921E20752EA2DFEE098CDCB7CBD39 > > lJfv9zMofyYhBBMU0c1Pn1jHLdC3+LNOppZ2cwYREpRP64gN7etWmy8dYlvtyzRx > z6UJ0scbv6dN+3Xbds85rT+DKKrpyQtFSzEtIUBNBTLrSD+v1mufmhns8d5TxrtA > N1oI5u+HxrxNjdt4Uw6ZrqfXP6jX7Sc7kVlx/tgG4AQRQ1Lp9w3bX689YdBVjPJR > exhLLxhbI4K1Y/svTEPIVf8qXL9aE3nkkjgetfNyachhYw7zJlH0ETtgJlDwkv/y > 7LuYcw38FKf9bIqt9QNIhv82Og5p1ERB1vbGMfAPtsw= > -----END EC PRIVATE KEY----- >
(In reply to Thomas Deutschmann from comment #7) > I cannot reproduce: It's not about a key being generated on an old version of openssh+openssl while being load on a newer version -- all repro could be done without downgrading to openssl-1.0.2o. The point was that 1.0.2o works but 1.0.2p does not. > The bug is somewhat subtle: it's always reproducible on the offending key, but not on every encrypted key. I should have been more accurate here. The problem doesn't consistently manifest itself even when the key type is fixed. I had to spin on key generation & loading to finally catch some "unlucky" keys: > $ openssl version > OpenSSL 1.0.2p 14 Aug 2018 > $ cat ./find_bad_key.sh > #!/bin/bash > set -o errexit > > readonly tmp_path=$(mktemp --directory) > function cleanup { [[ ! -z "$tmp_path" ]] && rm -r "$tmp_path"; } > trap cleanup EXIT > > cd "${tmp_path}" > for ((i=1;i<10000;++i)); do > ssh-keygen -t ecdsa -b 521 -N "12345678" -f testkey 1>/dev/null > if ssh-keygen -y -f testkey 0<&- 2>&1 | grep -q invalid; then > echo "[attempt #$i] found a broken key (passphrase = 12345678):" > echo; cat testkey; echo > exit 0 > fi > rm ./testkey* > done > echo "No broken keys" > > $ ./find_bad_key.sh > [attempt #82] found a broken key (passphrase = 12345678): > > -----BEGIN EC PRIVATE KEY----- > Proc-Type: 4,ENCRYPTED > DEK-Info: AES-128-CBC,24814239DFAA8BBAC60FAED983AD45DF > > I9KKWekBxm3q2lcVve0T/6f57Dm0WYPVQIiJg/qnKeICk8iGHt+gnWUBTD2Xu4ea > 6vutwkOYjRSfTmL+gT2IeIefIkw9Ibs9YD/Q5pH8bT2hdt2NdqI5WPkQMjyBHCtr > 4oCZq/m3CY3ckPoot98cTincBws4w+/hYCFHupKV3Dp1LPieZd80UfkiUVRxsXyB > 94u2XdNw49j3PXKyNfjrtfvWXINWHKYzeBo0Qvhfo9Pj4AfLqNlz4cf3zRlqHx4S > 5RDkIyJ2J4xXKx+btl0D9QqtgqCQb3MiL+nSdaPAtvs= > -----END EC PRIVATE KEY-----
Created attachment 545120 [details] find_bad_key.sh
For yet unknown reason, your script is still prompting for a passphrase so I cannot really run it.
(In reply to Thomas Deutschmann from comment #10) > For yet unknown reason, your script is still prompting for a passphrase so I > cannot really run it. It appears to be some tricky behaviour of openssh: it might go for opening /dev/tty (ie controlling terminal) and not touching stdin. But in X environment it delegates the task to /usr/lib64/misc/ssh-askpass, which I didn't configure: > # env DISPLAY= ssh-keygen -y -f testkey 0<&- > ssh_askpass: exec(/usr/lib64/misc/ssh-askpass): No such file or directory > Load key "testkey": incorrect passphrase supplied to decrypt private key In case askpass is configured, there is a way to work around that: > # env DISPLAY= SSH_ASKPASS=/bin/false ssh-keygen -y -f testkey 0<&- > Load key "testkey": incorrect passphrase supplied to decrypt private key To avoid these hacks [1] it's easier just to use a command which wouldn't try to ask the passphrase -- passphrase changing is a good candidate. I will update the script in a minute. [1] might be it'll get better someday: https://bugzilla.mindrot.org/show_bug.cgi?id=69
Created attachment 545134 [details] find_bad_key.sh (v2)
Thanks, v2 works. However, I am getting invalid keys also with dev-libs/openssl-1.0.2o-r6. So this doesn't look like an dev-libs/openssl-1.0.2p problem for me.
(In reply to Thomas Deutschmann from comment #13) > Thanks, v2 works. However, I am getting invalid keys also with > dev-libs/openssl-1.0.2o-r6. So this doesn't look like an > dev-libs/openssl-1.0.2p problem for me. Oh, it's a minefield. Some new observations: 1) error message could be misleading in passphrase change mode -- supplying a wrong passphrase might lead to the "invalid format" message [1]: > $ openssl version > OpenSSL 1.0.2o 27 Mar 2018 > $ ssh-keygen -p -P x -N '' -f testkey_rsa 0<&- 2>&1 > Failed to load key testkey_rsa: invalid format > $ ssh-keygen -p -P wrongpassphrase -N '' -f testkey_rsa 0<&- 2>&1 > Failed to load key testkey_rsa: incorrect passphrase supplied to decrypt private key > $ ssh-keygen -p -P 12345678 -N '' -f testkey_rsa 0<&- 2>&1 > Your identification has been saved with the new passphrase. 2) openssh + openssl-1.0.2p might blow up with RSA keys as well (change appropriate options for ssh-keygen): > [attempt #220] found a broken key (passphrase = 12345678): >· > # $ env DISPLAY= SSH_ASKPASS=/bin/false ssh-keygen -y -f testkey 0<&- 2>&1 > # Load key "testkey": invalid format >· > -----BEGIN RSA PRIVATE KEY----- > Proc-Type: 4,ENCRYPTED > DEK-Info: AES-128-CBC,4555D8E950EE897B40CC4073C0F7EFAE >· > RnK1jTUJJJZTR+OeyNRqg+ptcdjmYAZ9PPelnQpYMt7v918GkH497AdDuxCvaMoV > lj7xK2L04Ry6+ToWpWnrvljPAz6y8WinD2VGkYpsZ+HNazr3ey7c31m/Yv4tsAgJ > ToydA3Y5lA2EsEdpIPOr/nb5QILdPHmEAhuHLFkoVuwwpLH9YBFgk2Wd5r4JI91q > FaQjXpaYSOm4X8i6h48TN/XQJtGXcRRqiLhQCltld4ednh2N1JGhTUuGCAJIpg7L > eecaNY4Jrd1lrUfGsdSPvSwmup60NT0wwVLTS4apJhUtlgDCah0bqdrJGOftWWWf > JQxH5b/hLT8TJHqJFWkCA5HcLf++WwTILPXCIE5hwC+v6RezmRrKO7uTahFby3Xp > yJNFbgXv0+05rPfiBcgoDxzSk2MIf4mPJVSMlYS61HXBi1J+NHOEH/by5gHeAvi3 > 3UjRQhEGMjBycu4EK+J61Da5YeZUri5TjyxKGItar+zk8dOAPufnaXHqIS7ih+g8 > bELT6nwFekKE30dXJgWi3CpwHVgIrpw+LIkKS3ZweNBV+WSydwj7ZeSvbSuaunMg > OjeJOOtQURyJ/gkhWJGkF23QkDUrDBh0S9FZSqERwZxUEtUQKuWQ2xBBRsLBPmLR > y1pwPvecFyk0VEK/6se5mIL6B2ZexSDh7gpEwZ7dB90qbntEZqVwEEoJhUkKBLLW > Ze6/TkTtAl4l2EsfBD0oNXuDBdzQi7dmXgguHKaX03C1GZhn5JgJB/3Latt7ReHq > KuWIqFG+cNnrnXUvdWFZX+itg8Xz0ytWbgF6UhKB7K3fkRFJp8GxK6x1jUmuThpq > oZgLYdkMd2pPMXX6gSRbjsTpmuuWB4ZOUFJeKCKBl/o9q7KfVvptYvikcIEaRqlh > TKcw3/tdk3oYvOFCvH+x+uHIIDevY1264xLwP9UmT9rExGJkE/v7cVRuhURIW8uN > Z3/AVd8zxCJK4u6UqZCRaKWbtoytNea8ikffdqBBvLxQaAcP1kyTYOxG35tD0hPu > dGNuuLKkl6ulLgaRlq3IbB22B42Wr9QxGxJ9jM33OhbRovC/hHuAwcWvwFPs7NP+ > fIO4cQNSubiltPI5k60lDzP0pd992InfzJCbjMD240Mi73yr7tvWGI4ErJKlEcn4 > WrP2ACbgYrHP2mxuQFNgHW0bGLXgTJZtHRp0N8RuUCW/K2HI/DY8zMWQMiD91h9K > qYoK4/bSCSyGRqAlt3MoFrXeCkjag94VTtDRzgiJCP1u0XfIYvA86T+mr8q7dYC6 > 6yhdfT1WlOd9f4khsh8a7JQXEMOblZ0BhUCaCTB5T2F/MQQe0ZH8FfUATijTAT6l > Dq/ZhCbJpQYC4UvZCLYiTi97sAYUPz8aqk+04quo4Kqy0abhu2nGoimuywJB6OtS > ZA+ij9HwUYhRLRVh55IghWqVSgOaV0JOXFq6cevuepWqI1jiIDpyR+50NDIX9ASf > W0kcW8XtU8kaVJhiPTwl+Y/3kiJATWmvuBzUYW3otnkwJ/YqFPCainMOkSJmeZ9a > GWdB7+AxIcoQsml45uo/1naCmm/OO+3QPce83h6by5EZ0cGcpry+Hw+nfeN7lgYY > -----END RSA PRIVATE KEY----- [1] this means the script must be changed back to the previously described hacks (to avoid false positives).
Created attachment 545156 [details] find_bad_key.sh (v3)
> So this doesn't look like an dev-libs/openssl-1.0.2p problem for me. Could you try the third version? It reverted back to `ssh-keygen -y` because the output of `ssh-keygen -p` is unreliable.
This is an OpenSSH 7.7 bug. I was able to reproduce this on Debian Stretch with vanilla OpenSSH built from source.
I filed a bug upstream: The error disappears in 7.8 once the new key format is used (https://github.com/openssh/openssh-portable/commit/ed7bd5d93fe14c7bd90febd29b858ea985d14d45). However, I guess that this is just hiding the problem but I am unable at the moment to modify the script to produce keys in old format (just adding "-pm PEM" doesn't work).
(In reply to Thomas Deutschmann from comment #18) Thank you! I will try to play with v7.8 some time tomorrow.
(In reply to Thomas Deutschmann from comment #18) > However, I guess that this is just hiding the problem but I am unable at the moment to modify the script to produce keys in old format (just adding "-pm PEM" doesn't work). You're right, the problem is still there once PEM format is used: https://github.com/sergeev917/openssh-openssl-bug-repro/tree/openssh-7.8p1