Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 709748

Summary: net-misc/openssh-8.2_p1: sshd[4020250]: recv_rexec_state: buffer error: incomplete message
Product: Gentoo Linux Reporter: Kobboi <gentoo>
Component: Current packagesAssignee: Gentoo's Team for Core System packages <base-system>
Status: RESOLVED OBSOLETE    
Severity: normal CC: gentoo_eshoes, hjckr, hydrapolic, jaak, matoro_bugzilla_gentoo, sam, slawomir.nizio, upendra.gandhi, wgh, zmedico
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: https://www.gentoo.org/support/news-items/2020-02-19-openssh-8_2-service-breakage.html
See Also: https://bugs.gentoo.org/show_bug.cgi?id=751484
Whiteboard:
Package list:
Runtime testing required: ---

Description Kobboi 2020-02-16 00:31:05 UTC
Client side:

$ ssh -vvvv work 
OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /home/christophe/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "work" port 22
debug2: ssh_connect_direct
debug1: Connecting to work [10.8.0.6] port 22.
debug1: Connection established.
debug1: identity file /home/christophe/.ssh/id_rsa type -1
debug1: identity file /home/christophe/.ssh/id_rsa-cert type -1
debug1: identity file /home/christophe/.ssh/id_dsa type -1
debug1: identity file /home/christophe/.ssh/id_dsa-cert type -1
debug1: identity file /home/christophe/.ssh/id_ecdsa type -1
debug1: identity file /home/christophe/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/christophe/.ssh/id_ed25519 type -1
debug1: identity file /home/christophe/.ssh/id_ed25519-cert type -1
debug1: identity file /home/christophe/.ssh/id_xmss type -1
debug1: identity file /home/christophe/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
kex_exchange_identification: read: Connection reset by peer

Server side:
sshd[4020250]: recv_rexec_state: buffer error: incomplete message

Note that this is just after an update to 8.2_p1. I don't know how nicely OpenSSH updates
go with running sshd processes.

[I] net-misc/openssh
     Available versions:  7.5_p1-r4^t 7.7_p1-r9^t 7.9_p1-r4^t 8.0_p1-r4^t (~)8.1_p1-r2^t{tbz2} (~)8.2_p1^t{tbz2} {X X509 audit bindist debug (+)hpn kerberos ldap ldns libedit libressl livecd pam +pie sctp selinux skey ssh1 +ssl static test xmss ABI_MIPS="n32" KERNEL="linux"}
     Installed versions:  8.2_p1^t{tbz2}(00:51:06 16/02/20)(X pam pie ssl -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -selinux -static -test -xmss ABI_MIPS="-n32" KERNEL="linux")
     Homepage:            https://www.openssh.com/
     Description:         Port of OpenBSD's free SSH release


Reproducible: Always
Comment 1 Kobboi 2020-02-16 00:45:42 UTC
# emerge --info
Portage 2.3.89 (python 3.6.10-final-0, kobboi/samsonov, gcc-9.2.0, glibc-2.30-r3, 5.4.15 x86_64)
=================================================================
System uname: Linux-5.4.15-x86_64-Intel-R-_Core-TM-_i7-6700_CPU_@_3.40GHz-with-gentoo-2.7
KiB Mem:    32780868 total,  12445456 free
KiB Swap:          0 total,         0 free
Head commit of repository gentoo: 1336c66870a6ce204014218867d83f0975f25d79

sh bash 5.0_p16
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
distcc 3.3.3 x86_64-pc-linux-gnu [disabled]
ccache version 3.7.7 [disabled]
app-shells/bash:          5.0_p16::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.17-r1::gentoo, 3.6.10::gentoo, 3.7.6::gentoo, 3.8.1::gentoo
dev-util/ccache:          3.7.7::gentoo
dev-util/cmake:           3.16.4::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.13.4-r2::gentoo, 1.15.1-r2::gentoo, 1.16.1-r2::gentoo
sys-devel/binutils:       2.33.1-r1::gentoo
sys-devel/gcc:            6.5.0-r1::gentoo, 9.2.0-r4::gentoo
sys-devel/gcc-config:     2.2.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.5::gentoo (virtual/os-headers)
sys-libs/glibc:           2.30-r3::gentoo
Repositories:

gentoo
    location: /var/cache/kobboi/gentoo
    sync-type: git
    sync-uri: https://github.com/kobboi/gentoo.git
    priority: -1000

Installed sets: @kobboi-all
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=x86-64 -pipe -fomit-frame-pointer -g -ggdb"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.6/conf /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=x86-64 -pipe -fomit-frame-pointer -g -ggdb"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
INSTALL_MASK="/usr/src/debug /usr/lib/debug"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PKGDIR="/usr/portage/packages"
PORTAGE_BINHOST="ssh://christophe@10.8.0.6/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi alsa amd64 avahi berkdb bluetooth bzip2 cairo cdda cdr cli colord crypt cups cxx dbus djvu dri dts dvd dvdr eds egl emboss encode evo exif ffmpeg flac fluidsynth fortran frei0r gdbm gif glade gles gnome gnome-keyring gnome-online-accounts gpm gstreamer gtk gtk3 gtkstyle iconv icu introspection ipv6 jpeg lcms ldap libass libnotify libsecret libtirpc mad midi mng modemmanager mono mp3 mp4 mpeg mtp multilib nautilus ncurses networkmanager nls nptl ogg opengl openmp opus pam pango pcre pdf png policykit ppds pulseaudio qt5 readline samba sdl seccomp smartcard smp spell split-usr ssl startup-notification svg systemd tcpd tiff truetype udev udisks unicode upower usb vaapi vdpau vorbis wayland wxwidgets x264 xattr xcb xml xv xvid zeroconf zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="hda-intel ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse synaptics libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="vesa radeon nvidia intel amdgpu dummy fbdev nouveau radeonsi v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, LC_ALL, LINGUAS, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 2 Kobboi 2020-02-16 00:50:06 UTC
I downgraded to 

[ebuild     UD] net-misc/openssh-8.1_p1-r2 [8.2_p1] USE="X pam pie ssl -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp (-selinux) -static -test -xmss" 

keeping sshd running. Everything is OK again now.
Comment 3 Patrick McLean gentoo-dev 2020-02-16 05:33:17 UTC
It looks like this only happens after updating a server to openssh-8.2_p1, but not restarting sshd (I managed to reproduce it)

I would suspect this is an upstream issue, I am not sure if they support updating openssh, but not restarting sshd.
Comment 4 Larry the Git Cow gentoo-dev 2020-02-16 06:47:27 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3543f68d6fa8cce8c957e02a2aaae79914f40441

commit 3543f68d6fa8cce8c957e02a2aaae79914f40441
Author:     Patrick McLean <chutzpah@gentoo.org>
AuthorDate: 2020-02-16 06:43:05 +0000
Commit:     Patrick McLean <chutzpah@gentoo.org>
CommitDate: 2020-02-16 06:46:40 +0000

    net-misc/openssh-8.2_p1-r1: ewarn to restart sshd (bug #709748)
    
    Bug: https://bugs.gentoo.org/709748
    Package-Manager: Portage-2.3.89, Repoman-2.3.20
    Signed-off-by: Patrick McLean <chutzpah@gentoo.org>

 net-misc/openssh/openssh-8.2_p1-r1.ebuild | 6 ++++++
 1 file changed, 6 insertions(+)
Comment 5 inactive 2020-02-16 17:27:25 UTC
I can confirm having the Comment 4 change in the ebuild /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild
and I'm upgrading from 8.1_p1-r2 to 8.2_p1-r1 but I cannot see the ewarns neither on console nor in /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-171251.log nor in /tmp/ebuild.logs/elog/summary.log

My ELOG settings seem to be default:
PORTAGE_ELOG_CLASSES="log warn error"
PORTAGE_ELOG_MAILFROM="portage@localhost"
PORTAGE_ELOG_MAILSUBJECT="[portage] ebuild log for ${PACKAGE} on ${HOST}"
PORTAGE_ELOG_MAILURI="root"
PORTAGE_ELOG_SYSTEM="save_summary:log,warn,error,qa echo"

except for:
PORT_LOGDIR="/tmp/ebuild.logs/"



# time emerge -v openssh

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] net-misc/openssh-8.2_p1-r1::gentoo [8.1_p1-r2::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key% (-selinux) -static -test -xmss" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

Would you like to merge these packages? [Yes/No] 
Sorry, response '' not understood. [Yes/No] y
>>> Verifying ebuild manifests
>>> Running pre-merge checks for net-misc/openssh-8.2_p1-r1
>>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo
>>> Installing (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo
>>> Jobs: 1 of 1 complete                           Load avg: 0.66, 0.28, 0.21

 * Messages for package net-misc/openssh-8.2_p1-r1:
 * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-171251.log

 * User patches applied.
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.

real	1m10.063s
user	0m27.970s
sys	0m16.215s

# grep -F After /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-171251.log /tmp/ebuild.logs/elog/summary.log
/var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:		ewarn "After upgrading to openssh-8.2p1 please restart sshd, otherwise you"



Some already mentioned info: because I didn't `rc-service sshd restart` after upgrading(to net-misc/openssh-8.2_p1-r1) from prev. version net-misc/openssh-8.1_p1-r2

on client's side, trying to connect to sshd:

kex_exchange_identification: read: Connection reset by peer

on sshd's side:

Feb 16 17:51:35 somehost sshd[29329]: debug3: fd 4 is not O_NONBLOCK
Feb 16 17:51:35 somehost sshd[29329]: debug1: Forked child 9073.
Feb 16 17:51:35 somehost sshd[29329]: debug3: send_rexec_state: entering fd = 7 config len 1310
Feb 16 17:51:35 somehost sshd[29329]: debug3: ssh_msg_send: type 0
Feb 16 17:51:35 somehost sshd[29329]: debug3: send_rexec_state: done
Feb 16 17:51:35 somehost sshd[9073]: debug3: oom_adjust_restore
Feb 16 17:51:35 somehost sshd[9073]: debug1: Set /proc/self/oom_score_adj to 0
Feb 16 17:51:35 somehost sshd[9073]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Feb 16 17:51:35 somehost sshd[9073]: fatal: recv_rexec_state: buffer error: incomplete message
Comment 6 inactive 2020-02-16 17:46:56 UTC
my test shows that 'if' isn't entered, in this case:

# time emerge -v --ask n =openssh-8.1_p1-r2 && time emerge -v --ask n openssh

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     UD ] net-misc/openssh-8.1_p1-r2::gentoo [8.2_p1-r1::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp (-selinux) -static -test -xmss (-security-key%)" 0 KiB

Total: 1 package (1 downgrade), Size of downloads: 0 KiB

>>> Verifying ebuild manifests
>>> Running pre-merge checks for net-misc/openssh-8.1_p1-r2
>>> Emerging (1 of 1) net-misc/openssh-8.1_p1-r2::gentoo
>>> Installing (1 of 1) net-misc/openssh-8.1_p1-r2::gentoo
>>> Jobs: 1 of 1 complete                           Load avg: 1.15, 0.32, 0.16

 * Messages for package net-misc/openssh-8.1_p1-r2:
 * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.1_p1-r2:20200216-174141.log

 * User patches applied.
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.

 * IMPORTANT: 2 config files in '/etc' need updating.
 * See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS
 * sections of the emerge man page to learn how to update config files.

real	0m40.739s
user	0m28.359s
sys	0m16.714s

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] net-misc/openssh-8.2_p1-r1::gentoo [8.1_p1-r2::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key% (-selinux) -static -test -xmss" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

>>> Verifying ebuild manifests
>>> Running pre-merge checks for net-misc/openssh-8.2_p1-r1
>>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo
>>> Installing (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo
>>> Jobs: 1 of 1 complete                           Load avg: 1.08, 0.39, 0.20

 * Messages for package net-misc/openssh-8.2_p1-r1:
 * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-174221.log

 * User patches applied.
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.

 * IMPORTANT: 2 config files in '/etc' need updating.
 * See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS
 * sections of the emerge man page to learn how to update config files.

real	0m38.517s
user	0m27.660s
sys	0m16.458s


$ grep -nF 'hdjkshdskjahdsljkadhsak' /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-174221.log /tmp/ebuild.logs/elog/summary.log
/var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:472:		echo "hdjkshdskjahdsljkadhsak"

$ grep -nF 'gurl' /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-174221.log /tmp/ebuild.logs/elog/summary.log
/var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:471:		elog "whatever gurl"

this 'if':

    if has_version "<${CATEGORY}/${PN}-8.2_p1"; then
        ewarn "After upgrading to openssh-8.2p1 please restart sshd, otherwise you"
        ewarn "will not be able to establish new sessions. Restarting sshd over a ssh"
        ewarn "connection is generally safe."
        elog "whatever gurl"
        echo "hdjkshdskjahdsljkadhsak"
    fi


Perhaps I'm missing something? I know I usually do! kinda curious what's going on here
Comment 7 inactive 2020-02-16 17:53:28 UTC
Found the doc for has_version here: https://devmanual.gentoo.org/eclass-reference/ebuild/index.html

has_version [-b] [-d] [-r] [--host-root] <category/package-version>
    Check to see if category/package-version is installed. The parameter accepts all values that are acceptable in the DEPEND variable. The function returns 0 if category/package-version is installed, 1 otherwise. The package is searched for in ROOT by default.

    In EAPI 5 and EAPI 6, the package is searched for in the build host if the --host-root option is given.

    In EAPI 7 and later, the confusing --host-root option has been replaced with -b, which corresponds to a dependency satisfied by BDEPEND in the build host. Similarly, the -d option corresponds to DEPEND in SYSROOT and the -r option corresponds to RDEPEND in ROOT. 

since that 'if' is inside pkg_postinst()  does that mean that the currently emerging openssh version is considered installed? then it would never be less than itself?
Comment 8 inactive 2020-02-16 18:01:37 UTC
ok this 'if' enters(and I've no idea why):

if has_version ">${CATEGORY}/${PN}-8.2_p1"; then

ie.
    if has_version "<${CATEGORY}/${PN}-8.2_p1"; then
        elog "lower"
    fi
    if has_version "=${CATEGORY}/${PN}-8.2_p1"; then
        elog "equal"
    fi
    if has_version ">${CATEGORY}/${PN}-8.2_p1"; then
        elog "bigger"                                                                                                            
    fi

[ebuild     U  ] net-misc/openssh-8.2_p1-r1::gentoo [8.1_p1-r2::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key% (-selinux) -static -test -xmss" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

>>> Verifying ebuild manifests
>>> Running pre-merge checks for net-misc/openssh-8.2_p1-r1
>>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo
>>> Installing (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo
>>> Jobs: 1 of 1 complete                           Load avg: 0.88, 0.29, 0.15

 * Messages for package net-misc/openssh-8.2_p1-r1:
 * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-175825.log

 * User patches applied.
 * bigger
>>> Auto-cleaning packages...

$ grep -nE '(lower|bigger|equal)' /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/elog/summary.log /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-175825.log
/var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:468:		elog "lower"
/var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:471:		elog "equal"
/var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:474:		elog "bigger"
/tmp/ebuild.logs/elog/summary.log:165:bigger
/tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-175825.log:1710: * bigger
Comment 9 inactive 2020-02-16 18:15:58 UTC
Ok I think I get it, it returns bigger because the currently installed version has the -r1 and it is bigger than the one that lacks -r1

$ portageq has_version / '=openssh-8.2_p1'; echo -n "$? " ; portageq has_version / '>openssh-8.2_p1'; echo -n "$?"
1 0

$ portageq has_version / '=openssh-8.2_p1-r1'; echo -n "$? " ; portageq has_version / '>openssh-8.2_p1-r1'; echo -n "$?"
0 1
Comment 10 Larry the Git Cow gentoo-dev 2020-02-16 18:30:53 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a225fe10e4c21edd8915543c2a4318b00d2144c6

commit a225fe10e4c21edd8915543c2a4318b00d2144c6
Author:     Patrick McLean <chutzpah@gentoo.org>
AuthorDate: 2020-02-16 18:29:52 +0000
Commit:     Patrick McLean <chutzpah@gentoo.org>
CommitDate: 2020-02-16 18:30:41 +0000

    net-misc/openssh-8.1_p1-r2: Disable X509 and security-key (bug #709808)
    
    This also makes the warning about restarting sshd actually show when it
    is intended to. This refactors all version warnings by using a flag
    variable set in pkg_preinst to decide whether to show the warning in
    pkg_postinst.
    
    Closes: https://bugs.gentoo.org/709808
    Bug: https://bugs.gentoo.org/709748
    Package-Manager: Portage-2.3.89, Repoman-2.3.20
    Signed-off-by: Patrick McLean <chutzpah@gentoo.org>

 net-misc/openssh/openssh-8.2_p1-r1.ebuild | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
Comment 11 Preston Crow 2020-02-19 16:32:38 UTC
>>> Verifying ebuild manifests
>>> Running pre-merge checks for net-misc/openssh-8.2_p1-r2
>>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r2::gentoo
>>> Installing (1 of 1) net-misc/openssh-8.2_p1-r2::gentoo
>>> Jobs: 1 of 1 complete                           Load avg: 2.32, 1.36, 0.90

 * Messages for package net-misc/openssh-8.2_p1-r2:

 * After upgrading to openssh-8.2p1 please restart sshd, otherwise you
 * will not be able to establish new sessions. Restarting sshd over a ssh
 * connection is generally safe.
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.


I think the above is what is intended to resolve this.  That works for me.  With a restart, ssh connections work as expected.  I suppose there's no way to automatically restart the service at the completion of the package installation?
Comment 12 Sven E. 2020-02-19 22:39:09 UTC
Restarting SSHd or any other daemon after any upgrade is best practice.

Issues like this have risen for SSHd recurrently in the last two decades. And not just for SSHd, even though for SSHd it can really bite you.
Comment 13 matoro archtester 2020-02-20 00:50:28 UTC
This just took down my VPS, which I'm currently rebuilding from a 2-year-old snapshot.  I don't know what the best way is to make this message as loud as possible, but please make sure it is.
Comment 14 inactive 2020-02-20 11:06:39 UTC
ArchLinux opted to auto-restart sshd(iff it was already started), but they're only dealing with systemd not (also)openrc.

https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/openssh&id=042a0ad1ac04ff420d61f3acfdf196f1a5d7d9ef

I'm not sure it makes sense for Gentoo to do this. I assume people are upgrading over ssh, which upon seeing the message, would manually restart sshd. So, it seems ok as it is now, imho.

I don't understand how @matoro's VPS got taken down because of this...
Comment 15 matoro archtester 2020-02-20 12:47:25 UTC
(In reply to gentoo_eshoes from comment #14)
> ArchLinux opted to auto-restart sshd(iff it was already started), but
> they're only dealing with systemd not (also)openrc.
> 
> https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/
> openssh&id=042a0ad1ac04ff420d61f3acfdf196f1a5d7d9ef
> 
> I'm not sure it makes sense for Gentoo to do this. I assume people are
> upgrading over ssh, which upon seeing the message, would manually restart
> sshd. So, it seems ok as it is now, imho.
> 
> I don't understand how @matoro's VPS got taken down because of this...

It's an ssh-only VPS, and it's not designed to be rebooted.  Meaning if sshd goes down, the web services stay running but it's permanently inaccessible.  I don't think these are that uncommon - maybe a combination of bad circumstances, but there was no warning at all at the time.  This definitely needs to be *at least* as loud and annoying as that sys-fs/eudev warning about the renamed interfaces - if we don't just restart it for people altogether (should be pretty easy to check common scenarios - openrc, systemd service, systemd socket).
Comment 16 Nikolay Kichukov 2020-03-23 14:30:43 UTC
Agreed,
another case scenario is where updates to a headless system are done from detached 'screen' session where after sshd got upgraded, the system remained out of administrative control.
Comment 17 James Broadhead 2020-12-28 15:52:20 UTC
I was just hit by this, on a machine which is difficult to manually reboot. 
This was a similar case to Nikolay's -- the upgrade was happening invisibly, and completed at a time when I had no active sessions. 
I believe that Arch's approach is the correct one -- if the ssh daemon always needs restarting, then I think that portage should do it. In cases where there are more options to consider, then einfo/ewarn are fine. 

Since openssh-8.4_p1-r2 was made stable on most architectures around 25 Dec - I expect that this problem will affect most of the community in the coming days. Now would be a good time to take action to avoid locking more people out of their systems.
Comment 18 Eric Henry 2020-12-28 20:01:16 UTC
Ditto the above. I was upgrading remotely on several machines, using 'nohup emerge...' so that the jobs could complete without my having to stay logged in. In light of portage's not restarting the server after an upgrade (which I guess I hadn't expected), this makes the 'nohup ...' approach intrinsically dangerous. This in turn makes bulk (multi-machine) upgrades extremely inconvenient. Problem was aggravated in this case when my local internet went out during the upgrade; job finished, but new session required when my connectivity returned has meant that I am now shut out of all those machines--and I am 1000 miles away.
Only hope of re-connecting is to have somebody power-cycle all the machines for me. I would lobby for portage to do as much as possible to maintain state and integrity following any upgrade; this might still be a job for post-intall on any given package.
Comment 19 kfm 2020-12-28 23:49:39 UTC
(In reply to James Broadhead from comment #17)
> Since openssh-8.4_p1-r2 was made stable on most architectures around 25 Dec
> - I expect that this problem will affect most of the community in the coming
> days. Now would be a good time to take action to avoid locking more people
> out of their systems.

I concur. Assuming the use of portage as a sub-process of an interactive shell that the user has control over for the duration of any given operation doesn't cut it. I, too, was affected by this, though I had the good fortune of having out-of-band management facilities at my disposal, so recovering was only a matter of tedium.

It is highly unusual for an immediate restart to be imperative upon upgrading openssh. However, this incident suggests that its developers are under no obligation to maintain inter-process compatibility between major releases. I don't much like the idea of ebuilds being able to restart services. Yet, if there is even the slightest chance that it could happen again in the future, I think that the possibility should be considered, given the obvious importance of sshd.

This is just a thought, but perhaps the ebuild could elect to do so while avoiding it in the case that a major version upgrade/downgrade has not occurred. The following code demonstrates the concept.

pkg_postinst() {
	local ver
	declare -a versions
	read -ra versions <<<"${REPLACING_VERSIONS}"
	for ver in "${versions[@]}"; do
		if [[ ${ver%_*} == "${PV%_*}" ]]; then
			# No major version change has occurred
			return
		fi
	done
	if [[ -d /run/systemd/system ]]; then
		systemctl try-restart sshd
	else
		rc-service -q --ifstarted sshd restart
	fi
}
Comment 20 kfm 2020-12-29 03:11:42 UTC
I incorporated the previously mentioned code to a local copy of the ebuild and found that it works as expected. An upgrade from 8.2 to 8.4 triggered a restart, whereas re-emerging 8.4 did not. I'll be using that to upgrade the remainder of my fleet.
Comment 21 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2020-12-29 03:20:38 UTC
I was similarly shocked but this was actually part of a news item (it was just so long ago I forgot, as the stabilisation only recently happened): https://www.gentoo.org/support/news-items/2020-02-19-openssh-8_2-service-breakage.html.

I would like to see some mechanism for restarting services built-in or easily accessible, and also think that such a snippet as Kerin gave is a reasonable compromise (major versions only).
Comment 22 kfm 2020-12-29 03:22:34 UTC
(In reply to Sam James from comment #21)
> I was similarly shocked but this was actually part of a news item (it was
> just so long ago I forgot, as the stabilisation only recently happened):
> https://www.gentoo.org/support/news-items/2020-02-19-openssh-8_2-service-
> breakage.html.

I now recall reading this but had also completely forgotten.
Comment 23 Zac Medico gentoo-dev 2020-12-29 05:03:49 UTC
(In reply to Sam James from comment #21)
> I would like to see some mechanism for restarting services built-in or
> easily accessible, and also think that such a snippet as Kerin gave is a
> reasonable compromise (major versions only).

Though service restarts will be helpful for sshd issues like this one, when updating live systems, users also need to be well prepared for cases where a simple service restart will not correct the problem.

A good way to prepare for this sort of scenario is to create a failsafe sshd instance that listens on a different port from the regular instance. The dependencies of the failsafe sshd instance should be as isolated as possible from the regular system. For example, it can run in a chroot (which you can escape after login), or be built with USE="static -pam" in order to make it immune to breakage in shared libraries or pam configuration.
Comment 24 upendra 2021-02-03 19:22:47 UTC
Just encountered this while running world update.

net-misc/openssh
     Available versions:  8.1_p1-r5^t ~8.2_p1-r8^t ~8.3_p1-r6^t 8.4_p1-r3^t {X X509 audit bindist debug hpn kerberos ldns libedit libressl livecd pam +pie +scp sctp security-key selinux +ssl static test xmss ABI_MIPS="n32" KERNEL="linux"}
     Installed versions:  8.4_p1-r3^t(11:30:57 AM 02/03/2021)(X pam pie scp ssl -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key -selinux -static -test -xmss ABI_MIPS="-n32" KERNEL="linux")
     Homepage:            https://www.openssh.com/
     Description:         Port of OpenBSD's free SSH release


Could not open another SSH session, so sent upgrade to background and ran systemctl try-restart sshd to resolve the issue.


recv_rexec_state: buffer error: incomplete message was the original error while connecting via another ssh session.

Thank you for the bug and the comments!!
Comment 25 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-03-03 22:03:02 UTC
I think the lesson here is largely to coordinate news item with stabilisation more so people don't forget, and also if it happens again, consider our options carefully in the ebuild.

But no real point in leaving this one open.