Client side: $ ssh -vvvv work OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019 debug1: Reading configuration data /home/christophe/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug2: resolving "work" port 22 debug2: ssh_connect_direct debug1: Connecting to work [10.8.0.6] port 22. debug1: Connection established. debug1: identity file /home/christophe/.ssh/id_rsa type -1 debug1: identity file /home/christophe/.ssh/id_rsa-cert type -1 debug1: identity file /home/christophe/.ssh/id_dsa type -1 debug1: identity file /home/christophe/.ssh/id_dsa-cert type -1 debug1: identity file /home/christophe/.ssh/id_ecdsa type -1 debug1: identity file /home/christophe/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/christophe/.ssh/id_ed25519 type -1 debug1: identity file /home/christophe/.ssh/id_ed25519-cert type -1 debug1: identity file /home/christophe/.ssh/id_xmss type -1 debug1: identity file /home/christophe/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.1 kex_exchange_identification: read: Connection reset by peer Server side: sshd[4020250]: recv_rexec_state: buffer error: incomplete message Note that this is just after an update to 8.2_p1. I don't know how nicely OpenSSH updates go with running sshd processes. [I] net-misc/openssh Available versions: 7.5_p1-r4^t 7.7_p1-r9^t 7.9_p1-r4^t 8.0_p1-r4^t (~)8.1_p1-r2^t{tbz2} (~)8.2_p1^t{tbz2} {X X509 audit bindist debug (+)hpn kerberos ldap ldns libedit libressl livecd pam +pie sctp selinux skey ssh1 +ssl static test xmss ABI_MIPS="n32" KERNEL="linux"} Installed versions: 8.2_p1^t{tbz2}(00:51:06 16/02/20)(X pam pie ssl -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -selinux -static -test -xmss ABI_MIPS="-n32" KERNEL="linux") Homepage: https://www.openssh.com/ Description: Port of OpenBSD's free SSH release Reproducible: Always
# emerge --info Portage 2.3.89 (python 3.6.10-final-0, kobboi/samsonov, gcc-9.2.0, glibc-2.30-r3, 5.4.15 x86_64) ================================================================= System uname: Linux-5.4.15-x86_64-Intel-R-_Core-TM-_i7-6700_CPU_@_3.40GHz-with-gentoo-2.7 KiB Mem: 32780868 total, 12445456 free KiB Swap: 0 total, 0 free Head commit of repository gentoo: 1336c66870a6ce204014218867d83f0975f25d79 sh bash 5.0_p16 ld GNU ld (Gentoo 2.33.1 p2) 2.33.1 distcc 3.3.3 x86_64-pc-linux-gnu [disabled] ccache version 3.7.7 [disabled] app-shells/bash: 5.0_p16::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.17-r1::gentoo, 3.6.10::gentoo, 3.7.6::gentoo, 3.8.1::gentoo dev-util/ccache: 3.7.7::gentoo dev-util/cmake: 3.16.4::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.7::gentoo sys-apps/sandbox: 2.18::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r5::gentoo sys-devel/automake: 1.11.6-r3::gentoo, 1.13.4-r2::gentoo, 1.15.1-r2::gentoo, 1.16.1-r2::gentoo sys-devel/binutils: 2.33.1-r1::gentoo sys-devel/gcc: 6.5.0-r1::gentoo, 9.2.0-r4::gentoo sys-devel/gcc-config: 2.2.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.5::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r3::gentoo Repositories: gentoo location: /var/cache/kobboi/gentoo sync-type: git sync-uri: https://github.com/kobboi/gentoo.git priority: -1000 Installed sets: @kobboi-all ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=x86-64 -pipe -fomit-frame-pointer -g -ggdb" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.6/conf /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=x86-64 -pipe -fomit-frame-pointer -g -ggdb" DISTDIR="/usr/portage/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" INSTALL_MASK="/usr/src/debug /usr/lib/debug" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" PKGDIR="/usr/portage/packages" PORTAGE_BINHOST="ssh://christophe@10.8.0.6/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi alsa amd64 avahi berkdb bluetooth bzip2 cairo cdda cdr cli colord crypt cups cxx dbus djvu dri dts dvd dvdr eds egl emboss encode evo exif ffmpeg flac fluidsynth fortran frei0r gdbm gif glade gles gnome gnome-keyring gnome-online-accounts gpm gstreamer gtk gtk3 gtkstyle iconv icu introspection ipv6 jpeg lcms ldap libass libnotify libsecret libtirpc mad midi mng modemmanager mono mp3 mp4 mpeg mtp multilib nautilus ncurses networkmanager nls nptl ogg opengl openmp opus pam pango pcre pdf png policykit ppds pulseaudio qt5 readline samba sdl seccomp smartcard smp spell split-usr ssl startup-notification svg systemd tcpd tiff truetype udev udisks unicode upower usb vaapi vdpau vorbis wayland wxwidgets x264 xattr xcb xml xv xvid zeroconf zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="hda-intel ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse synaptics libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="vesa radeon nvidia intel amdgpu dummy fbdev nouveau radeonsi v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, LC_ALL, LINGUAS, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
I downgraded to [ebuild UD] net-misc/openssh-8.1_p1-r2 [8.2_p1] USE="X pam pie ssl -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp (-selinux) -static -test -xmss" keeping sshd running. Everything is OK again now.
It looks like this only happens after updating a server to openssh-8.2_p1, but not restarting sshd (I managed to reproduce it) I would suspect this is an upstream issue, I am not sure if they support updating openssh, but not restarting sshd.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3543f68d6fa8cce8c957e02a2aaae79914f40441 commit 3543f68d6fa8cce8c957e02a2aaae79914f40441 Author: Patrick McLean <chutzpah@gentoo.org> AuthorDate: 2020-02-16 06:43:05 +0000 Commit: Patrick McLean <chutzpah@gentoo.org> CommitDate: 2020-02-16 06:46:40 +0000 net-misc/openssh-8.2_p1-r1: ewarn to restart sshd (bug #709748) Bug: https://bugs.gentoo.org/709748 Package-Manager: Portage-2.3.89, Repoman-2.3.20 Signed-off-by: Patrick McLean <chutzpah@gentoo.org> net-misc/openssh/openssh-8.2_p1-r1.ebuild | 6 ++++++ 1 file changed, 6 insertions(+)
I can confirm having the Comment 4 change in the ebuild /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild and I'm upgrading from 8.1_p1-r2 to 8.2_p1-r1 but I cannot see the ewarns neither on console nor in /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-171251.log nor in /tmp/ebuild.logs/elog/summary.log My ELOG settings seem to be default: PORTAGE_ELOG_CLASSES="log warn error" PORTAGE_ELOG_MAILFROM="portage@localhost" PORTAGE_ELOG_MAILSUBJECT="[portage] ebuild log for ${PACKAGE} on ${HOST}" PORTAGE_ELOG_MAILURI="root" PORTAGE_ELOG_SYSTEM="save_summary:log,warn,error,qa echo" except for: PORT_LOGDIR="/tmp/ebuild.logs/" # time emerge -v openssh These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] net-misc/openssh-8.2_p1-r1::gentoo [8.1_p1-r2::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key% (-selinux) -static -test -xmss" 0 KiB Total: 1 package (1 upgrade), Size of downloads: 0 KiB Would you like to merge these packages? [Yes/No] Sorry, response '' not understood. [Yes/No] y >>> Verifying ebuild manifests >>> Running pre-merge checks for net-misc/openssh-8.2_p1-r1 >>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo >>> Installing (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo >>> Jobs: 1 of 1 complete Load avg: 0.66, 0.28, 0.21 * Messages for package net-misc/openssh-8.2_p1-r1: * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-171251.log * User patches applied. >>> Auto-cleaning packages... >>> No outdated packages were found on your system. * GNU info directory index is up-to-date. real 1m10.063s user 0m27.970s sys 0m16.215s # grep -F After /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-171251.log /tmp/ebuild.logs/elog/summary.log /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild: ewarn "After upgrading to openssh-8.2p1 please restart sshd, otherwise you" Some already mentioned info: because I didn't `rc-service sshd restart` after upgrading(to net-misc/openssh-8.2_p1-r1) from prev. version net-misc/openssh-8.1_p1-r2 on client's side, trying to connect to sshd: kex_exchange_identification: read: Connection reset by peer on sshd's side: Feb 16 17:51:35 somehost sshd[29329]: debug3: fd 4 is not O_NONBLOCK Feb 16 17:51:35 somehost sshd[29329]: debug1: Forked child 9073. Feb 16 17:51:35 somehost sshd[29329]: debug3: send_rexec_state: entering fd = 7 config len 1310 Feb 16 17:51:35 somehost sshd[29329]: debug3: ssh_msg_send: type 0 Feb 16 17:51:35 somehost sshd[29329]: debug3: send_rexec_state: done Feb 16 17:51:35 somehost sshd[9073]: debug3: oom_adjust_restore Feb 16 17:51:35 somehost sshd[9073]: debug1: Set /proc/self/oom_score_adj to 0 Feb 16 17:51:35 somehost sshd[9073]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7 Feb 16 17:51:35 somehost sshd[9073]: fatal: recv_rexec_state: buffer error: incomplete message
my test shows that 'if' isn't entered, in this case: # time emerge -v --ask n =openssh-8.1_p1-r2 && time emerge -v --ask n openssh These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild UD ] net-misc/openssh-8.1_p1-r2::gentoo [8.2_p1-r1::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp (-selinux) -static -test -xmss (-security-key%)" 0 KiB Total: 1 package (1 downgrade), Size of downloads: 0 KiB >>> Verifying ebuild manifests >>> Running pre-merge checks for net-misc/openssh-8.1_p1-r2 >>> Emerging (1 of 1) net-misc/openssh-8.1_p1-r2::gentoo >>> Installing (1 of 1) net-misc/openssh-8.1_p1-r2::gentoo >>> Jobs: 1 of 1 complete Load avg: 1.15, 0.32, 0.16 * Messages for package net-misc/openssh-8.1_p1-r2: * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.1_p1-r2:20200216-174141.log * User patches applied. >>> Auto-cleaning packages... >>> No outdated packages were found on your system. * GNU info directory index is up-to-date. * IMPORTANT: 2 config files in '/etc' need updating. * See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS * sections of the emerge man page to learn how to update config files. real 0m40.739s user 0m28.359s sys 0m16.714s These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] net-misc/openssh-8.2_p1-r1::gentoo [8.1_p1-r2::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key% (-selinux) -static -test -xmss" 0 KiB Total: 1 package (1 upgrade), Size of downloads: 0 KiB >>> Verifying ebuild manifests >>> Running pre-merge checks for net-misc/openssh-8.2_p1-r1 >>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo >>> Installing (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo >>> Jobs: 1 of 1 complete Load avg: 1.08, 0.39, 0.20 * Messages for package net-misc/openssh-8.2_p1-r1: * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-174221.log * User patches applied. >>> Auto-cleaning packages... >>> No outdated packages were found on your system. * GNU info directory index is up-to-date. * IMPORTANT: 2 config files in '/etc' need updating. * See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS * sections of the emerge man page to learn how to update config files. real 0m38.517s user 0m27.660s sys 0m16.458s $ grep -nF 'hdjkshdskjahdsljkadhsak' /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-174221.log /tmp/ebuild.logs/elog/summary.log /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:472: echo "hdjkshdskjahdsljkadhsak" $ grep -nF 'gurl' /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-174221.log /tmp/ebuild.logs/elog/summary.log /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:471: elog "whatever gurl" this 'if': if has_version "<${CATEGORY}/${PN}-8.2_p1"; then ewarn "After upgrading to openssh-8.2p1 please restart sshd, otherwise you" ewarn "will not be able to establish new sessions. Restarting sshd over a ssh" ewarn "connection is generally safe." elog "whatever gurl" echo "hdjkshdskjahdsljkadhsak" fi Perhaps I'm missing something? I know I usually do! kinda curious what's going on here
Found the doc for has_version here: https://devmanual.gentoo.org/eclass-reference/ebuild/index.html has_version [-b] [-d] [-r] [--host-root] <category/package-version> Check to see if category/package-version is installed. The parameter accepts all values that are acceptable in the DEPEND variable. The function returns 0 if category/package-version is installed, 1 otherwise. The package is searched for in ROOT by default. In EAPI 5 and EAPI 6, the package is searched for in the build host if the --host-root option is given. In EAPI 7 and later, the confusing --host-root option has been replaced with -b, which corresponds to a dependency satisfied by BDEPEND in the build host. Similarly, the -d option corresponds to DEPEND in SYSROOT and the -r option corresponds to RDEPEND in ROOT. since that 'if' is inside pkg_postinst() does that mean that the currently emerging openssh version is considered installed? then it would never be less than itself?
ok this 'if' enters(and I've no idea why): if has_version ">${CATEGORY}/${PN}-8.2_p1"; then ie. if has_version "<${CATEGORY}/${PN}-8.2_p1"; then elog "lower" fi if has_version "=${CATEGORY}/${PN}-8.2_p1"; then elog "equal" fi if has_version ">${CATEGORY}/${PN}-8.2_p1"; then elog "bigger" fi [ebuild U ] net-misc/openssh-8.2_p1-r1::gentoo [8.1_p1-r2::gentoo] USE="X bindist pam pie ssl -X509 -audit -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key% (-selinux) -static -test -xmss" 0 KiB Total: 1 package (1 upgrade), Size of downloads: 0 KiB >>> Verifying ebuild manifests >>> Running pre-merge checks for net-misc/openssh-8.2_p1-r1 >>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo >>> Installing (1 of 1) net-misc/openssh-8.2_p1-r1::gentoo >>> Jobs: 1 of 1 complete Load avg: 0.88, 0.29, 0.15 * Messages for package net-misc/openssh-8.2_p1-r1: * Log file: /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-175825.log * User patches applied. * bigger >>> Auto-cleaning packages... $ grep -nE '(lower|bigger|equal)' /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild /tmp/ebuild.logs/elog/summary.log /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-175825.log /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:468: elog "lower" /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:471: elog "equal" /var/db/repos/gentoo/net-misc/openssh/openssh-8.2_p1-r1.ebuild:474: elog "bigger" /tmp/ebuild.logs/elog/summary.log:165:bigger /tmp/ebuild.logs/build/net-misc/openssh-8.2_p1-r1:20200216-175825.log:1710: * bigger
Ok I think I get it, it returns bigger because the currently installed version has the -r1 and it is bigger than the one that lacks -r1 $ portageq has_version / '=openssh-8.2_p1'; echo -n "$? " ; portageq has_version / '>openssh-8.2_p1'; echo -n "$?" 1 0 $ portageq has_version / '=openssh-8.2_p1-r1'; echo -n "$? " ; portageq has_version / '>openssh-8.2_p1-r1'; echo -n "$?" 0 1
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a225fe10e4c21edd8915543c2a4318b00d2144c6 commit a225fe10e4c21edd8915543c2a4318b00d2144c6 Author: Patrick McLean <chutzpah@gentoo.org> AuthorDate: 2020-02-16 18:29:52 +0000 Commit: Patrick McLean <chutzpah@gentoo.org> CommitDate: 2020-02-16 18:30:41 +0000 net-misc/openssh-8.1_p1-r2: Disable X509 and security-key (bug #709808) This also makes the warning about restarting sshd actually show when it is intended to. This refactors all version warnings by using a flag variable set in pkg_preinst to decide whether to show the warning in pkg_postinst. Closes: https://bugs.gentoo.org/709808 Bug: https://bugs.gentoo.org/709748 Package-Manager: Portage-2.3.89, Repoman-2.3.20 Signed-off-by: Patrick McLean <chutzpah@gentoo.org> net-misc/openssh/openssh-8.2_p1-r1.ebuild | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
>>> Verifying ebuild manifests >>> Running pre-merge checks for net-misc/openssh-8.2_p1-r2 >>> Emerging (1 of 1) net-misc/openssh-8.2_p1-r2::gentoo >>> Installing (1 of 1) net-misc/openssh-8.2_p1-r2::gentoo >>> Jobs: 1 of 1 complete Load avg: 2.32, 1.36, 0.90 * Messages for package net-misc/openssh-8.2_p1-r2: * After upgrading to openssh-8.2p1 please restart sshd, otherwise you * will not be able to establish new sessions. Restarting sshd over a ssh * connection is generally safe. >>> Auto-cleaning packages... >>> No outdated packages were found on your system. I think the above is what is intended to resolve this. That works for me. With a restart, ssh connections work as expected. I suppose there's no way to automatically restart the service at the completion of the package installation?
Restarting SSHd or any other daemon after any upgrade is best practice. Issues like this have risen for SSHd recurrently in the last two decades. And not just for SSHd, even though for SSHd it can really bite you.
This just took down my VPS, which I'm currently rebuilding from a 2-year-old snapshot. I don't know what the best way is to make this message as loud as possible, but please make sure it is.
ArchLinux opted to auto-restart sshd(iff it was already started), but they're only dealing with systemd not (also)openrc. https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/openssh&id=042a0ad1ac04ff420d61f3acfdf196f1a5d7d9ef I'm not sure it makes sense for Gentoo to do this. I assume people are upgrading over ssh, which upon seeing the message, would manually restart sshd. So, it seems ok as it is now, imho. I don't understand how @matoro's VPS got taken down because of this...
(In reply to gentoo_eshoes from comment #14) > ArchLinux opted to auto-restart sshd(iff it was already started), but > they're only dealing with systemd not (also)openrc. > > https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/ > openssh&id=042a0ad1ac04ff420d61f3acfdf196f1a5d7d9ef > > I'm not sure it makes sense for Gentoo to do this. I assume people are > upgrading over ssh, which upon seeing the message, would manually restart > sshd. So, it seems ok as it is now, imho. > > I don't understand how @matoro's VPS got taken down because of this... It's an ssh-only VPS, and it's not designed to be rebooted. Meaning if sshd goes down, the web services stay running but it's permanently inaccessible. I don't think these are that uncommon - maybe a combination of bad circumstances, but there was no warning at all at the time. This definitely needs to be *at least* as loud and annoying as that sys-fs/eudev warning about the renamed interfaces - if we don't just restart it for people altogether (should be pretty easy to check common scenarios - openrc, systemd service, systemd socket).
Agreed, another case scenario is where updates to a headless system are done from detached 'screen' session where after sshd got upgraded, the system remained out of administrative control.
I was just hit by this, on a machine which is difficult to manually reboot. This was a similar case to Nikolay's -- the upgrade was happening invisibly, and completed at a time when I had no active sessions. I believe that Arch's approach is the correct one -- if the ssh daemon always needs restarting, then I think that portage should do it. In cases where there are more options to consider, then einfo/ewarn are fine. Since openssh-8.4_p1-r2 was made stable on most architectures around 25 Dec - I expect that this problem will affect most of the community in the coming days. Now would be a good time to take action to avoid locking more people out of their systems.
Ditto the above. I was upgrading remotely on several machines, using 'nohup emerge...' so that the jobs could complete without my having to stay logged in. In light of portage's not restarting the server after an upgrade (which I guess I hadn't expected), this makes the 'nohup ...' approach intrinsically dangerous. This in turn makes bulk (multi-machine) upgrades extremely inconvenient. Problem was aggravated in this case when my local internet went out during the upgrade; job finished, but new session required when my connectivity returned has meant that I am now shut out of all those machines--and I am 1000 miles away. Only hope of re-connecting is to have somebody power-cycle all the machines for me. I would lobby for portage to do as much as possible to maintain state and integrity following any upgrade; this might still be a job for post-intall on any given package.
(In reply to James Broadhead from comment #17) > Since openssh-8.4_p1-r2 was made stable on most architectures around 25 Dec > - I expect that this problem will affect most of the community in the coming > days. Now would be a good time to take action to avoid locking more people > out of their systems. I concur. Assuming the use of portage as a sub-process of an interactive shell that the user has control over for the duration of any given operation doesn't cut it. I, too, was affected by this, though I had the good fortune of having out-of-band management facilities at my disposal, so recovering was only a matter of tedium. It is highly unusual for an immediate restart to be imperative upon upgrading openssh. However, this incident suggests that its developers are under no obligation to maintain inter-process compatibility between major releases. I don't much like the idea of ebuilds being able to restart services. Yet, if there is even the slightest chance that it could happen again in the future, I think that the possibility should be considered, given the obvious importance of sshd. This is just a thought, but perhaps the ebuild could elect to do so while avoiding it in the case that a major version upgrade/downgrade has not occurred. The following code demonstrates the concept. pkg_postinst() { local ver declare -a versions read -ra versions <<<"${REPLACING_VERSIONS}" for ver in "${versions[@]}"; do if [[ ${ver%_*} == "${PV%_*}" ]]; then # No major version change has occurred return fi done if [[ -d /run/systemd/system ]]; then systemctl try-restart sshd else rc-service -q --ifstarted sshd restart fi }
I incorporated the previously mentioned code to a local copy of the ebuild and found that it works as expected. An upgrade from 8.2 to 8.4 triggered a restart, whereas re-emerging 8.4 did not. I'll be using that to upgrade the remainder of my fleet.
I was similarly shocked but this was actually part of a news item (it was just so long ago I forgot, as the stabilisation only recently happened): https://www.gentoo.org/support/news-items/2020-02-19-openssh-8_2-service-breakage.html. I would like to see some mechanism for restarting services built-in or easily accessible, and also think that such a snippet as Kerin gave is a reasonable compromise (major versions only).
(In reply to Sam James from comment #21) > I was similarly shocked but this was actually part of a news item (it was > just so long ago I forgot, as the stabilisation only recently happened): > https://www.gentoo.org/support/news-items/2020-02-19-openssh-8_2-service- > breakage.html. I now recall reading this but had also completely forgotten.
(In reply to Sam James from comment #21) > I would like to see some mechanism for restarting services built-in or > easily accessible, and also think that such a snippet as Kerin gave is a > reasonable compromise (major versions only). Though service restarts will be helpful for sshd issues like this one, when updating live systems, users also need to be well prepared for cases where a simple service restart will not correct the problem. A good way to prepare for this sort of scenario is to create a failsafe sshd instance that listens on a different port from the regular instance. The dependencies of the failsafe sshd instance should be as isolated as possible from the regular system. For example, it can run in a chroot (which you can escape after login), or be built with USE="static -pam" in order to make it immune to breakage in shared libraries or pam configuration.
Just encountered this while running world update. net-misc/openssh Available versions: 8.1_p1-r5^t ~8.2_p1-r8^t ~8.3_p1-r6^t 8.4_p1-r3^t {X X509 audit bindist debug hpn kerberos ldns libedit libressl livecd pam +pie +scp sctp security-key selinux +ssl static test xmss ABI_MIPS="n32" KERNEL="linux"} Installed versions: 8.4_p1-r3^t(11:30:57 AM 02/03/2021)(X pam pie scp ssl -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -sctp -security-key -selinux -static -test -xmss ABI_MIPS="-n32" KERNEL="linux") Homepage: https://www.openssh.com/ Description: Port of OpenBSD's free SSH release Could not open another SSH session, so sent upgrade to background and ran systemctl try-restart sshd to resolve the issue. recv_rexec_state: buffer error: incomplete message was the original error while connecting via another ssh session. Thank you for the bug and the comments!!
I think the lesson here is largely to coordinate news item with stabilisation more so people don't forget, and also if it happens again, consider our options carefully in the ebuild. But no real point in leaving this one open.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b9aab3ef968b7a6d58fa215223d116b98af7d399 commit b9aab3ef968b7a6d58fa215223d116b98af7d399 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-07-01 09:59:36 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-07-01 10:22:02 +0000 net-misc/openssh: restart sshd on major version upgrades openssh-9.8_p1 again breaks cross-version compatibility, meaning that a running sshd with 9.7_p1 will no longer be able to accept connections after upgrading to 9.8_p1. We tried doing a news item on this in the past (bug #709748) and it ended up being insufficient and poorly coordinated (as you really need it again when stabling). Nobody is going to thank us for leaving their sshd broken, so pick the lesser evil and attempt to restart sshd on major version upgrades. This is especially important as people may be racing to upgrade to 9.8_p1 for the CVE-2024-6387 fix (although we have backported a fix to older versions). I also note there's precedent here with e.g. the systemd rebuild where it's done to avoid immediate breakage of user sessions. Thanks to kerframil who proposed a snippet for this some time ago whose work I've lifted here. Bug: https://bugs.gentoo.org/709748 Bug: https://bugs.gentoo.org/935271 Signed-off-by: Sam James <sam@gentoo.org> ...nssh-9.8_p1.ebuild => openssh-9.8_p1-r1.ebuild} | 33 ++++++++++++++++++++++ 1 file changed, 33 insertions(+)
In view of the commit preceding this comment, I'd suggest having the /run/openrc test precede the /run/systemd test. Otherwise, it can fall through to testing for openrc in the case that systemd is in use but the config check fails. Such a fallthrough is generally harmless but the logic is still mildly jarring.