Disclaimer: I have heeded the related news all right, since it says I am not concerned if I never touched the pam config: https://www.gentoo.org/support/news-items/2020-06-23-upgrade-to-sys-libs_pam-1_4_0.html Extra disclaimer: dispatch-conf, which I had been running diligently after each update, showed no config to update, at all, even during the ongoing update while this happened. In the middle of today's system update, I noticed that it was impossible to connect to the box via ssh. This looked like: ~ $ ssh <IP address here> Connection closed by <IP address here> Thankfully I still had an open ssh connection to the machine, with which I could see that the syslog contained the following messages at the timestamp of each ssh connection attempt: sshd[30418]: PAM unable to dlopen(/lib64/security/pam_tally2.so): /lib64/security/pam_tally2.so: cannot open shared object file: No such file or directory sshd[30418]: PAM adding faulty module: /lib64/security/pam_tally2.so sshd[30418]: PAM unable to dlopen(/lib64/security/pam_cracklib.so): /lib64/security/pam_cracklib.so: cannot open shared object file: No such file or directory sshd[30418]: PAM adding faulty module: /lib64/security/pam_cracklib.so Here is a snapshot of the terminal output at that moment: ~ # emerge --update --deep --newrepo --newuse --changed-deps --with-bdeps=y --keep-going -j2 @world These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] dev-util/re2c-2.0.3 [1.3-r1] [ebuild U ] app-shells/quoter-4.2 [3.0_p2-r1] [ebuild U ] media-libs/lcms-2.11 [2.9] [ebuild R ] net-libs/miniupnpc-2.1.20191224 USE="-verify-sig%" [ebuild U ] dev-lang/python-exec-2.4.6-r2 [2.4.6-r1] [ebuild U ] dev-lang/python-2.7.18-r4 [2.7.18-r2] [blocks b ] <=dev-lang/python-2.7.18-r3:2.7 ("<=dev-lang/python-2.7.18-r3:2.7" is blocking dev-lang/python-exec-2.4.6-r2) [ebuild U ] sys-power/acpid-2.0.32-r2 [2.0.32-r1] [ebuild U ] app-portage/gentoolkit-0.5.0-r1 [0.5.0] [ebuild U ] sys-libs/pam-1.4.0_p20200829 [1.3.1_p20200128-r1] [ebuild N ] sys-auth/passwdqc-1.4.0-r1 [ebuild U ] dev-lang/rust-1.46.0 [1.45.2] [ebuild N ] media-libs/dav1d-0.7.0 USE="10bit 8bit asm" [ebuild N ] dev-python/markupsafe-1.1.1-r1 USE="-test" PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8 (-python3_9)" [ebuild N ] dev-python/jinja-2.11.2-r1 USE="-doc -examples -test" PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8 (-python3_9)" [ebuild U ] sys-auth/pambase-20201010 [20200304] USE="passwdqc* -gnome-keyring% -pwhistory% -pwquality%" [ebuild U ] dev-python/html5lib-1.1 [1.0.1-r2] [ebuild U ] virtual/rust-1.46.0 [1.45.2] [ebuild U ] dev-db/mariadb-10.4.13-r3 [10.4.13-r2] [ebuild U ] x11-libs/libva-intel-driver-2.4.1 [2.3.0] [ebuild U ] media-video/ffmpeg-4.3.1 [4.2.4] USE="dav1d* -rav1e% -vulkan%" Would you like to merge these packages? [Yes/No] >>> Verifying ebuild manifests >>> Running pre-merge checks for sys-power/acpid-2.0.32-r2 * Determining the location of the kernel source code * Found kernel source directory: * /usr/src/linux * Found sources for kernel version: * 5.4.66-gentoo * Checking for suitable kernel configuration options... [ ok ] >>> Running pre-merge checks for dev-lang/rust-1.46.0 * Checking for at least 9216 MiB disk space at "/home/portage-tmp/portage/dev-lang/rust-1.46.0/temp" ... [ ok ] >>> Emerging (1 of 20) dev-util/re2c-2.0.3::gentoo >>> Emerging (2 of 20) app-shells/quoter-4.2::gentoo >>> Installing (2 of 20) app-shells/quoter-4.2::gentoo >>> Emerging (3 of 20) media-libs/lcms-2.11::gentoo >>> Installing (3 of 20) media-libs/lcms-2.11::gentoo >>> Emerging (4 of 20) dev-lang/python-exec-2.4.6-r2::gentoo >>> Installing (1 of 20) dev-util/re2c-2.0.3::gentoo >>> Installing (4 of 20) dev-lang/python-exec-2.4.6-r2::gentoo >>> Emerging (5 of 20) net-libs/miniupnpc-2.1.20191224::gentoo >>> Installing (5 of 20) net-libs/miniupnpc-2.1.20191224::gentoo >>> Emerging (6 of 20) dev-lang/python-2.7.18-r4::gentoo >>> Installing (6 of 20) dev-lang/python-2.7.18-r4::gentoo >>> Emerging (7 of 20) sys-power/acpid-2.0.32-r2::gentoo >>> Emerging (8 of 20) app-portage/gentoolkit-0.5.0-r1::gentoo >>> Installing (8 of 20) app-portage/gentoolkit-0.5.0-r1::gentoo >>> Emerging (9 of 20) sys-libs/pam-1.4.0_p20200829::gentoo >>> Installing (7 of 20) sys-power/acpid-2.0.32-r2::gentoo >>> Installing (9 of 20) sys-libs/pam-1.4.0_p20200829::gentoo >>> Emerging (10 of 20) sys-auth/passwdqc-1.4.0-r1::gentoo >>> Installing (10 of 20) sys-auth/passwdqc-1.4.0-r1::gentoo >>> Emerging (11 of 20) dev-lang/rust-1.46.0::gentoo >>> Jobs: 10 of 20 complete, 1 running Load avg: 4.38, 4.42, 4.36 In the above, notice how pambase 20201010 isn't merged yet, but pam 1.4 is. Conclusion: if the update had failed midway for whatever reason, this would result in being locked out of SSH. As it happens, this closely resulted in a several-hour lockout; as soon as I noticed the cause of the error, I commented out the offending lines in /etc/pam.d/system-login in case my flaky Internet connection would cut. This is rather a post-mortem, but may affect people updating their system in the very close future.
A way to avoid this sort of scenario was suggested: merging pam and pambase.
I talked about this with Zlogene and we're not really sure how this could be avoided. There are only two options: 1) Forcing an upgrade somehow immediately before/after pam 2) Merging them into the same package (which has been considered for other reasons in the past) I'll CC portage-dev in case they have any ideas on how to do 1) here but in some sense, this is an unavoidable problem. If you were running the update in e.g. screen or tmux, and the update was allowed to finish, you would be fine. Interrupting updates in general is not a solved problem in Gentoo. I think I have heard something like libostree could somehow solve that, but we're not using that right now.
To clarify, I was running this in a screen but was worried that if the build of another package failed for whatever reason, emerge would exit between the updates of pam and pambase, leaving the system in that state. This is, however, exactly the reason I pass --keep-going to emerge.
(In reply to Sam James from comment #2) > If you were running the update in e.g. screen or tmux, and the update was > allowed to finish, you would be fine. Interrupting updates in general is not > a solved problem in Gentoo. I solve this for myself by creating a btrfs subvolume snapshot of my root filesystem, updating the snapshot in a chroot, and then rebooting into the updated snapshot. > I think I have heard something like libostree could somehow solve that, but > we're not using that right now. That would be very simliar to my btrfs approach. The user has to reboot into the new image, which is a much different user experience than the sort of in-place update that can temporarily break PAM authentication.
