Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 802267

Summary: sys-libs/libxcrypt-4.4.23-r1: On first login after migration from sys-libs/glibc[crypt], password may need to be changed without warning, breaking setups with / read-only, locking out users
Product: Gentoo Linux Reporter: Duncan <1i5t5.duncan>
Component: Current packagesAssignee: Gentoo Toolchain Maintainers <toolchain>
Status: RESOLVED FIXED    
Severity: normal CC: ionen, sam
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 802807    
Bug Blocks: 699422    

Description Duncan 2021-07-15 10:53:05 UTC
[After successfully working around bug #802207.]

sys-libs/libxcrypt-4.4.23-r1 needs an ewarn and the libxcrypt-migration news item needs an update to reflect that (likely only for old installations with outdated password hash types?) all system passwords may need changed on (I think) first password login to each user. (For me it was first reboot, but I think that was because I hadn't relogged in until the reboot.)

The problem here is that my / is mounted ro by default.  Because the password-change prompt came up on login before giving me a shell prompt, I couldn't remount / rw in ordered to allow the password change to take, so it wouldn't let me past that in ordered to mount rw in ordered to actually let it take.

If I'd have know it was going to prompt for a password change I could have mounted / rw and logged in (in another terminal-window/VT) each of my user hats to do the password change, before fully logging out and rebooting.  Since I didn't, after figuring out the problem given the confusing error message that didn't tell me the real problem at all, I had to reboot to the backup, mount the normal working / and change its fstab to mount it rw by default, before rebooting again to the normal working root with it default-mounted rw so I could change the passwords, then change its fstab back to ro by default and remount it.

Also, consider adding a suggestion to the news item and a pkg_pretend message for the glibc upgrade strongly recommending that one have an extra root login session going during the upgrade to take care of any problems during the upgrade, since if there /are/ problems it may not be possible to authenticate again until they are fixed.  And after the upgrade, do not logout of that root login until a successful root/admin relogin has been done, to be sure it's possible before losing that existing root login.  And before the glibc/libxcrypt updates, verify that system backups are current and emergency-boot procedures work, just in case.

Because this is the second bug for this upgrade that would have locked me out if I didn't have a good emergency-bootable system backup, when my working system locked me out due to bugs with this upgrade.
Comment 1 Mikle Kolyada (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2021-07-15 16:07:19 UTC
(In reply to Duncan from comment #0)
> [After successfully working around bug #802207.]
> 
> sys-libs/libxcrypt-4.4.23-r1 needs an ewarn and the libxcrypt-migration news
> item needs an update to reflect that (likely only for old installations with
> outdated password hash types?) all system passwords may need changed on (I
> think) first password login to each user.

I am really having hard times understanding your logic here. There is no "old" or "new" password hashes per se. We enforce sha512 by default via pam_unix.so, both glibc's and libxcrypt's crypt() support it.
Comment 2 Duncan 2021-07-15 19:06:11 UTC
(In reply to Mikle Kolyada from comment #1)
> (In reply to Duncan from comment #0)
> > [After successfully working around bug #802207.]
> > 
> > sys-libs/libxcrypt-4.4.23-r1 needs an ewarn and the libxcrypt-migration news
> > item needs an update to reflect that (likely only for old installations with
> > outdated password hash types?) all system passwords may need changed on (I
> > think) first password login to each user.
> 
> I am really having hard times understanding your logic here. There is no
> "old" or "new" password hashes per se. We enforce sha512 by default via
> pam_unix.so, both glibc's and libxcrypt's crypt() support it.

The old installations thing *was* just a guess, based on an old memory of password-hashing updates years ago, but I *could* *have* been mistaken as I in no way claim any expertise in the area and it /was/ a long time ago.  Keep in mind that the gentoo installation on this box is /quite/ old, now over 1.7 decades as I installed from Gentoo 2004.1 and have been updating both hardware and software on the same base since.

The box is limited-physical-access (normally just me) and behind both a NAPT-based router (with the automatic UPNP hole-carving disabled) and an on-box IPTables firewall, and not running any servers on anything but localhost, so in theory I could probably run passwordless, but I was never quite comfortable with that.  However, I never felt the need to change the passwords and never had, since the installation isn't encrypted meaning anyone with possession and wanting access bad enough has it, in practice meaning any passwords are really no more than default-boot login obfuscation no matter how strong they are.

Meanwhile, the "*was*" in the first paragraph is correct.  After a bit of research to figure out how I was able to actually verify the hash in /etc/shadow.

Turns out my dim memory was correct.  Original passwords (verified in a backup) were $1$ aka MD5.  That and gentoo's default sha-512 enforcing would be what forced the password change on first login after the upgrade to split libxcrypt -- whatever enforcing was being done before apparently never triggered here, but the upgrade triggered it.  And FWIW, the new ones are $6$ aka sha-512 as expected.

So yeah, as I said, a warning that the upgrade may trigger (quite?) old installations to force a password change and to be prepared for it (rw-mounted /etc and preferably keep an active root login from before the upgrade until after first post-upgrade login) just in case, would have been useful.
Comment 3 Mikle Kolyada (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2021-07-17 20:46:11 UTC
(In reply to Duncan from comment #2)
> (In reply to Mikle Kolyada from comment #1)
> > (In reply to Duncan from comment #0)

> So yeah, as I said, a warning that the upgrade may trigger (quite?) old
> installations to force a password change and to be prepared for it
> (rw-mounted /etc and preferably keep an active root login from before the
> upgrade until after first post-upgrade login) just in case, would have been
> useful.

But what is the point?

The crypt() calls are still irrelevant. sha512 encryption was added as a pambase feature on Aug 01 2008. It is now an optional feature enabled in pambase by default. If you disable a sha512 USE on pambase you are going to end up with old md5 encryption (also supported by libxcrypt as per their github).
Comment 4 Duncan 2021-07-18 03:33:35 UTC
(In reply to Mikle Kolyada from comment #3)
> (In reply to Duncan from comment #2)
> > [snip]
> 
> But what is the point?
> 
> The crypt() calls are still irrelevant. sha512 encryption was added as a
> pambase feature on Aug 01 2008. It is now an optional feature enabled in
> pambase by default. If you disable a sha512 USE on pambase you are going to
> end up with old md5 encryption (also supported by libxcrypt as per their
> github).

Sorry.  Something's not communicating and it seems to be frustrating us both.

The point is, I didn't disable it, pambase USE=sha512.  But until this upgrade, nothing had ever forced me to change passwords so they were still md5.  On login after the libxcrypt upgrade I got an unexpected forced password change.  I don't know why it forced the password change now and never did before, but it did.  But the forced password change, had I known about it, isn't a problem.

The problem is I didn't know it was going to happen, so when I rebooted to the default-read-only root (including /etc) and got the unexpectedly forced password change at login, the password couldn't be written, resulting in a confusing token-busy error that didn't hint at the real problem (the read-only filesystem) and locking me out until I booted the backup and arranged for the (normal-working not the backup I was then on) root filesystem to be mounted writable on next reboot, so the password change could take.

All I'm asking is for a warning that a password change /may/ be necessary and that an admin should be prepared for it with this upgrade if it does, including writable /etc on next login after upgrade (which for me was at reboot but I'd have done it with the freshly upgraded and still writable root before reboot if I had that warning), because if they're unprepared it could lock them out if the password change can't be written, as it did me.
Comment 5 cyrillic 2021-07-18 14:14:29 UTC
Thank you for the hint.

After completing this migration, I was forced to change my password after *every* login, not just the first.

I realized that I had sys-auth/pambase[-sha512] and my new passwords were still using MD5, and there was no message saying that this was not acceptable anymore.
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-07-18 14:48:20 UTC
Yeah, that aligns with my understanding, thanks both. Had a report on IRC too. I’ll update the news item tomorrow.
Comment 7 Duncan 2021-07-18 22:12:42 UTC
So cyrillic's comment #5 stimulated my curiosity and being a bit bored I spent several hours tracking this down upstream today...

The original issue filed against libxcrypt is here (tho it's on sha256 but it's the same issue, resulting in an arch user getting a forced-password-change on upgrade and locked out due to read-only root): https://github.com/besser82/libxcrypt/issues/130 .

But it turned out to be a pam bug which was filed here, https://github.com/linux-pam/linux-pam/issues/367 .  That resulted in an upstream pam commit-revert on June 10 that should be in the next release -- the forced-password-updating wasn't intended.

I'll file a gentoo bug on pam asking for the revert-patch cherry-pick with the details, then link it here.

But all things considered, this is anything but an ordinary upgrade, and a warning suggesting people approach it with appropriate caution, an extra root login at the ready and testing a new login after the upgrade before logging out, can't be anything but a good thing, even if the particular trigger that hit me (and cyrillic) is fixed.
Comment 8 Duncan 2021-07-18 23:07:53 UTC
(In reply to Duncan from comment #7)
> I'll file a gentoo bug on pam asking for the revert-patch cherry-pick with
> the details, then link it here.

Bug #802807: https://bugs.gentoo.org/802807
Comment 9 Mikle Kolyada (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2021-07-19 20:14:07 UTC
As new pam snap is in the tree this bug should be solved.
Comment 10 Duncan 2021-07-19 20:51:36 UTC
(In reply to Mikle Kolyada from comment #9)
> As new pam snap is in the tree this bug should be solved.

Umm... "snap" had me *very* confused for a moment!

You did /not/ mean the Ubuntu/whatever binary all-in-one package type, but rather, "snapshot"!

Or at least I /hope/ that's the case! =:^)

Symptom of being a Linux-sphere news reader, I guess.
Comment 11 Larry the Git Cow gentoo-dev 2021-07-22 23:41:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/data/gentoo-news.git/commit/?id=d6020a98d364bbc0c0c8f8d5f5efde5bcb1b383e

commit d6020a98d364bbc0c0c8f8d5f5efde5bcb1b383e
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-07-20 17:42:12 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-07-22 23:41:09 +0000

    2021-06-30-libxcrypt-migration: mention PAM/md5crypt issues
    
    Bug: https://bugs.gentoo.org/802267
    Bug: https://bugs.gentoo.org/802807
    Signed-off-by: Sam James <sam@gentoo.org>

 .../2021-06-30-libxcrypt-migration.en.txt                     | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)