Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 821496 - sys-libs/libxcrypt-4.4.25 built using the gold linker on ppc64le generates 'illegal instruction' in 'libcrypt.so.2.0.0' when using sudo / su
Summary: sys-libs/libxcrypt-4.4.25 built using the gold linker on ppc64le generates 'i...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: PPC64 Linux
: Normal minor (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: systemwide-gold libxcrypt-migration
  Show dependency tree
 
Reported: 2021-11-03 16:33 UTC by zin0
Modified: 2022-05-15 05:45 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
output of emerge --info (emerge-info.txt,13.20 KB, text/plain)
2021-11-03 16:36 UTC, zin0
Details
strace output for sudo (strace-sudo-ppc64le.log,11.55 KB, text/plain)
2021-11-03 20:16 UTC, zin0
Details
strace output for su (strace-su-ppc64le.log,25.31 KB, text/plain)
2021-11-03 20:17 UTC, zin0
Details
build log for libxcrypt (build-libxcrytp-ppc64le.log,147.99 KB, text/plain)
2021-11-03 20:18 UTC, zin0
Details
kernel error messages (kernel-ppc64le.log,23.04 KB, text/plain)
2021-11-03 20:20 UTC, zin0
Details
libcrypt.so.2.0.0 bad version (gold linker) (libcrypt.so.2.0.0.bad,258.31 KB, application/octet-stream)
2021-11-21 15:53 UTC, zin0
Details
libcrypt.so.2.0.0 good version (bfd linker) (libcrypt.so.2.0.0.good,257.72 KB, application/octet-stream)
2021-11-21 15:54 UTC, zin0
Details

Note You need to log in before you can comment on or make changes to this bug.
Description zin0 2021-11-03 16:33:39 UTC
When sys-libs/libxcrypt-4.4.25 is built using the gold linker on a ppc64le system, sudo cannot be used anymore (fails with the message 'illegal instruction').
Furthermore, local logins (root or normal user) arent't possible either.

Chrooting into the system and rebuilding libxcrypt using the bdf linker makes the problem go away. 

Reproducible: Always

Steps to Reproduce:
1.build sys-libs/libxcrypt-4.4.25 using the gold linker
2.try to use sudo
3.try to login locally



At least in my case the problem seems to be ppc64le (don't know about big endian yet) specific. I didn't have the same issue on amd64 and aarch64.
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 16:34:48 UTC
Please can you run it under gdb or strace to find which instruction gets it killed? Thanks.
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 16:35:16 UTC
(In reply to Sam James from comment #1)
> Please can you run it under gdb or strace to find which instruction gets it
> killed? Thanks.

The version of sudo and emerge --info would be very helpful too.

Grabbing the build log from libxcrypt if you can may be helpful.
Comment 3 zin0 2021-11-03 16:36:51 UTC
Created attachment 748284 [details]
output of emerge --info
Comment 4 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 16:47:31 UTC
(In reply to zin0 from comment #3)
> Created attachment 748284 [details]
> output of emerge --info

Could you share the sudo version, possibly the libxcrypt build log (not critical but it'd be useful to have), and also tell us if it happens with lld (from LLVM)?
Comment 5 zin0 2021-11-03 17:02:40 UTC
Thanks for responding so quickly, Sam. The sudo version is 1.9.8_p2.
As for the rest of the requested info, in need some additional time. 
(I'll try to reproduce the problem in a kvm guest to minimize potential damage to my host.)
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 17:07:38 UTC
(In reply to zin0 from comment #5)
> Thanks for responding so quickly, Sam. The sudo version is 1.9.8_p2.
> As for the rest of the requested info, in need some additional time. 
> (I'll try to reproduce the problem in a kvm guest to minimize potential
> damage to my host.)

That's no problem at all, thank you for reporting it. I'm working on upgrading my ppc64le chroot to test this too.
Comment 7 zin0 2021-11-03 20:16:21 UTC
Created attachment 748317 [details]
strace output for sudo
Comment 8 zin0 2021-11-03 20:17:08 UTC
Created attachment 748320 [details]
strace output for su
Comment 9 zin0 2021-11-03 20:18:24 UTC
Created attachment 748323 [details]
build log for libxcrypt
Comment 10 zin0 2021-11-03 20:20:47 UTC
Created attachment 748326 [details]
kernel error messages
Comment 11 zin0 2021-11-03 20:50:15 UTC
It seems that big endian ppc64 isn't affected by this.
I just did the libxcrypt migration and sudo / su still work as intended.

However, the problem might not be exclusively 64bit related. I have an additional system with the 'default/linux/powerpc/ppc64/17.0/32bit-userland' profile and the problem appears there, too.
Comment 12 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 20:51:45 UTC
(In reply to zin0 from comment #11)
> It seems that big endian ppc64 isn't affected by this.
> I just did the libxcrypt migration and sudo / su still work as intended.
> 
> However, the problem might not be exclusively 64bit related. I have an
> additional system with the 'default/linux/powerpc/ppc64/17.0/32bit-userland'
> profile and the problem appears there, too.

That's.. weird. I've tried a ppc64le system and I can't reproduce it there.
Comment 13 Georgy Yakovlev archtester gentoo-dev 2021-11-04 03:59:10 UTC
can't repro on my power9 LE system

straces lack those things, I think, not a single SIGILL


I've noticed you've built libxcrypt with -mpcu=power8, is it intentional?
rest of the system seems to be configured differently with -mcpu=native -mtune=native, it's generally not a wise thing to do.

also, as a test point, can you maybe install `sys-kernel/gentoo-kernel` WITHOUT savedconfig useflag (it provides a working config and will likely work on your machine, tested on blackbird and talos) and give it a go with a known-good kernel configuration?


also there's a TON of ppc64 bugfixes in gold landed recently: https://sourceware.org/git/?p=binutils-gdb.git;a=history;f=gold;hb=HEAD

maybe look at testing some.

and last: my humble advice is not to use gold, it's under-maintained.
some brief info and links can be found here: https://en.wikipedia.org/wiki/Gold_(linker)
Comment 14 zin0 2021-11-04 17:32:30 UTC
(In reply to Georgy Yakovlev from comment #13)
> can't repro on my power9 LE system
> 
> straces lack those things, I think, not a single SIGILL
> 
> 
> I've noticed you've built libxcrypt with -mpcu=power8, is it intentional?
> rest of the system seems to be configured differently with -mcpu=native
> -mtune=native, it's generally not a wise thing to do.
> 
> also, as a test point, can you maybe install `sys-kernel/gentoo-kernel`
> WITHOUT savedconfig useflag (it provides a working config and will likely
> work on your machine, tested on blackbird and talos) and give it a go with a
> known-good kernel configuration?
> 
> 
> also there's a TON of ppc64 bugfixes in gold landed recently:
> https://sourceware.org/git/?p=binutils-gdb.git;a=history;f=gold;hb=HEAD
> 
> maybe look at testing some.
> 
> and last: my humble advice is not to use gold, it's under-maintained.
> some brief info and links can be found here:
> https://en.wikipedia.org/wiki/Gold_(linker)

Not using the gold linker - at least system wide - is definitely sound advice. (I have been burnt by this linker before, so I guess it's time to reconsider USE='default-gold'.)

As for -mcpu=power8 vs -mcpu=native, those stem from two different systems. I first encountered the problem on my bare metal installation on a RCS Blackbird (-mcpu=native), filed the bug report and then did further testing / debugging on a KVM ppc64le guest (-mcpu=power8) in order to not mess with my bare metal installation too much. The issue, however, remained the same. Sorry for not mentioning this explicitly.

I will to some further testing regarding this issue out of sheer curiosity, but since neither you, Georgy, nor Sam can reproduce the bug, I don't think there's any need for you guys to spend any more time on this.

Thank you both for your help.
Comment 15 zin0 2021-11-19 17:33:22 UTC
After some more testing / experimenting, it seems that the bug described above
can indeed be narrowed down to sys-libs/libxcrypt being built using the gold
linker.

It doesn't seem to matter whether gold or bfd have been used to build the rest
of @world.

I'm however unable to further determine how exactly the gold linker messes up
libxcrypt or more specifically /lib64/libcrypt.so.2.0.0 .

Interestingly enough, the same problem also occurs when using app-admin/doas
instead of app-admin/sudo. So it doesn't seem to be a sudo specific issue.

On the positive side, building sys-libs/libxcrypt using sys-devel/clang with
sys-devel/lld as the linker to build sys-libs/libxcrypt works without
problems (compiles and works as expected).

In conclusion, the fix for this bug is not to use the gold linker.

Furthermore, the less active development of the gold linker is refernced in the Gentoo Wiki. (See https://wiki.gentoo.org/wiki/Gold)

P.S.: As suggested, the testing was done using sys-kernel/gentoo-kernel
without the savedconfig useflag.
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-19 17:37:36 UTC
(In reply to zin0 from comment #15)

Thanks for investigating this further. Given how gold (as you note) is heading to the grave, I'm inclined to just force it off in lieu of more information, given I really don't think people should use it for critical utilities anyway.

It'd be interesting to compare a broken binary and a good one if you could upload them though.
Comment 17 Larry the Git Cow gentoo-dev 2021-11-19 17:42:04 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f9db221ddfadd7518e96fb63d93d1edabc5b97eb

commit f9db221ddfadd7518e96fb63d93d1edabc5b97eb
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-11-19 17:39:30 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-11-19 17:41:56 +0000

    sys-libs/libxcrypt: disable gold
    
    Disable gold as a precaution given report of odd behaviour (possible
    illegal instructions being emitted) with gold.
    
    It's not quite clear how this could happen but gold is already
    on its way out and I'd rather play it safe here pending
    more information.
    
    Bug: https://bugs.gentoo.org/821496
    Signed-off-by: Sam James <sam@gentoo.org>

 .../{libxcrypt-4.4.25.ebuild => libxcrypt-4.4.25-r1.ebuild}         | 6 +++++-
 .../{libxcrypt-4.4.26.ebuild => libxcrypt-4.4.26-r1.ebuild}         | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
Comment 18 zin0 2021-11-21 15:53:34 UTC
Created attachment 754354 [details]
libcrypt.so.2.0.0 bad version (gold linker)
Comment 19 zin0 2021-11-21 15:54:32 UTC
Created attachment 754358 [details]
libcrypt.so.2.0.0 good version (bfd linker)
Comment 20 zin0 2021-11-21 16:09:10 UTC
A quick note on reproducing this bug: The type of password hash used for a user account in '/etc/shadow' seems to matter.

sha512 password hash -> sudo generates illegal instruction
md5 password hash -> sudo works as expected

As before, '/usr/bin/doas' and '/bin/login' show the same behaviour.