TL;DR: After upgrading systemd to >sys-apps/systemd-249.4-r4, commands like 'ls -al' (not ls!), 'sudo su', and others die with SIGBUS: ``` $ sudo su Bus error (core dumped) ``` Known good versions: - sys-apps/systemd-249.4-r4 (but not recompiled it yet) Known bad versions: - sys-apps/systemd-249.6-r1 - sys-apps/systemd-249.7 - sys-apps/systemd-250-r1 The test suite for 250-r1 passes other than normal looking cgroup/dbus related failures which we tend to get in Portage. If I change out /etc/nsswitch.conf to remove any reference to systemd, e.g. 250-r1 is fine and 'ls -al' and such works, which makes sense for the "NSS lookups are somehow broken" hypothesis.
# emerge --info Portage 3.0.28 (python 3.9.9-final-0, default/linux/sparc/17.0/64ul, gcc-11.2.0, glibc-2.33-r1, 5.15.5-gentoo sparc64) ================================================================= System uname: Linux-5.15.5-gentoo-sparc64-sun4v-with-glibc2.33 KiB Mem: 531346648 total, 528228688 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Thu, 30 Dec 2021 07:06:48 +0000 Head commit of repository gentoo: 167d78bef25a526aea29168537543b98f2e52c31 sh bash 5.1_p8 ld GNU ld (Gentoo 2.37_p1 p0) 2.37 app-misc/pax-utils: 1.3.3::gentoo app-shells/bash: 5.1_p8::gentoo dev-lang/perl: 5.34.0-r3::gentoo dev-lang/python: 2.7.18_p13::gentoo, 3.9.9::gentoo, 3.10.0_p1::gentoo dev-util/cmake: 3.20.5::gentoo dev-util/meson: 0.59.4::gentoo sys-apps/baselayout: 2.7-r3::gentoo sys-apps/sandbox: 2.25::gentoo sys-apps/systemd: 250-r1::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.71-r1::gentoo sys-devel/automake: 1.16.4::gentoo sys-devel/binutils: 2.37_p1::gentoo sys-devel/binutils-config: 5.4::gentoo sys-devel/gcc: 11.2.0::gentoo sys-devel/gcc-config: 2.4::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.10-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.33-r1::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: https://anongit.gentoo.org/git/repo/sync/gentoo.git priority: -1000 sync-git-verify-commit-signature: true sync-git-clone-extra-opts: -b master ACCEPT_KEYWORDS="sparc" ACCEPT_LICENSE="@FREE" CBUILD="sparc64-unknown-linux-gnu" CFLAGS="-O2 -mcpu=niagara4 -pipe" CHOST="sparc64-unknown-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -mcpu=niagara4 -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -mcpu=niagara4 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -mcpu=niagara4 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j256 -l256" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="acl big-endian bzip2 caps cli crypt dri fortran gdbm iconv ipv6 libglvnd libtirpc llvm-libunwind lz4 ncurses nls nptl openmp pam pcre readline sparc split-usr sqlite ssl systemd udev unicode xattr zlib" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" VIDEO_CARDS="fbdev glint mga r128 radeon dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Not very useful gdb: ``` (gdb) file ls Reading symbols from ls... (No debugging symbols found in ls) (gdb) run -al Starting program: /bin/ls -al Program received signal SIGBUS, Bus error. 0xfff8000100a590c8 in ?? () ```
OK, it is NSS related. Stack trace with debugging symbols but not great yet: ``` Stack trace of thread 595290: #0 0xfff8000100b3d0c8 copy_synthesized_group (libnss_systemd.so.2 + 0x90c8) #1 0xfff8000100b3e730 _nss_systemd_getgrgid_r (libnss_systemd.so.2 + 0xa730) #2 0xfff80001003dd108 getgrgid_r (libc.so.6 + 0xc1108) #3 0xfff80001003dc63c getgrgid (libc.so.6 + 0xc063c) #4 0x00000100000163d8 n/a (ls + 0x163d8) ELF object binary architecture: SPARC v9 ``` Before debugging symbols, supposedly: ``` #0 0xfff8000100b490c8 n/a (libnss_systemd.so.2 + 0x90c8) ``` but this may well just be the stack getting corrupted instead.
(In reply to Sam James from comment #3) > OK, it is NSS related. Stack trace with debugging symbols but not great yet: Bit more verbose but not sure it helps much: ``` Stack trace of thread 723755: #0 0xfff80001005b50c8 copy_synthesized_group (libnss_systemd.so.2 + 0x90c8) #1 0xfff80001005b6730 _nss_systemd_getgrgid_r (libnss_systemd.so.2 + 0xa730) #2 0xfff80001002f1f54 __getgrgid_r (libc.so.6 + 0xb9f54) #3 0xfff80001002f150c getgrgid (libc.so.6 + 0xb950c) #4 0x0000010000012d3c getgroup (ls + 0x12d3c) #5 0x000001000000dc84 format_group_width (ls + 0xdc84) #6 0x000001000000e484 print_dir (ls + 0xe484) #7 0x00000100000046e4 main (ls + 0x46e4) #8 0xfff800010025bd08 __libc_start_main (libc.so.6 + 0x23d08) #9 0x000001000000604c _start (ls + 0x604c) ```
journalctl/coredumpctl wasn't showing line numbers, but whatever, gdb does: ``` (gdb) bt #0 copy_synthesized_group (dest=0xfff800010049bf90 <resbuf>, src=0xfff8000100706b08 <root_group>, buffer=0x100001382a0 "root", buflen=1024, errnop=0xfff800010002fbd8) at ../systemd-250/src/nss-systemd/nss-systemd.c:254 #1 0xfff80001005b6730 in _nss_systemd_getgrgid_r (gid=<optimized out>, gr=0xfff800010049bf90 <resbuf>, buffer=0x100001382a0 "root", buflen=1024, errnop=0xfff800010002fbd8) at ../systemd-250/src/nss-systemd/nss-systemd.c:498 #2 0xfff80001002f1f54 in __getgrgid_r (gid=<optimized out>, resbuf=0xfff800010049bf90 <resbuf>, buffer=0x100001382a0 "root", buflen=1024, result=0x7fefff207e0) at ../nss/getXXbyYY_r.c:274 #3 0xfff80001002f150c in getgrgid (gid=<optimized out>) at ../nss/getXXbyYY.c:135 #4 0x0000010000012d3c in getgroup (gid=<optimized out>) at lib/idcache.c:167 #5 0x000001000000dc84 in format_group_width (g=0) at src/ls.c:4183 #6 gobble_file (name=0x1000012f623 "glibc-2.30-r9.ebuild", type=<optimized out>, command_line_arg=<optimized out>, dirname=<optimized out>, inode=0) at src/ls.c:3534 #7 0x000001000000e484 in print_dir (name=0x100001276e0 ".", realname=<optimized out>, command_line_arg=<optimized out>) at src/ls.c:2989 #8 0x00000100000046e4 in main (argc=<optimized out>, argv=<optimized out>) at src/ls.c:1778 (gdb) ```
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6ca40167e29dc86229788294508ba28472a9598d commit 6ca40167e29dc86229788294508ba28472a9598d Author: Sam James <sam@gentoo.org> AuthorDate: 2022-01-13 00:14:04 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-01-13 00:18:42 +0000 sys-apps/systemd: add 249.9 (Note that 250.1 also contains the SPARC/alignment fixes for NSS.) Bug: https://bugs.gentoo.org/830275 Bug: https://bugs.gentoo.org/830967 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/systemd/Manifest | 1 + sys-apps/systemd/systemd-249.9.ebuild | 505 ++++++++++++++++++++++++++++++++++ 2 files changed, 506 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f67566dcf077a2d67a3de2b377d938a3362c3368 commit f67566dcf077a2d67a3de2b377d938a3362c3368 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-01-13 04:44:01 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-01-13 04:44:01 +0000 profiles/arch/sparc: mask broken systemd versions on sparc Some 250.x versions were affected too but too awkward to try specify this in the mask. Latest version is fine. Closes: https://bugs.gentoo.org/830275 Signed-off-by: Sam James <sam@gentoo.org> profiles/arch/sparc/package.mask | 6 ++++++ 1 file changed, 6 insertions(+)
