I'm aware this is a bit of a pain as a case, but until now (< 247), I've been able to run tests on non-systemd hosts. I don't have access to systemd hosts for all architectures unfortunately, which complicates matters. It looks like only a few tests have actually failed here. I don't know if it was supposed to spawn some sort of prefixed daemon for tests or if it wanted to interact with the host for e.g. the systemd-oomd test (seems unlikely given they've been quite good about this until now). In any case, I appreciate it's tempting to WONTFIX this because I bet they pass on a systemd host (this might be a wrong guess, however), but skipping such tests would help a lot with stabilisation. (I also need to work on scripting packages which can't be _installed_ due to blockers but run "clean test install" like I do manually right now.) ---- Portage 3.0.13 (python 3.8.7-final-0, default/linux/sparc/17.0/64ul/desktop, gcc-9.3.0, glibc-2.32-r6, 5.10.12-gentoo sparc) ================================================================= System uname: Linux-5.10.12-gentoo-sparc-sun4v-with-glibc2.2 KiB Mem: 531343128 total, 501658768 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Sun, 21 Feb 2021 05:37:20 +0000 sh bash 5.0_p18 ld GNU ld (Gentoo 2.35.1 p2) 2.35.1 distcc 3.3.3 sparc64-unknown-linux-gnu [disabled] app-shells/bash: 5.0_p18::gentoo dev-lang/perl: 5.30.3::gentoo dev-lang/python: 2.7.18-r6::gentoo, 3.7.9-r2::gentoo, 3.8.7-r1::gentoo, 3.9.1-r1::gentoo dev-util/cmake: 3.17.4-r1::gentoo sys-apps/baselayout: 2.7::gentoo sys-apps/openrc: 0.42.1-r1::gentoo sys-apps/sandbox: 2.20::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r5::gentoo sys-devel/automake: 1.16.2-r1::gentoo sys-devel/binutils: 2.35.1-r1::gentoo sys-devel/gcc: 9.3.0-r2::gentoo sys-devel/gcc-config: 2.3.2-r1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.10::gentoo (virtual/os-headers) sys-libs/glibc: 2.32-r6::gentoo Repositories: gentoo location: /bound/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-extra-opts: sync-rsync-verify-metamanifest: yes sync-rsync-verify-max-age: 24 sync-rsync-verify-jobs: 1 ACCEPT_KEYWORDS="sparc" ACCEPT_LICENSE="*" CBUILD="sparc64-unknown-linux-gnu" CFLAGS="-O2 -mcpu=ultrasparc -pipe -fdiagnostics-show-option -frecord-gcc-switches" CHOST="sparc64-unknown-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -mcpu=ultrasparc -pipe" DISTDIR="/bound/distfiles" EMERGE_DEFAULT_OPTS="--complete-graph --with-bdeps=y --keep-going --deep" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -mcpu=ultrasparc -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs cgroup config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync mount-sandbox network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -mcpu=ultrasparc -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="C.UTF8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu" MAKEOPTS="-j150 -l257" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl alsa berkdb big-endian branding bzip2 cairo caps cdda cdr cli crypt cups dbus dri dts dvd dvdr elogind emboss encode exif filecaps flac fortran gdbm gif gmp gpm gtk gui iconv icu introspection ipv6 jit jpeg lcms libglvnd libnotify libtirpc mad mng mp3 mp4 mpeg ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit polkit ppds readline sdl sparc spell split-usr ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis wxwidgets x264 xattr xcb xml xv xvid zlib" ADA_TARGET="gnat_2018" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_8" PYTHON_TARGETS="python3_8 python3_7 python3_9" RUBY_TARGETS="ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="fbdev glint mga r128 radeon dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 687807 [details] build.log.xz (sparc)
Created attachment 687810 [details] testlog.txt.xz (sparc)
Created attachment 687813 [details] build.log.xz (sparc) (Originally, dbus wasn't running, so there were extra fixable errors). # grep "FAIL" /var/tmp/portage/sys-apps/systemd-247.2-r4/temp/build.log 309/562 test-oomd-util FAIL 2.83s (killed by signal 6 SIGABRT) 321/562 test-engine FAIL 2.36s (killed by signal 6 SIGABRT) 329/562 test-unit-name FAIL 2.71s (killed by signal 6 SIGABRT) 330/562 test-load-fragment FAIL 2.81s (killed by signal 6 SIGABRT) 430/562 test-cgroup-util FAIL 2.92s (killed by signal 6 SIGABRT) 436/562 test-path-util FAIL 3.01s (killed by signal 6 SIGABRT) 438/562 test-path FAIL 3.01s (killed by signal 6 SIGABRT) 443/562 test-sched-prio FAIL 2.74s (killed by signal 6 SIGABRT) 491/562 test-bus-creds FAIL 0.30s (killed by signal 6 SIGABRT) 497/562 test-login FAIL 0.64s (killed by signal 6 SIGABRT)
Created attachment 687816 [details] testlog.txt.xz (sparc)
Created attachment 687819 [details] build.log.xz (amd64) 9 failures on amd64 vs SPARC's 10: # grep "FAIL" /var/tmp/portage/sys-apps/systemd-247.2-r4/temp/build.log 300/541 test-oomd-util FAIL 0.09s (killed by signal 6 SIGABRT) 311/541 test-engine FAIL 0.09s (killed by signal 6 SIGABRT) 319/541 test-unit-name FAIL 0.03s (killed by signal 6 SIGABRT) 320/541 test-load-fragment FAIL 0.03s (killed by signal 6 SIGABRT) 419/541 test-cgroup-util FAIL 0.02s (killed by signal 6 SIGABRT) 427/541 test-path FAIL 0.02s (killed by signal 6 SIGABRT) 432/541 test-sched-prio FAIL 0.02s (killed by signal 6 SIGABRT) 480/541 test-bus-creds FAIL 0.02s (killed by signal 6 SIGABRT) 486/541 test-login FAIL 0.01s (killed by signal 6 SIGABRT) The test which fails only on SPARC (both with and without dbus, so I guess it's not a fluke): 436/562 test-path-util FAIL 3.01s (killed by signal 6 SIGABRT)
Created attachment 687822 [details] testlog.txt.xz (amd64) Found at /var/tmp/portage/sys-apps/systemd-247.2-r4/work/systemd-stable-247.2-abi_x86_32.x86/meson-logs/testlog.txt. No other testlog.txt exists, so I assume because it failed the 32-bit one, it didn't bother running 64-bit.
> I'm aware this is a bit of a pain as a case, but until now (< 247), I've been able to run tests on non-systemd hosts. I would like to see proof of this: as far as I am aware, the systemd test suite has never worked without systemd running as PID 1.
When I run the tests in a container running sysvinit as PID 1 and OpenRC as the service manager, I get no failures. I wonder if there is something subtle like a missing mount point or kernel sysctl that is causing the problem on your baremetal OpenRC systems.
All of the failing tests have a similar cause related to cgroups that looks something like this: statfs("/sys/fs/cgroup/systemd" failed: No such file or directory Assertion 'r >= 0' failed at src/shared/tests.c:269, function enter_cgroup(). Aborting. Basically, I need to figure out exactly how this line of code gets reached. https://github.com/systemd/systemd/blob/v247/src/basic/cgroup-util.c#L2026 What is rc_cgroup_mode set to in rc.conf? What does /sys/fs/cgroup look like? Please check /proc/self/mounts for any mounts starting with that path.
(In reply to Mike Gilbert from comment #9) > All of the failing tests have a similar cause related to cgroups that looks > something like this: > > statfs("/sys/fs/cgroup/systemd" failed: No such file or directory > Assertion 'r >= 0' failed at src/shared/tests.c:269, function > enter_cgroup(). Aborting. > > Basically, I need to figure out exactly how this line of code gets reached. > > https://github.com/systemd/systemd/blob/v247/src/basic/cgroup-util.c#L2026 > > What is rc_cgroup_mode set to in rc.conf? > > What does /sys/fs/cgroup look like? Please check /proc/self/mounts for any > mounts starting with that path. For this host, catbus, the host is systemd with an OpenRC chroot mounted using gentoo-chrootiez [0]. Nothing has (knowingly) really changed here. # grep "cgroup" /proc/self/mounts tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0 tmpfs /sys/fs/cgroup/portage cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/portage/python3.8/cgroup-release-agent,name=portage 0 0 rc.conf within chroot: >rc_shell=/sbin/sulogin >unicode="YES" >rc_tty_number=12 >rc_sys="prefix" >rc_controller_cgroups="NO" >rc_depend_strict="NO" >rc_need="!net !dev !udev-mount !sysfs !checkfs !fsck !netmount !logger !clock !modules" [0] https://github.com/trofi/gentoo-chrootiez
Try running the following commands in the chroot. This will put a cgroup instance where systemd expects to find it. > mkdir /sys/fs/cgroup/systemd > mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
(In reply to Mike Gilbert from comment #11) > Try running the following commands in the chroot. This will put a cgroup > instance where systemd expects to find it. > > > mkdir /sys/fs/cgroup/systemd > > mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd I've tried poking but I'm not sure how to get past this: # mkdir /sys/fs/cgroup/systemd mkdir: cannot create directory ‘/sys/fs/cgroup/systemd’: No such file or directory
Ok, here's what I think is happening: 1. When you enter the chroot, /sys/fs/cgroup is not mounted. 2. When you run emerge, it mounts /sys/fs/cgroup and sets up its own hierarchy. https://gitweb.gentoo.org/proj/portage.git/tree/lib/_emerge/AbstractEbuildProcess.py?h=portage-3.0.17#n73 3. The systemd code in cg_unified_cached() sees that /sys/fs/cgroup is mounted as a tmpfs, and assumes that it will find /sys/fs/cgroup/unified or /sys/fs/cgroup/systemd mounted as cgroup2 or cgroup, respectively. https://github.com/systemd/systemd/blob/v247/src/basic/cgroup-util.c#L1997 A workaround would be to ensure that /sys/fs/cgroup is already mounted, with fstype equal to either cgroup2 or tmpfs. If /sys/fs/cgroup is mounted as a tmpfs, ensure that /sys/fs/cgroup/unified is mounted as cgroup2, or /sys/fs/cgroup/systemd is mounted as cgroup.
Could you give this patch a try? https://github.com/floppym/systemd/commit/236ce6ce005fecfdc80e0e9fb8a77f698bbc6aa7
(In reply to Mike Gilbert from comment #14) > Could you give this patch a try? > > https://github.com/floppym/systemd/commit/ > 236ce6ce005fecfdc80e0e9fb8a77f698bbc6aa7 Before: Ok: 565 Expected Fail: 0 Fail: 10 Unexpected Pass: 0 Skipped: 17 Timeout: 0 After: Ok: 567 Expected Fail: 0 Fail: 1 Unexpected Pass: 0 Skipped: 24 Timeout: 0 The failure was one we already discussed as acceptable for now: 387/592 test-fs-util FAIL 0.11s (killed by signal 6 SIGABRT) So, thanks -- works!
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e473f70fbcfc239779f91c1649af4f369e0f2b6e commit e473f70fbcfc239779f91c1649af4f369e0f2b6e Author: Mike Gilbert <floppym@gentoo.org> AuthorDate: 2021-03-17 14:18:23 +0000 Commit: Mike Gilbert <floppym@gentoo.org> CommitDate: 2021-03-17 14:18:23 +0000 sys-apps/systemd: fix cgroup-related test failures Closes: https://bugs.gentoo.org/771819 Signed-off-by: Mike Gilbert <floppym@gentoo.org> sys-apps/systemd/files/247-cgroup-test.patch | 35 ++++++++++++++++++++++++++++ sys-apps/systemd/systemd-247.2-r4.ebuild | 1 + 2 files changed, 36 insertions(+)
