| Summary: | sys-libs/glibc-2.30-r8: arm64 test failures (nptl/{tst-rwlock18,tst-rwlock9,tst-stack4}) | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Sam James <sam> |
| Component: | Current packages | Assignee: | Gentoo Toolchain Maintainers <toolchain> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | arm64 |
| Priority: | Normal | Keywords: | TESTFAILURE |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
Build log is 60MB. Will upload somewhere shortly...
----
XPASS: elf/tst-latepthread
UNSUPPORTED: elf/tst-ldconfig-bad-aux-cache
UNSUPPORTED: elf/tst-pldd
XPASS: elf/tst-protected1a
XPASS: elf/tst-protected1b
UNSUPPORTED: iconv/tst-gconv-init-failure
UNSUPPORTED: io/tst-getcwd-abspath
FAIL: io/tst-open-tmpfile
XPASS: locale/tst-locale-locpath
UNSUPPORTED: math/test-fesetexcept-traps
UNSUPPORTED: math/test-fexcept-traps
UNSUPPORTED: math/test-nearbyint-except-2
FAIL: misc/tst-pkey
UNSUPPORTED: nptl/test-cond-printers
UNSUPPORTED: nptl/test-condattr-printers
UNSUPPORTED: nptl/test-mutex-printers
UNSUPPORTED: nptl/test-mutexattr-printers
UNSUPPORTED: nptl/test-rwlock-printers
UNSUPPORTED: nptl/test-rwlockattr-printers
FAIL: nptl/tst-rwlock18
FAIL: nptl/tst-rwlock9
UNSUPPORTED: nptl/tst-sem16
UNSUPPORTED: nss/tst-nss-db-endgrent
UNSUPPORTED: nss/tst-nss-db-endpwent
UNSUPPORTED: nss/tst-nss-files-alias-leak
UNSUPPORTED: nss/tst-nss-files-alias-truncated
UNSUPPORTED: nss/tst-nss-files-hosts-erange
UNSUPPORTED: nss/tst-nss-files-hosts-getent
UNSUPPORTED: nss/tst-nss-files-hosts-long
UNSUPPORTED: nss/tst-nss-test3
UNSUPPORTED: posix/tst-spawn4-compat
UNSUPPORTED: posix/tst-sysconf-empty-chroot
UNSUPPORTED: resolv/tst-resolv-threads
FAIL: rt/tst-shm
UNSUPPORTED: rt/tst-shm-cancel
UNSUPPORTED: sunrpc/tst-svc_register
Summary of test results:
5 FAIL
5945 PASS
27 UNSUPPORTED
18 XFAIL
4 XPASS
make[1]: *** [Makefile:413: tests] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-libs/glibc-2.30-r8/work/glibc-2.30'
make: *** [Makefile:9: check] Error 2
^[[31;01m*^[[0m ERROR: sys-libs/glibc-2.30-r8::gentoo failed (test phase):
^[[31;01m*^[[0m emake failed
Failures from 2.29-r8:
UNSUPPORTED: elf/tst-pldd
XPASS: elf/tst-protected1a
XPASS: elf/tst-protected1b
UNSUPPORTED: iconv/tst-gconv-init-failure
UNSUPPORTED: io/tst-getcwd-abspath
FAIL: io/tst-open-tmpfile
UNSUPPORTED: math/test-fesetexcept-traps
UNSUPPORTED: math/test-fexcept-traps
UNSUPPORTED: math/test-nearbyint-except-2
FAIL: misc/tst-pkey
UNSUPPORTED: nptl/test-cond-printers
UNSUPPORTED: nptl/test-condattr-printers
UNSUPPORTED: nptl/test-mutex-printers
UNSUPPORTED: nptl/test-mutexattr-printers
UNSUPPORTED: nptl/test-rwlock-printers
UNSUPPORTED: nptl/test-rwlockattr-printers
UNSUPPORTED: nptl/tst-sem16
FAIL: nptl/tst-stack4
UNSUPPORTED: nss/tst-nss-db-endgrent
UNSUPPORTED: nss/tst-nss-db-endpwent
UNSUPPORTED: nss/tst-nss-files-alias-leak
UNSUPPORTED: nss/tst-nss-files-hosts-erange
UNSUPPORTED: nss/tst-nss-files-hosts-getent
UNSUPPORTED: nss/tst-nss-test3
UNSUPPORTED: posix/tst-spawn4-compat
UNSUPPORTED: posix/tst-sysconf-empty-chroot
UNSUPPORTED: resolv/tst-resolv-threads
FAIL: rt/tst-shm
UNSUPPORTED: rt/tst-shm-cancel
UNSUPPORTED: sunrpc/tst-svc_register
Summary of test results:
4 FAIL
5809 PASS
24 UNSUPPORTED
17 XFAIL
2 XPASS
----
So, new failures?:
FAIL: nptl/tst-rwlock18
FAIL: nptl/tst-rwlock9
Can you attach *.out files? They usually contain actual tests' stderr that explais the failure. Okay, here's an updated one:
FAIL: misc/tst-pkey
UNSUPPORTED: nptl/test-cond-printers
UNSUPPORTED: nptl/test-condattr-printers
UNSUPPORTED: nptl/test-mutex-printers
UNSUPPORTED: nptl/test-mutexattr-printers
UNSUPPORTED: nptl/test-rwlock-printers
UNSUPPORTED: nptl/test-rwlockattr-printers
FAIL: nptl/tst-rwlock18
FAIL: nptl/tst-rwlock9
FAIL: nptl/tst-stack4
UNSUPPORTED: nss/tst-nss-db-endgrent
UNSUPPORTED: nss/tst-nss-db-endpwent
UNSUPPORTED: nss/tst-nss-files-alias-leak
UNSUPPORTED: nss/tst-nss-files-alias-truncated
UNSUPPORTED: nss/tst-nss-files-hosts-erange
UNSUPPORTED: nss/tst-nss-files-hosts-getent
UNSUPPORTED: nss/tst-nss-files-hosts-long
UNSUPPORTED: nss/tst-nss-test3
UNSUPPORTED: posix/tst-spawn4-compat
UNSUPPORTED: posix/tst-sysconf-empty-chroot
UNSUPPORTED: resolv/tst-resolv-threads
UNSUPPORTED: sunrpc/tst-svc_register
Summary of test results:
4 FAIL
5948 PASS
25 UNSUPPORTED
18 XFAIL
4 XPASS
----
Note that only the nptl ones matter here. The rest (inc. tst-pkey) are known to fail by upstream.
Attaching each of the 3 logs now.
(In reply to Sam James (sec padawan) from comment #5) > Attaching each of the 3 logs now. https://cmpct.info/~sam/gentoo/tst-rwlock9.out https://cmpct.info/~sam/gentoo/tst-rwlock18.out https://cmpct.info/~sam/gentoo/tst-stack4.out (In reply to Sam James (sec padawan) from comment #6) > (In reply to Sam James (sec padawan) from comment #5) > > Attaching each of the 3 logs now. > > https://cmpct.info/~sam/gentoo/tst-rwlock9.out > https://cmpct.info/~sam/gentoo/tst-rwlock18.out > https://cmpct.info/~sam/gentoo/tst-stack4.out These don't seem to provide much more detail. You can pisk a command from build.log that ran the test and explore why each test fails. Looking at tst-rwlock9 test now: https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/tst-rwlock9.c;h=408bbcdd5d47f93337c87cdb0be6e3cb20115efb;hb=HEAD It's not a new test. tst-rwlock9 is a simple timed rwlock test: - it spawns 15 writer threads, 15 readers threads - against a single rwlock with priority for writes - each write thread acquires the lock , waits for 1 ms and releases it. repeats 10 times - each read thread acquires the lock , waits for 1 ms and releases it. repeats 15 times - each test is ran for 3 clock types My understanding is that: - best case is when all writers sequentially acquire the lock (15 * 10 * 1 ms = 150ms), then all readers in parallel acquire the lock (15 * 10 * 1 ms = 150ms). All in all 300 ms. 3 x 300ms = 900ms. - worst case: readers and writers are interspersed and effectively serialize one another. Still not too bad: 15 * 10 * 1ms writers + 15 * 15 * 1 ms = 375 ms. 3 x 375ms = 1.125s. This does not account for actual locking overhead, which might be very significant. My math is off somewhere as on amd64 tests finish in 537ms for me: $ time { build-x86-x86_64-pc-linux-gnu-nptl/nptl/tst-rwlock9 > /dev/null; } real 0m0,537s user 0m0,049s sys 0m0,099s But that at least matches order of magnitude for best case. On 96-core arm64 tests timeout in 30 seconds for me. That looks wrong. rwlock should be somewhat fair for mostly sleeping threads. We have 25 contending cores though and they all have the same timedlock timeout. It's also probably not an effect of contention across cores:
$ time { nptl/tst-rwlock9 >o; }
real 0m30.121s
user 6m53.276s
sys 0m5.408s
$ time { taskset -c 0 nptl/tst-rwlock9 >o; }
real 0m30.108s
user 0m20.936s
sys 0m8.932s
That's quite a lot of system time. As if nanosleep was implemented as busy non-preemptible loop.
I also created up-to-date chroot with gcc-9.3.0 and now my tests pass in 0.6 seconds as expected:
"""
# time { env GCONV_PATH=/var/tmp/portage/sy...
real 0m0.633s
user 0m0.100s
sys 0m0.404s
"""
Can you retest against latest stable gcc?
(In reply to Sergei Trofimovich from comment #10) > Can you retest against latest stable gcc? Specificall since gcc-9.2.0 we had a few potential fixes: https://gcc.gnu.org/PR92692 / https://sourceware.org/PR24924 which mentions the same tests. With gcc-9.3.0 I'm down to 1 test failure:
"""
FAIL: nptl/tst-stack4
Summary of test results:
1 FAIL
5949 PASS
25 UNSUPPORTED
21 XFAIL
3 XPASS
"""
Looking at it now:
cat $ nptl/tst-stack4.out
Didn't expect signal from child: got `Aborted'
Test on it's own takes 14 seconds:
real 0m14.315s
user 0m7.176s
sys 0m9.136s
Yet another threaded test: https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/tst-stack4.c;h=3030f75027f837c92f4c833a146d4148119a898f;hb=HEAD
- it has 100s timeout (ok, 10s is a lot)
- test is flaky: If it run in in the loop it passes most of time time, but occasionally fails with various crashes related to seeming memory corruption
# while :; do date; env GCONV_PATH=/var/tmp/portage/sys...; done
Sun 03 May 2020 10:42:08 AM UTC
Sun 03 May 2020 10:42:21 AM UTC
Sun 03 May 2020 10:42:34 AM UTC
Didn't expect signal from child: got `Segmentation fault'
Sun 03 May 2020 10:42:43 AM UTC
Sun 03 May 2020 10:42:56 AM UTC
malloc(): invalid size (unsorted)
Didn't expect signal from child: got `Aborted'
Sun 03 May 2020 10:42:57 AM UTC
Sun 03 May 2020 10:43:12 AM UTC
Sun 03 May 2020 10:43:28 AM UTC
Sun 03 May 2020 10:43:46 AM UTC
Sun 03 May 2020 10:44:01 AM UTC
malloc(): invalid size (unsorted)
Didn't expect signal from child: got `Aborted'
Sun 03 May 2020 10:44:03 AM UTC
Sun 03 May 2020 10:44:18 AM UTC
(In reply to Sergei Trofimovich from comment #12) > FAIL: nptl/tst-stack4 https://sourceware.org/PR19329 looks relevant. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f55f2be1bd6791465453c7b0dfc60bf4c3c75613 commit f55f2be1bd6791465453c7b0dfc60bf4c3c75613 Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2020-05-03 11:01:18 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2020-05-03 11:01:35 +0000 sys-libs/glibc: disable flaky tst-stack4, bug #719674 tst-stack4 exposes known race condition in glibc (https://sourceware.org/PR19329). Let's disable this test until it's fixed upstream. Reported-by: Sam James Bug: https://bugs.gentoo.org/719674 Bug: https://sourceware.org/PR19329 Package-Manager: Portage-2.3.99, Repoman-2.3.22 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> sys-libs/glibc/glibc-2.30-r8.ebuild | 5 +++++ sys-libs/glibc/glibc-2.31-r2.ebuild | 5 +++++ sys-libs/glibc/glibc-9999.ebuild | 5 +++++ 3 files changed, 15 insertions(+) I assume tests don't fail anymore. |
Portage 2.3.89 (python 3.6.10-final-0, default/linux/arm64/17.0, gcc-9.2.0, glibc-2.30-r8, 4.9.0-4-arm64 aarch64) ================================================================= System uname: Linux-4.9.0-4-arm64-aarch64-with-gentoo-2.6 KiB Mem: 131544964 total, 71042340 free KiB Swap: 3321056 total, 2644464 free Timestamp of repository gentoo: Sun, 26 Apr 2020 00:30:01 +0000 Head commit of repository gentoo: 8b5014db8b6b79a2831236bdc1b564247473e50a sh bash 4.4_p23-r1 ld GNU ld (Gentoo 2.33.1 p2) 2.33.1 app-shells/bash: 4.4_p23-r1::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.17-r2::gentoo, 3.6.10-r1::gentoo, 3.7.7-r1::gentoo, 3.8.2-r1::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.69-r4::gentoo sys-devel/automake: 1.16.1-r1::gentoo sys-devel/binutils: 2.33.1-r1::gentoo sys-devel/gcc: 9.2.0-r2::gentoo, 9.3.0::gentoo sys-devel/gcc-config: 2.2.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r8::gentoo Repositories: gentoo location: /usr/gentoo sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-metamanifest: yes sync-rsync-extra-opts: sync-rsync-verify-jobs: 1 sync-rsync-verify-max-age: 24 ACCEPT_KEYWORDS="arm64" ACCEPT_LICENSE="@FREE" CBUILD="aarch64-unknown-linux-gnu" CFLAGS="-O2 -pipe -march=native" CHOST="aarch64-unknown-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=native" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--keep-going" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe -march=native" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news nodoc noinfo noman parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe -march=native" GENTOO_MIRRORS="http://distfiles.gentoo.org" INSTALL_MASK="/usr/share/doc/*/*.pdf /usr/share/man/*/*" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j40" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="acl arm64 berkdb bzip2 cli crypt dri fortran gdbm iconv ipv6 libtirpc ncurses nls nptl openmp pam pcre readline seccomp split-usr ssl tcpd unicode xattr zlib" ADA_TARGET="gnat_2018" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_ARM="edsp thumb vfp vfpv3 vfpv4 vfp-d32 aes sha1 sha2 crc32 v4 v5 v6 v7 v8 thumb2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="fbdev dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS