https://blogs.gentoo.org/ago/2020/07/04/gentoo-tinderbox/ Issue: mail-filter/opendkim-2.10.3-r18 fails tests. Discovered on: amd64 (internal ref: ci)
Created attachment 663424 [details] build.log build log and emerge --info
Created attachment 663427 [details] 1-test-suite.log 1-test-suite.log
Comment on attachment 663427 [details] 1-test-suite.log According to the log, one (!) out of 156 tests fails. Is that a concern? If so, what would you recommend?
"A man with one watch always knows what time it is. A man with two watches is never sure." If the test exposes a real bug, we can report it upstream and disable the test for now. If the test is buggy and easy to fix, we can fix it in Gentoo and submit the fix upstream. If the test is buggy and hard to fix, we can just disable it in Gentoo after reporting the problem upstream.
Given that our last OpenDKIM submission from April 2019 has still not been commented on, let alone processed, I don't intend to spend any time dealing with upstream. If you feel differently, go right ahead. ;-)
@ago That test kills itself because it's expecting an "OK" result, but got something else: /* libopendkim/tests/t-test153.c */ status = dkim_eom(dkim, &testkey); assert(status == DKIM_STAT_OK); I don't suppose you can reproduce the problem? If you can, it would be helpful if you could have it print that status code rather than immediately committing suicide. The DKIM_STAT_* constants are all defined in libopendkim/dkim.h, and I'm curious which one came back from dkim_eom().
No detailed information about the failed test has been provided for a month. I don't have the time to investigate, so I close this issue for the time being.
I have tried to reproduce this and failed to do so. I have: * Run the ebuild test without USE berkdb * Run the ebuild test with USE berkdb with sys-libs/db (5.3.28-r2) and libmilter. * Tried also with >lib-db/6 (6.0.35-r3) All this on my (hardened) amd64 system. All three times all tests passed. Ago, can you rerun the tests and reopen if it fails again? Can you include the coredump and the compiled test file too in that case?
*** Bug 761697 has been marked as a duplicate of this bug. ***
Reopening again as this seems to be happening see #761697
Created attachment 679850 [details, diff] Patcxh the ebuild to run tests sequentially Hey Mjo! I think I figured this one out, since we run the tests in parallel, Ago's tinderbox is running some tests before t-cleanup. This is how the log looks now: PASS: t-cleanup ../../build-aux/test-driver: line 107: 10770 Aborted (core dumped) "$@" > $log_file 2>&1 FAIL: t-test153 This is how it looks with -j1: *PASS: t-test153 PASS: t-test154 PASS: t-signperf PASS: t-verifyperf *PASS: t-cleanup PASS: t-signperf-sha1 PASS: t-signperf-relaxed-relaxed PASS: t-signperf-simple-simple The solution is fairly simple, for now we can just run the tests (and that sadly also means the building of test programs) sequentially. In the future we could try to get a better fix for this (for example by adding a specific target to build the test suite so this can be parallel or using a stronger dependency list for the tests). The attached patch modifies the ebuild to force -j1 so all tests are run sequentially.
Mjo, I don't have commit access to the portage tree so now it is up to you to review and apply the patch :) Klondike
Good work! There's a related problem with the "setup" test. It's supposed to run first, but when the tests are run in parallel, it's possible for the others to get compiled and begin executing before t-setup completes.
Also a silly security vulnerability in the hard-coded key path, #define KEYFILE "/tmp/testkeys" in libopendkim/tests/t-testdata.h :(
Yes it is a can of worms, although since the tests are run with portage privileges and the key is only used for testing, (if I remember correctly) that shouldn't be SO serious from the security perspective.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6f8585de7ea586a36cbc06530346f855540d25e0 commit 6f8585de7ea586a36cbc06530346f855540d25e0 Author: Michael Orlitzky <mjo@gentoo.org> AuthorDate: 2020-12-28 19:08:38 +0000 Commit: Michael Orlitzky <mjo@gentoo.org> CommitDate: 2020-12-28 19:08:55 +0000 mail-filter/opendkim: new revision to disable parallel tests. The test suite for OpenDKIM currently relies on two "setup" and "cleanup" tests being run at the beginning/end of the test suite, respectively. But when the test suite is run in parallel, that's not always guaranteed. This new revision runs the test suite sequentially to avoid the problem, which has been reported upstream. The credit for this goes to klondike, who was able to determine the root cause of the heisenbug. In the process, a minor security vulnerability (insecure /tmp path) was also reported upstream and fixed in the ebuild. Closes: https://bugs.gentoo.org/745969 Package-Manager: Portage-3.0.9, Repoman-3.0.2 Signed-off-by: Michael Orlitzky <mjo@gentoo.org> ...ndkim-2.10.3-r25.ebuild => opendkim-2.10.3-r26.ebuild} | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
