Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 711390 - dev-libs/check-0.14.0 test failure
Summary: dev-libs/check-0.14.0 test failure
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Joonas Niilola
URL:
Whiteboard:
Keywords: TESTFAILURE
Depends on:
Blocks: 721600
  Show dependency tree
 
Reported: 2020-03-02 23:05 UTC by Alexey
Modified: 2020-05-13 13:23 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info (file_711390.txt,19.18 KB, text/plain)
2020-03-02 23:05 UTC, Alexey
Details
test-suite.log (test-suite.log,344.44 KB, text/x-log)
2020-03-02 23:05 UTC, Alexey
Details
build.log (build.log,77.71 KB, text/x-log)
2020-03-02 23:06 UTC, Alexey
Details
[PATCH 1/3] dev-libs/check-0.14.0-r1: Do not use deprecated testing option (file_711390.txt,575 bytes, patch)
2020-03-22 09:15 UTC, gentoo_usr
Details | Diff
[PATCH 2/3] dev-libs/check-0.14.0-r1: Fix warning in patched CMakeLists.txt (file_711390.txt,805 bytes, patch)
2020-03-22 09:16 UTC, gentoo_usr
Details | Diff
[PATCH 3/3] dev-libs/check-0.14.0-r1: Streamline subunit-check (file_711390.txt,1.38 KB, patch)
2020-03-22 09:16 UTC, gentoo_usr
Details | Diff
528233887.log (528233887.log,43.83 KB, text/plain)
2020-05-13 11:53 UTC, Agostino Sarubbo
Details
LastTest.log (LastTest.log,342.63 KB, text/plain)
2020-05-13 12:33 UTC, Agostino Sarubbo
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey 2020-03-02 23:05:17 UTC
Created attachment 616918 [details]
emerge --info

PASS: test_check_nofork_teardown.sh
PASS: test_check_nofork.sh
PASS: test_xml_output.sh
PASS: test_log_output.sh
PASS: test_set_max_msg_size.sh
PASS: test_tap_output.sh
PASS: test_output.sh
XPASS: check_check_export
XPASS: check_check
Comment 1 Alexey 2020-03-02 23:05:43 UTC
Created attachment 616920 [details]
test-suite.log
Comment 2 Alexey 2020-03-02 23:06:06 UTC
Created attachment 616922 [details]
build.log
Comment 3 Rolf Eike Beer 2020-03-04 17:10:29 UTC
Same on sparc.
Comment 4 gentoo_usr 2020-03-20 14:05:14 UTC
The reason for the emerge failure is that the test suite (tests/Makefile.am) is patched by the Gentoo maintainer to mark the tests `check_check` and `check_check_export` as XFAIL. This means, if the test succeed, they are counted as failures.

Maybe this was a mistake and the Gentoo maintainer meant to mark the test as "test can fail and if so ignore the failure, if it succeeds thats ok, too". However, the automake manual [0] states that the success state is inverted, i.e. if `check_check` now succeeds the test suite fails. And that is what happens here.

There is an old upstream issue [1], which mentions that the two tests check_check and check_check_export are quite resource heavy and thus are thrown to fail due to timeouts. However, there are no indicator that on today's "normal" hardware (whatever that means) this is an issue.

I suggest to remove the patch and not mark the tests as XFAIL as this is the upstreams default. Instead I would issue a warning that these test can fail due to timeout and heavy load. Alternatively, if they are a problem for many machines, the test could be marked as skipped or removed completely.

Workaround:

You can workaround this by *not* applying the patch check-0.14-xfail-tests.patch in the ebuild (e.g. in a local overlay).

[0] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html
[1] https://github.com/libcheck/check/issues/157
Comment 5 Joonas Niilola gentoo-dev 2020-03-21 08:31:21 UTC
(In reply to gentoo_usr from comment #4)
> The reason for the emerge failure is that the test suite (tests/Makefile.am)
> is patched by the Gentoo maintainer to mark the tests `check_check` and
> `check_check_export` as XFAIL. This means, if the test succeed, they are
> counted as failures.
> 
> Maybe this was a mistake and the Gentoo maintainer meant to mark the test as
> "test can fail and if so ignore the failure, if it succeeds thats ok, too".
> However, the automake manual [0] states that the success state is inverted,
> i.e. if `check_check` now succeeds the test suite fails. And that is what
> happens here.

That's it, however the tests apparently then always failed for me, because I can't emerge this without the patch. And with the patch, it always "succeeds". Thus I haven't been able to reproduce this error.


> 
> Workaround:
> 

First I ended up just removing them with a patch, 

--- a/Makefile.am	2020-03-21 08:30:03.571251087 +0200
+++ b/Makefile.am	2020-03-21 08:30:53.400877511 +0200
@@ -1,8 +1,6 @@
 ## Process this file with automake to produce Makefile.in
 
 TESTS = \
-	check_check_export	\
-	check_check		\
 	test_output.sh		\
 	test_check_nofork.sh    \
 	test_check_nofork_teardown.sh    \


But now I rewrote the whole ebuild using CMake, since upstream has been moving towards building with CMake anyway. Tests pass for me when using CMake so I'm gonna push it and see what happens.
Comment 6 Larry the Git Cow gentoo-dev 2020-03-21 08:53:44 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6c5026b88f869f521d85d6a130982c2e92f63e75

commit 6c5026b88f869f521d85d6a130982c2e92f63e75
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2020-03-21 08:52:58 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2020-03-21 08:53:30 +0000

    dev-libs/check: migrate to CMake, "fix" tests, revbump 0.14.0
    
    Bug: https://bugs.gentoo.org/711390
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 dev-libs/check/check-0.14.0-r1.ebuild              | 50 ++++++++++++++++++++++
 .../check-0.14.0-r1-disable-automagic-dep.patch    | 23 ++++++++++
 2 files changed, 73 insertions(+)
Comment 7 gentoo_usr 2020-03-22 09:14:42 UTC
Thanks Joonas for acting so quickly and for your efforts converting the ebuild to use CMake. I've tested the new ebuild and it works fine on my platform (and all tests pass).

I have two small remarks:

* According to the option description of CHECK_ENABLE_TESTS (in CMakeLists.txt) that option is deprecated and BUILD_TESTING should be used instead.

* The patch introducing the conditional dependency on subunit has non-matching if-then-else expressions. (The expression in parenthesis must match exactly the one used in the if-part). This seems strange for the else part as the if-condition is repeated, but that's how it is. You can can make that slightly less unpleasing  by omitting the 'EQUALS 1' in the expression.

I've attached both suggestions as a diff/patch series.
Comment 8 gentoo_usr 2020-03-22 09:15:37 UTC
Created attachment 624150 [details, diff]
[PATCH 1/3] dev-libs/check-0.14.0-r1: Do not use deprecated testing option
Comment 9 gentoo_usr 2020-03-22 09:16:25 UTC
Created attachment 624152 [details, diff]
[PATCH 2/3] dev-libs/check-0.14.0-r1: Fix warning in patched CMakeLists.txt
Comment 10 gentoo_usr 2020-03-22 09:16:59 UTC
Created attachment 624154 [details, diff]
[PATCH 3/3] dev-libs/check-0.14.0-r1: Streamline subunit-check
Comment 11 Joonas Niilola gentoo-dev 2020-03-23 13:09:14 UTC
(In reply to gentoo_usr from comment #7)
> 
> * According to the option description of CHECK_ENABLE_TESTS (in
> CMakeLists.txt) that option is deprecated and BUILD_TESTING should be used
> instead.

Yeaa I saw it but got lazy to change after first one worked. CHECK_ENABLE_TESTS gets removed in 0.15.0 by upstream so it needed to be updated soon anyway.

Thanks for the patches, I applied each one. Couldn't author you but I have feeling you don't mind... :)
Comment 12 Larry the Git Cow gentoo-dev 2020-03-23 13:14:40 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=06d2b112fde3faca9767a93835c07009ada5b177

commit 06d2b112fde3faca9767a93835c07009ada5b177
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2020-03-23 13:12:07 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2020-03-23 13:13:50 +0000

    dev-libs/check: update subunit logic, fix pkgconfig file
    
     - subunit & doc handling patches from #711390
     - .pc libdir issue when using multilib #714058
    
    Bug: https://bugs.gentoo.org/711390
    Closes: https://bugs.gentoo.org/714058
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 dev-libs/check/check-0.14.0-r2.ebuild              | 58 ++++++++++++++++++++++
 .../check-0.14.0-r2-disable-automagic-dep.patch    | 23 +++++++++
 2 files changed, 81 insertions(+)
Comment 13 Agostino Sarubbo gentoo-dev 2020-05-13 11:53:32 UTC
Created attachment 638582 [details]
528233887.log

build log on ppc64
Comment 14 Agostino Sarubbo gentoo-dev 2020-05-13 12:33:16 UTC
Created attachment 638588 [details]
LastTest.log

LastTest.log on ppc64