Summary: | dev-lang/perl-5.10.1 doesn't halt on test failures. | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Kent Fredric (IRC: kent\n) (RETIRED) <kentnl> |
Component: | [OLD] Development | Assignee: | Gentoo Perl team <perl> |
Status: | RESOLVED LATER | ||
Severity: | normal | CC: | mephinet |
Priority: | High | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 280724 |
My analysis has concluded something weird is occuring with the TAP::Harness based test that are run instead of Test::Harness, when TEST_JOBS is defined. Essentially, this results in perl returning a success exit state regardless of what happens, and I can't make out why. I broke my ebuild intentionally so that tests fail like its the apocalypse, and yet, with TEST_JOBS defined, it *STILL* returns a truth value, and would normally progress with installation, despite being broken as ass. However, my testing indicates stripping the TEST_JOBS env var prevents this problem occurring, so will recommend removing this variable from the ebuild until a better solution is found. src_test() { make test_harness || die "test failed"; } # behaves properly and dies at the right time. I removed TEST_JOBS for now. Does the return value depend on the number of failed tests? Here is a failure from perl-5.8.8: | Failed 1 test script out of 939, 99.89% okay. | ... | ### in the 't' directory since most (>=80%) of the tests succeeded | ... | make[1]: [_test_notty] Error 1 (ignored) | make[1]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.8.8-r8/work/perl-5.8.8' | | >>> Install perl-5.8.8-r8 into ... Upstream recognise it as a bug. Its fixed in bleed, but I can't figure out how to adapt the code in the current tree to be closer to bleeds (assumedly) working code. 18:00:52 <kentnl> I would imagine it only matters if the spec url is updated. 19:43:50 <kentnl> I've found an oddity. when I defined TEST_JOBS=3 make harness_test, regardless of how much I encourage the tests to fail ( ie: collossal failures ), make returns a success value, and indeed, perl itself returns a success exit state. I can't for the life of me work out why. 19:44:06 <kentnl> without TEST_JOBS defined, it behaves fine. 19:44:20 <kentnl> ( that is, dying when things are wrong ) 19:45:27 <kentnl> the only difference I can see is that one uses Test::Harness, and the other use TAP::Harness, but I can't see why that makes a difference 21:57:38 <Nicholas> kentnl: which version of perl are you building? I think that it was a bug that has been fixed 21:57:57 <kentnl> 5.10.1 21:58:35 <Nicholas> blead only uses TAP::Harness 22:06:36 <kentnl> f483bab Thu Oct 1 20:51:30 2009 -0400 Make t/harness have non-zero exit if tests fail # xdg++ (In reply to comment #4) > Upstream recognise it as a bug. Its fixed in bleed, but I can't figure out how > to adapt the code in the current tree to be closer to bleeds (assumedly) > working code. So be it. No parallel testing then. Resolved LATER/UPSTREAM/CANTFIX. Thanks Kent |
The whole point of FEATURES="test" is that things don't install unless they pass test. Current Perl ebuild however presently just installs anyway. Many test failures may have gone unreported as a side effect of this problem. x2p/s2p.t ........................................................ ok Test Summary Report ------------------- op/filetest.t (Wstat: 6 Tests: 0 Failed: 0) Non-zero wait status: 6 Parse errors: No plan found in TAP output op/getppid.t (Wstat: 0 Tests: 8 Failed: 2) Failed tests: 6-7 Files=1712, Tests=227069, 933 wallclock secs (32.50 usr 43.47 sys + 346.78 cusr 267.70 csys = 690.45 CPU) Result: FAIL make[2]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.10.1/work/perl-5.10.1' make[1]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.10.1/work/perl-5.10.1' >>> Install perl-5.10.1 into /var/tmp/portage/dev-lang/perl-5.10.1/image/ category dev-lang # this shouldn't be happening. [ebuild U ] dev-lang/perl-5.10.1 [5.8.8-r6] USE="berkdb gdbm -build -debug -doc -ithreads (-perlsuid%)" einfo/environement/blog probably not relevant to this bug as its appears a symptom of how the Makefile is written. ( Will attempt a patch )