The whole point of FEATURES="test" is that things don't install unless they pass test. Current Perl ebuild however presently just installs anyway. Many test failures may have gone unreported as a side effect of this problem. x2p/s2p.t ........................................................ ok Test Summary Report ------------------- op/filetest.t (Wstat: 6 Tests: 0 Failed: 0) Non-zero wait status: 6 Parse errors: No plan found in TAP output op/getppid.t (Wstat: 0 Tests: 8 Failed: 2) Failed tests: 6-7 Files=1712, Tests=227069, 933 wallclock secs (32.50 usr 43.47 sys + 346.78 cusr 267.70 csys = 690.45 CPU) Result: FAIL make[2]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.10.1/work/perl-5.10.1' make[1]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.10.1/work/perl-5.10.1' >>> Install perl-5.10.1 into /var/tmp/portage/dev-lang/perl-5.10.1/image/ category dev-lang # this shouldn't be happening. [ebuild U ] dev-lang/perl-5.10.1 [5.8.8-r6] USE="berkdb gdbm -build -debug -doc -ithreads (-perlsuid%)" einfo/environement/blog probably not relevant to this bug as its appears a symptom of how the Makefile is written. ( Will attempt a patch )
My analysis has concluded something weird is occuring with the TAP::Harness based test that are run instead of Test::Harness, when TEST_JOBS is defined. Essentially, this results in perl returning a success exit state regardless of what happens, and I can't make out why. I broke my ebuild intentionally so that tests fail like its the apocalypse, and yet, with TEST_JOBS defined, it *STILL* returns a truth value, and would normally progress with installation, despite being broken as ass. However, my testing indicates stripping the TEST_JOBS env var prevents this problem occurring, so will recommend removing this variable from the ebuild until a better solution is found.
src_test() { make test_harness || die "test failed"; } # behaves properly and dies at the right time.
I removed TEST_JOBS for now. Does the return value depend on the number of failed tests? Here is a failure from perl-5.8.8: | Failed 1 test script out of 939, 99.89% okay. | ... | ### in the 't' directory since most (>=80%) of the tests succeeded | ... | make[1]: [_test_notty] Error 1 (ignored) | make[1]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.8.8-r8/work/perl-5.8.8' | | >>> Install perl-5.8.8-r8 into ...
Upstream recognise it as a bug. Its fixed in bleed, but I can't figure out how to adapt the code in the current tree to be closer to bleeds (assumedly) working code. 18:00:52 <kentnl> I would imagine it only matters if the spec url is updated. 19:43:50 <kentnl> I've found an oddity. when I defined TEST_JOBS=3 make harness_test, regardless of how much I encourage the tests to fail ( ie: collossal failures ), make returns a success value, and indeed, perl itself returns a success exit state. I can't for the life of me work out why. 19:44:06 <kentnl> without TEST_JOBS defined, it behaves fine. 19:44:20 <kentnl> ( that is, dying when things are wrong ) 19:45:27 <kentnl> the only difference I can see is that one uses Test::Harness, and the other use TAP::Harness, but I can't see why that makes a difference 21:57:38 <Nicholas> kentnl: which version of perl are you building? I think that it was a bug that has been fixed 21:57:57 <kentnl> 5.10.1 21:58:35 <Nicholas> blead only uses TAP::Harness 22:06:36 <kentnl> f483bab Thu Oct 1 20:51:30 2009 -0400 Make t/harness have non-zero exit if tests fail # xdg++
(In reply to comment #4) > Upstream recognise it as a bug. Its fixed in bleed, but I can't figure out how > to adapt the code in the current tree to be closer to bleeds (assumedly) > working code. So be it. No parallel testing then. Resolved LATER/UPSTREAM/CANTFIX. Thanks Kent
