Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 293241 - dev-lang/perl-5.10.1 doesn't halt on test failures.
Summary: dev-lang/perl-5.10.1 doesn't halt on test failures.
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Development (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Gentoo Perl team
Depends on:
Blocks: perl-5.10.1
  Show dependency tree
Reported: 2009-11-15 03:55 UTC by Kent Fredric (IRC: kent\n) (RETIRED)
Modified: 2009-11-26 12:49 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2009-11-15 03:55:38 UTC
The whole point of FEATURES="test" is that things don't install unless they pass test.

Current Perl ebuild however presently just installs anyway.

Many test failures may have gone unreported as a side effect of this problem.

x2p/s2p.t ........................................................ ok

Test Summary Report
op/filetest.t                                                  (Wstat: 6 Tests: 0 Failed: 0)
  Non-zero wait status: 6
  Parse errors: No plan found in TAP output
op/getppid.t                                                   (Wstat: 0 Tests: 8 Failed: 2)
  Failed tests:  6-7
Files=1712, Tests=227069, 933 wallclock secs (32.50 usr 43.47 sys + 346.78 cusr 267.70 csys = 690.45 CPU)
Result: FAIL
make[2]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.10.1/work/perl-5.10.1'
make[1]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.10.1/work/perl-5.10.1'

>>> Install perl-5.10.1 into /var/tmp/portage/dev-lang/perl-5.10.1/image/ category dev-lang

# this shouldn't be happening. 

[ebuild     U ] dev-lang/perl-5.10.1 [5.8.8-r6] USE="berkdb gdbm -build -debug -doc -ithreads (-perlsuid%)" 

einfo/environement/blog probably not relevant to this bug as its appears a symptom of how the Makefile is written. ( Will attempt a patch )
Comment 1 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2009-11-15 06:41:31 UTC
My analysis has concluded something weird is occuring with the TAP::Harness based test that are run instead of Test::Harness, when TEST_JOBS is  defined. 

Essentially, this results in perl returning a success exit state regardless of what happens, and I can't make out why. 

I broke my ebuild intentionally so that tests fail like its the apocalypse, and yet, with TEST_JOBS defined, it *STILL* returns a truth value, and would normally progress with installation, despite being broken as ass. 

However, my testing indicates stripping the TEST_JOBS env var prevents this problem occurring, so will recommend removing this variable from the ebuild until a better solution is found.
Comment 2 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2009-11-15 07:32:06 UTC
src_test() {
        make test_harness || die "test failed";

# behaves properly and dies at the right time.
Comment 3 Torsten Veller (RETIRED) gentoo-dev 2009-11-15 09:43:20 UTC
I removed TEST_JOBS for now.

Does the return value depend on the number of failed tests?
Here is a failure from perl-5.8.8:
| Failed 1 test script out of 939, 99.89% okay.
| ...
| ### in the 't' directory since most (>=80%) of the tests succeeded
| ...
| make[1]: [_test_notty] Error 1 (ignored)
| make[1]: Leaving directory `/var/tmp/portage/dev-lang/perl-5.8.8-r8/work/perl-5.8.8'
| >>> Install perl-5.8.8-r8 into ...
Comment 4 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2009-11-15 11:18:32 UTC
Upstream recognise it as a bug. Its fixed in bleed, but I can't figure out how to adapt the code in the current tree to be closer to bleeds (assumedly) working code.

18:00:52 <kentnl> I would imagine it only matters if the spec url is updated.
19:43:50 <kentnl> I've found an oddity. when I defined TEST_JOBS=3 make harness_test, regardless of how much I encourage the tests to fail ( ie: collossal failures ), make returns a success value, and indeed, perl itself returns a success exit state.  I can't for the life of me work out why.
19:44:06 <kentnl> without TEST_JOBS defined, it behaves fine.
19:44:20 <kentnl> ( that is, dying when things are wrong ) 
19:45:27 <kentnl> the only difference I can see is that one uses Test::Harness, and the other use TAP::Harness, but I can't see why that makes a difference
21:57:38 <Nicholas> kentnl: which version of perl are you building? I think that it was a bug that has been fixed
21:57:57 <kentnl> 5.10.1
21:58:35 <Nicholas> blead only uses TAP::Harness
22:06:36 <kentnl> f483bab Thu Oct 1 20:51:30 2009 -0400 Make t/harness have non-zero exit if tests fail # xdg++
Comment 5 Torsten Veller (RETIRED) gentoo-dev 2009-11-26 12:49:29 UTC
(In reply to comment #4)
> Upstream recognise it as a bug. Its fixed in bleed, but I can't figure out how
> to adapt the code in the current tree to be closer to bleeds (assumedly)
> working code.

So be it. No parallel testing then. Resolved LATER/UPSTREAM/CANTFIX.

Thanks Kent