Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 874345

Summary: dev-cpp/doctest: tests fail intermittently on alpha, hppa (timeout-related)
Product: Gentoo Linux Reporter: matoro <matoro_bugzilla_gentoo>
Component: Current packagesAssignee: Petr Vaněk <arkamar>
Status: CONFIRMED ---    
Severity: normal CC: alpha, hppa, matoro_bugzilla_gentoo, mgorny, proxy-maint
Priority: Normal Keywords: PullRequest
Version: unspecified   
Hardware: All   
OS: Linux   
URL: https://github.com/doctest/doctest/issues/711
See Also: https://github.com/gentoo/gentoo/pull/30607
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 837098, 873814    
Attachments: LastTest.log
build.log

Description matoro archtester 2022-10-03 04:11:25 UTC
On alpha, tests for dev-cpp/doctest fail with timeout-related errors, but the exact number of failures varies intermittently, usually between 5 and 10.  Possibly related to clock resolution?


Here's an example of one of the failing cases:

35/155 Testing: test_cases_and_suites.cpp_xml
35/155 Test: test_cases_and_suites.cpp_xml
Command: "/usr/bin/cmake" "-DCOMMAND=/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9_build/examples/all_featur
es/all_features --no-skipped-summary --no-version -sf=*test_cases_and_suites.cpp --reporters=xml --dt-no-path-filenames=
1 --dt-no-line-numbers=1 --dt-no-exitcode=1 --dt-gnu-file-line=0 --dt-no-time-in-output=1" "-DTEST_OUTPUT_FILE=/var/tmp/
portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9/examples/all_features/test_output/test_cases_and_suites.cpp_xml.txt" "-
DTEST_TEMP_FILE=/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9_build/examples/all_features/CMakeFiles/temp_te
st_output_test_cases_and_suites.cpp_xml.txt" "-DTEST_MODE=COMPARE" "-P" "/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doc
test-2.4.9/scripts/cmake/exec_test.cmake"
Directory: /var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9_build/examples/all_features
"test_cases_and_suites.cpp_xml" start time: Oct 03 04:09 UTC
Output:
----------------------------------------------------------
-- Found Git: /usr/bin/git (found version "2.37.3") 
diff --git a/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9/examples/all_features/test_output/test_cases_and_s
uites.cpp_xml.txt b/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9_build/examples/all_features/CMakeFiles/temp
_test_output_test_cases_and_suites.cpp_xml.txt
index 6fc245a..e9d1e79 100644
--- a/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9/examples/all_features/test_output/test_cases_and_suites.c
pp_xml.txt
+++ b/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9_build/examples/all_features/CMakeFiles/temp_test_output_t
est_cases_and_suites.cpp_xml.txt
@@ -54,7 +54,7 @@
           failing because of the timeout decorator!
         </Text>
       </Message>
-      <OverallResultsAsserts successes="0" failures="0" test_case_success="false"/>
+      <OverallResultsAsserts successes="0" failures="0" test_case_success="true"/>
     </TestCase>
   </TestSuite>
   <TestSuite name="skipped test cases">
@@ -109,6 +109,6 @@
     </TestCase>
   </TestSuite>
   <OverallResultsAsserts successes="1" failures="11"/>
-  <OverallResultsTestCases successes="6" failures="9"/>
+  <OverallResultsTestCases successes="7" failures="8"/>
 </doctest>
 Program code.

CMake Error at /var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9/scripts/cmake/exec_test.cmake:69 (message):
  Running
  '/var/tmp/portage/dev-cpp/doctest-2.4.9/work/doctest-2.4.9_build/examples/all_features/all_features
  --no-skipped-summary --no-version -sf=*test_cases_and_suites.cpp
  --reporters=xml --dt-no-path-filenames=1 --dt-no-line-numbers=1
  --dt-no-exitcode=1 --dt-gnu-file-line=0 --dt-no-time-in-output=1' ended
  with code 'Output is different from reference file!'

<end of output>
Test time =   0.22 sec
----------------------------------------------------------
Test Failed.
"test_cases_and_suites.cpp_xml" end time: Oct 03 04:09 UTC
"test_cases_and_suites.cpp_xml" time elapsed: 00:00:00
----------------------------------------------------------

Reproducible: Always
Comment 1 matoro archtester 2022-10-03 04:12:16 UTC
Created attachment 816124 [details]
LastTest.log
Comment 2 matoro archtester 2022-10-03 04:12:54 UTC
Created attachment 816127 [details]
build.log
Comment 3 matoro archtester 2022-10-03 04:13:33 UTC
It also requires dev-vcs/git in BDEPEND, but that's a moot point until we can get the tests passing.
Comment 4 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2022-10-03 05:14:59 UTC
This is a true horror — and the output is completely unreadable.
Comment 5 Petr Vaněk gentoo-dev 2022-10-03 18:55:49 UTC
(In reply to matoro from comment #3)
> It also requires dev-vcs/git in BDEPEND, but that's a moot point until we
> can get the tests passing.

Actually git is optional, it is used only if it is found only for showing difference in test results, see [1]. I decided to not put it to the BDEPEND list because it is useful only for debugging purposes.

[1] https://github.com/doctest/doctest/blob/86892fc480f80fb57d9a3926cb506c0e974489d8/scripts/cmake/exec_test.cmake#L42-L47
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2022-11-29 16:40:26 UTC
Could you try ignoring the problem and seeing if revdeps work?  I suppose they may not be affected by this.
Comment 7 matoro archtester 2022-11-29 18:36:31 UTC
(In reply to Michał Górny from comment #6)
> Could you try ignoring the problem and seeing if revdeps work?  I suppose
> they may not be affected by this.

They almost certainly will since this pretty much only affects the timeout decorator feature, but signing off on a keyword when tests are explicitly not passing kind of defeats the point of the tests.  This is on my todo list to go back and re-investigate, and of course if anyone else would like to take a crack at it, just let me know for hardware access.  This blocks two of the three remaining items left to transition alpha from exp -> dev.
Comment 8 matoro archtester 2022-11-30 04:35:13 UTC
This also affects hppa apparently.
Comment 9 matoro archtester 2023-04-15 15:20:56 UTC
https://github.com/doctest/doctest/pull/770

This fixes HPPA, and partially fixes alpha (remaining items are segfaults, not test failures).
Comment 10 Larry the Git Cow gentoo-dev 2023-04-26 04:03:31 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b88bedd14cd548acfdbabb96a4300522db9f2109

commit b88bedd14cd548acfdbabb96a4300522db9f2109
Author:     Matoro Mahri <matoro@users.noreply.github.com>
AuthorDate: 2023-04-15 19:33:59 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-04-26 04:02:59 +0000

    dev-cpp/doctest: add patch for tests with clock resolution problems
    
    See: https://github.com/doctest/doctest/pull/770
    See: https://github.com/doctest/doctest/issues/711
    Bug: https://bugs.gentoo.org/874345
    Signed-off-by: Matoro Mahri <matoro@users.noreply.github.com>
    Closes: https://github.com/gentoo/gentoo/pull/30607
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-cpp/doctest/doctest-2.4.11.ebuild              |  1 +
 .../files/doctest-2.4.11-backport-pr770.patch      | 72 ++++++++++++++++++++++
 2 files changed, 73 insertions(+)