i think it'd be useful if we saved gcc testsuite results (.sum and .log files) and installed them in DOCDIR or somewhere. i'd also like to add the -k option to make check in gcc_src_test() so we run the whole testsuite instead of bailing out after the first error.
Created attachment 146986 [details, diff] toolchain.eclass.diff something like this might work.
i seem to recall those files being particularly large ... how small do they get when compressed ? the -k part in src_test is OK
they compress surprisingly well. lots of repeated lines i guess. dirtyepic@tycho /usr/share/doc/gcc-4.3.1_pre20080320/testsuite $ du -a 4.0K ./treelang.sum.bz2 20K ./libstdc++.sum.bz2 4.0K ./treelang.log.bz2 8.0K ./libgomp.log.bz2 4.0K ./libgomp.sum.bz2 64K ./libstdc++.log.bz2 148K ./gcc.sum.bz2 260K ./g++.log.bz2 340K ./gcc.log.bz2 60K ./g++.sum.bz2 916K . 916K total uncompressed they're around 50MiB.
should be fine then if you want to commit it
Done, thanks.
I recently did: ``` commit 9ac3f1cf62b522236ba9efd7e923071c37df1e6d Author: Sam James <sam@gentoo.org> Date: Sat Jul 30 05:46:15 2022 +0100 toolchain.eclass: store previous test results & compare in src_test This is what GCC developers do, so let's give it a try. Note that folks will have to (for now?) set GCC_TESTS_IGNORE_NO_BASELINE=1 for it to succeed if src_test failed for the first time & they don't have any test data saved for that slot. The test data is stored in ${EPREFIX}/var/cache/gcc/${SLOT}. GCC upstream's contrib/compare_tests script performs the comparison. We may want to investigate shipping baseline results / gathering them in a repo somewhere. We may also in future want to install this data as orphaned so that it's kept around and not lost when upgrading versions and to allow cross-slot comparison when an old one is depcleaned (although I guess doesn't matter so much if it's first time, but still). Signed-off-by: Sam James <sam@gentoo.org> ``` ... I didn't realise we did this in the past: ``` commit 3c6057a074453955008b2b8cae0f35e1b07aa313 Author: Ryan Hill <dirtyepic@gentoo.org> Date: Sun Mar 23 16:33:17 2008 +0000 Add -k to make check so we always run the full testsuite rather than bail on the first error. Save test results in /usr/share/doc/. ``` slyfox dropped it in: ``` commit d16e284022086cb9dad3337099eebe4e90d52691 Author: Sergei Trofimovich <slyfox@gentoo.org> Date: Mon Nov 4 22:52:55 2019 +0000 toolchain.eclass: rename USE=regression-test to USE=test gcc-8.3 is stable on all the platforms for a while. Renaming 'regression-test' to 'test' for older ebuilds as well. While at it: - make test runs unconditional - drop installation of test results - add RESTRICT="!test? ( test )" - drop 'regression-test' from metadata files Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> ``` It's not clear to me why.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c4eccac296c608e3fafad22d678540ddaab950d7 commit c4eccac296c608e3fafad22d678540ddaab950d7 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-17 06:47:09 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-23 15:40:52 +0000 toolchain.eclass: don't install all .sum files Just rely on the validate_failures.py manifests instead. These logs get real big real fast. People can save build logs if they want to look at the tests otherwise. Bug: https://bugs.gentoo.org/214345 Bug: https://bugs.gentoo.org/253926 Signed-off-by: Sam James <sam@gentoo.org> eclass/toolchain.eclass | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1d93a491096f1cc0234fcf44458bfec142c213bb commit 1d93a491096f1cc0234fcf44458bfec142c213bb Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-16 07:00:23 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-23 15:40:51 +0000 toolchain.eclass: rework tests more Rework src_test around contrib/testsuite-management/validate_failures.py in gcc.git. This script is being used by the new Linaro CI effort and it appears well-suited to us, as it allows marking expected failures easily. Followup to 9ac3f1cf62b522236ba9efd7e923071c37df1e6d. Bug: https://bugs.gentoo.org/214345 Bug: https://bugs.gentoo.org/253926 Signed-off-by: Sam James <sam@gentoo.org> eclass/toolchain.eclass | 148 ++++++++++++++++++------- sys-devel/gcc/Manifest | 1 + sys-devel/gcc/gcc-10.5.0.ebuild | 1 + sys-devel/gcc/gcc-11.4.1_p20240111.ebuild | 1 + sys-devel/gcc/gcc-11.4.1_p20240208.ebuild | 1 + sys-devel/gcc/gcc-11.4.1_p20240222.ebuild | 1 + sys-devel/gcc/gcc-11.4.1_p20240229.ebuild | 1 + sys-devel/gcc/gcc-11.4.1_p20240307.ebuild | 1 + sys-devel/gcc/gcc-11.4.1_p20240314.ebuild | 1 + sys-devel/gcc/gcc-11.5.9999.ebuild | 1 + sys-devel/gcc/gcc-12.3.1_p20240112.ebuild | 1 + sys-devel/gcc/gcc-12.3.1_p20240209.ebuild | 1 + sys-devel/gcc/gcc-12.3.1_p20240223.ebuild | 1 + sys-devel/gcc/gcc-12.3.1_p20240301.ebuild | 1 + sys-devel/gcc/gcc-12.3.1_p20240308.ebuild | 1 + sys-devel/gcc/gcc-12.3.1_p20240315.ebuild | 1 + sys-devel/gcc/gcc-12.4.9999.ebuild | 1 + sys-devel/gcc/gcc-13.2.1_p20240113-r1.ebuild | 1 + sys-devel/gcc/gcc-13.2.1_p20240210.ebuild | 1 + sys-devel/gcc/gcc-13.2.1_p20240224.ebuild | 1 + sys-devel/gcc/gcc-13.2.1_p20240302.ebuild | 1 + sys-devel/gcc/gcc-13.2.1_p20240309.ebuild | 1 + sys-devel/gcc/gcc-13.2.1_p20240316.ebuild | 1 + sys-devel/gcc/gcc-13.3.9999.ebuild | 1 + sys-devel/gcc/gcc-14.0.1_pre20240218.ebuild | 1 + sys-devel/gcc/gcc-14.0.1_pre20240225.ebuild | 1 + sys-devel/gcc/gcc-14.0.1_pre20240303-r1.ebuild | 1 + sys-devel/gcc/gcc-14.0.1_pre20240310.ebuild | 1 + sys-devel/gcc/gcc-14.0.1_pre20240317.ebuild | 1 + sys-devel/gcc/gcc-14.0.9999.ebuild | 1 + sys-devel/gcc/gcc-8.5.0-r1.ebuild | 1 + sys-devel/gcc/gcc-9.5.0.ebuild | 1 + 32 files changed, 141 insertions(+), 38 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=abf8e2ee55c52c8ae894e0b3845ea1cebfcfd4e8 commit abf8e2ee55c52c8ae894e0b3845ea1cebfcfd4e8 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-16 01:38:45 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-23 15:40:50 +0000 toolchain.eclass: install test results as orphaned files in /var/cache/gcc This allows comparison across versions even after they get upgraded, which is useful in itself (and across series), but also for looking into when regressions started if they're reported but started a while ago. Followup to 9ac3f1cf62b522236ba9efd7e923071c37df1e6d. Bug: https://bugs.gentoo.org/214345 Bug: https://bugs.gentoo.org/253926 Signed-off-by: Sam James <sam@gentoo.org> eclass/toolchain.eclass | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
