| Summary: | set src_test to run by default | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | SpanKY <vapier> |
| Component: | Core | Assignee: | Portage team <dev-portage> |
| Status: | CONFIRMED --- | ||
| Severity: | enhancement | CC: | cardoe, esigra, mgorny, pacho, pms, rhill |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
SpanKY
2007-07-10 09:19:48 UTC
From the gentoo-dev thread about this it didn't seem like most people would have been in favor of this. note my note ... the hate on the gentoo-dev mailing list was for the idea of always forcing test regardless of anything else for EAPI-1 (In reply to comment #2) > note my note ... the hate on the gentoo-dev mailing list was for the idea of > always forcing test regardless of anything else for EAPI-1 > Well to be on by default it probably needs one of those categorization things discussed on the mailing list for resource hungry, very long tests etc. Maybe we should have an additional variable that in global scope that controls whether the tests are run by default. classifying the resource needs of tests is a different topic i think ... people can still do FEATURES=-test in their make.conf considering i have 11 system packages[i] i have to override with FEATURES="-test" in /etc/portage/env[ii] to be able to get through an `emerge -e system` successfully, i'd say we're a ways off from having tests enabled by default.
i would love to have a FEATURES="developer" or somesuch that would enable tests, the QA source code warnings, FEATURES="cvs", collision-protect, splitdebug, etc.
[i] dirtyepic@halo /etc/portage/env $ tree
.
|-- app-text
| `-- poppler-bindings
|-- cross-mips-unknown-linux-gnu
| |-- binutils
| |-- gcc
| `-- glibc
|-- dev-libs
| |-- boost
| `-- elfutils
|-- dev-python
| `-- dbus-python
|-- dev-util
| |-- dejagnu
| `-- valgrind
|-- gnome-extra
| `-- gucharmap
|-- media-libs
| `-- gst-plugins-base
|-- sci-libs
| `-- fftw
|-- sys-apps
| |-- coreutils
| `-- hal
|-- sys-devel
| |-- autoconf
| |-- automake
| |-- binutils
| |-- gcc
| |-- gdb
| |-- libtool
| `-- prelink
|-- sys-libs
| |-- db
| `-- glibc
`-- x11-libs
`-- cairo
(some of these might be fixed and i just haven't checked lately)
[ii] yes, ugly hack ;)
Not at all. When people bump their packages and move to EAPI 2, they'll either fix the test suites that they previously ignored, or if that's not doable they'll restrict tests for that package. This means that as the tree moves to using newer EAPIs, we'll go from src_test being a lottery (does a failure mean something's wrong or just a lazy maintainer?) to it being reliable. While I'd like to see this happen, there are several packages in the tree that require additional dependencies that cause circular dependencies to resolve. For example, I would VERY VERY much like this to be run for cairo because it would reduce the number of bugs filed for cairo to virtually zero. But... cario tests need GTK+ to be installed and available, however GTK+ depends on cairo. So what we'd need is an ability to specify dependencies for tests that means the package source isn't cleaned and we come back to it after the dependencies are satisfied. How is PMS actually affected in this one? We add in a phrase requiring the PM to run src_test unless the user explicitly disables it, or unless it's RESTRICTed. So does anyone still care about this one? I doubt we'll get community agreement on running tests by default but feel free to try. Otherwise, we should close this as WONTFIX. This one needs someone to step in and show some leadership. In the long term it's best for the distribution. As long as people continue to blatantly ignore testsuite failures in their packages this feature is worse than useless for the general user. We've actually made negative progress here; we have many more failures now than we did six years ago. (I'm actually surprised when a python package passes these days.) At this point any potential benefit of enabling tests by default is drowned out by the noise of false positives, the added cost in build times and additional dependencies, and the general broken window syndrome that results when our users see every second or third package throwing errors (even non-fatal ones as in this case). I'd really like it to be enabled. Barring some major changes in our QA policies, which would make everyone unhappy, I don't see it happening. I still don't think this is a PMS matter. Furthermore, I think this is better handled by encouraging developers to enable tests for the packages they maintain. This ofc brings problems for project-maintained packages which usually have many 'bumpers' and no real caring maintainer. Yet another problem is how developers handle failing tests. It's still a common practice to silently disable any test that fails. In many cases, tests that are broken are straight disabled without attempting to fix them or report the problem upstream. In some cases, the failing tests are disabled without even checking if they don't indicate major problems (it always failed, so why users should be bothered about some known problem with the package?). If you can get Portage to default to enabling tests, we can think about putting a note about it to PMS. But still, I don't think PMS is really supposed to say what the default for this should be. |
