As was discovered in Bug 661218, the mere presence of the pytest-relaxed plugin on the system would break unit tests of packages that were not intended to be used with it. I recently ran into similar issues with the pytest-twisted plugin. Back in 2018, at the request of Gentoo developer Virgil Dupras, Pytest upstream implemented an option to disable autoloading of plugins entirely: https://github.com/pytest-dev/pytest/issues/3784 However, this was apparently never followed up on, as the epytest function in python-utils-r1.eclass only disables a select few plugins explicitly and does not make use of the option that was implemented for us. Please make epytest set PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 while running Pytest. This should mean that "-p no:cov -p no:flake8 -p no:flakes -p no:pylint" no longer need to be specified, as all plugins will be disabled by default. Note that this change *will* break the tests in many ebuilds, which will now need to explicitly specify their pytest plugins rather than relying on autoloading of whatever plugins happen to be installed on the system. As such, I do not make this request lightly, but I do believe it is in the best interest of all Gentoo Python developers and users to reduce environment variability when running unit tests of Python packages. Switching to explicit specification of required Pytest plugins seems like a win in the long run.
I thought I'd remembered mgorny exploring this but maybe not (might be thinking of https://github.com/gentoo/gentoo/commit/f10588909c89639ce3a68ed2bcff6d7b05da1ba8, which just disables some plugins).
I've been thinking about this in the past but I think the breakage is too large. I mean, figuring out all the test failures and finding the right set of plugins is a huge undertaking. It doesn't help that usually the resulting failures and errors are far from explanatory. It doesn't help that some packages use some plugins optionally but don't check correctly for them. For example, they enable xdist-related code if 'import xdist' works, which causes the package to explode when xdist plugin is not actually loaded.
(if at all, it can only happen in a new EAPI)
Finally, there's the problem of indirect pytest tests that would be affected by the envvar but can't be trivially fixed to load the right set of plugins.
All good points. Thank you for your consideration. Perhaps just recommend setting the envvar as a best practice in your Python development guide, although you're right that the errors that result from not having a plugin loaded when it's needed (and also having a plugin loaded when it's not expected!) are far from explanatory. In my case, the autoloading of the pytest-twisted plugin was breaking dynamic class method dispatch in the python-bitcointx library, a totally non-obvious and perplexing side effect.
While I'm for adding this wherever we can, I don't think it's possible or even reasonable to force it everywhere.
