Following the talk about the continuous integration testing in CentOS, I started to consider that we could go a step further in testing the software in Gentoo, for environments such as the tinderbox, by running one further step beside the src_test() function. While src_test() is executed within the scope of the build process, we could make use of a pkg_test() that is executed with the package installed. This is not intended as a general-usage testing framework (src_test() would be used for that still) but for actual integration, which means it needs to be able to start and stop services on the system, configure them and so on so forth, even incurring in the risk of breaking the system it is being run on. Things needed: - a pkg_test() function; - a separate feature to set it up (as I said it might be destructive so it's _much_ better to avoid letting it to be used for general availability); - a separate USE flag (I'd rather not abuse the test USE flag at runtime) for the dependencies required to run pkg_test code. Things nice to have: - a way to deal with services requiring network, to avoid collide with system services that _might_ be required for the continuous integration jobs, or for Portage: we can either decide to get everything open on a separate, loopback IP address (such as 127.255.255.254 as I'm using for dev-ruby/mongoid), or we could run them in their own network namespace, so they have lo just for themselves; As an example, MySQL and Ruby both have extensive testsuites that can only be ran against the installed copy of the package, and not the one just built: being able to test this stuff would be pretty amazing for us.
I recently came across a similar feature in EasyBuild called sanity-check: https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#sanity-check It is especially convenient for packages which provide (unreliable) python extensions: sanity_check_commands = ["python -c 'import pythonextension'"]
See also https://public-inbox.gentoo.org/gentoo-dev/87ttftr9ay.fsf@gentoo.org/ regarding installed tests.
I wonder if we could trial this with an eclass that users would opt-in to testing for w/ pkg_postinst.
preinst may be preferable. That would enable us to abort if tests fail, right?
(In reply to Michał Górny from comment #4) > preinst may be preferable. That would enable us to abort if tests fail, > right? I was thinking postinst, in case software really requires to use the installed instance (think DESTDIR vs PREFIX). BUT the two examples I'm thinking of today (the kernel selftests, dtrace) don't have that requirement AFAIK as they support running in the source tree, just are too expensive / ... / to run in sandbox. So, it'd work for me to do the experiment with preinst, I think.
(In reply to Michał Górny from comment #4) > preinst may be preferable. That would enable us to abort if tests fail, > right? I've just discovered a regression in our texlive packages where 'tlmgr' doesn't work any more. It would be great if we had a pkg_test() phase where we could rudimentary test for tlmgr's functionality (in this particular case, that the Perl INC paths are correct). However, 'tlmgr' requires the TexLive Package Database (tlpdb) to be initialized, which we dynamically create, for reasons, in pkg_postinst(). Therefore, to be useful for this use-case, pkg_test() should be run *after* pkg_postinst().
