Following the talk about the continuous integration testing in CentOS, I started to consider that we could go a step further in testing the software in Gentoo, for environments such as the tinderbox, by running one further step beside the src_test() function. While src_test() is executed within the scope of the build process, we could make use of a pkg_test() that is executed with the package installed. This is not intended as a general-usage testing framework (src_test() would be used for that still) but for actual integration, which means it needs to be able to start and stop services on the system, configure them and so on so forth, even incurring in the risk of breaking the system it is being run on. Things needed: - a pkg_test() function; - a separate feature to set it up (as I said it might be destructive so it's _much_ better to avoid letting it to be used for general availability); - a separate USE flag (I'd rather not abuse the test USE flag at runtime) for the dependencies required to run pkg_test code. Things nice to have: - a way to deal with services requiring network, to avoid collide with system services that _might_ be required for the continuous integration jobs, or for Portage: we can either decide to get everything open on a separate, loopback IP address (such as 127.255.255.254 as I'm using for dev-ruby/mongoid), or we could run them in their own network namespace, so they have lo just for themselves; As an example, MySQL and Ruby both have extensive testsuites that can only be ran against the installed copy of the package, and not the one just built: being able to test this stuff would be pretty amazing for us.
I recently came across a similar feature in EasyBuild called sanity-check: https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#sanity-check It is especially convenient for packages which provide (unreliable) python extensions: sanity_check_commands = ["python -c 'import pythonextension'"]
