Test phase fails with the following errors: sudo python3 tests/run_tests.py /bin/bash: line 1: sudo: command not found Looks like app-admin/sudo is a test phase runtime dependency. ModuleNotFoundError: No module named 'bytesize' Looks like dev-libs/libbytesize[python] is a test phase runtime dependency FileNotFoundError: [Errno 2] No such file or directory: 'targetcli' Looks like sys-block/targetcli-fb is a test phase runtime dependency. None of these seem to be pulled in by the ebuild if testing is enabled with the exception of libbytesize, but doesn't check for the python USE requirement. It also seems that some tests skip based on USE flags such as USE="-vdo" but others like USE="kbd lvm" don't. Test phase also complains about the kernel not having ZRAM which doesn't get a kernel config pre-merge check. There's more stuff going on with the testing but I don't have more time to put into checking them, so I'm sharing what I've found since it doesn't seem to be reported at the moment.
Note that in general, we don't _want_ sudo as a test dep, as it implies it's going to do naughty things. Anyway, the tests are restricted in this ebuild, for exactly this reason: https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/libblockdev/libblockdev-2.27.ebuild#n28 ># Tests require root. In a future release, we may be able to run a smaller ># subset with new run_tests.py arguments. >RESTRICT="!test? ( test ) test" (Notice the last "test" bit, first part is fine.) If interested in trying to get some of the test suite running, patches very much welcome though.
(In reply to Sam James from comment #1) > Note that in general, we don't _want_ sudo as a test dep, as it implies it's > going to do naughty things. > > Anyway, the tests are restricted in this ebuild, for exactly this reason: > https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/libblockdev/ > libblockdev-2.27.ebuild#n28 > ># Tests require root. In a future release, we may be able to run a smaller > ># subset with new run_tests.py arguments. > >RESTRICT="!test? ( test ) test" > > (Notice the last "test" bit, first part is fine.) > > If interested in trying to get some of the test suite running, patches very > much welcome though. Yes I know. I'm compiling and reporting on test errors as I come along them even if they are restricted. Maybe some stuff is good to know about such as CFLAGS breaking expected behavior causing test suites to fail even when restricted. Is there a way to find out what tests require sudo to run?
(In reply to unhappy-ending from comment #2) > (In reply to Sam James from comment #1) > > Note that in general, we don't _want_ sudo as a test dep, as it implies it's > > going to do naughty things. > > > > Anyway, the tests are restricted in this ebuild, for exactly this reason: > > https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/libblockdev/ > > libblockdev-2.27.ebuild#n28 > > ># Tests require root. In a future release, we may be able to run a smaller > > ># subset with new run_tests.py arguments. > > >RESTRICT="!test? ( test ) test" > > > > (Notice the last "test" bit, first part is fine.) > > > > If interested in trying to get some of the test suite running, patches very > > much welcome though. > > Yes I know. I'm compiling and reporting on test errors as I come along them > even if they are restricted. Maybe some stuff is good to know about such as > CFLAGS breaking expected behavior causing test suites to fail even when > restricted. Is there a way to find out what tests require sudo to run? It'd be far more valuable to try get test suites in a state where we can run them, even a subset of. Like, in this case, sudo is very much deliberately not a test dependency because we don't want to run those tests anyway. Any sane subset we might run would never involve sudo either. bytesize & targetcli are valid things which could be added though. But really, and I do welcome adding more test suites!, the effort needs to go into just running some sane subset and fixing up the rest -- or if you insist on running restricted tests and not making an effort to unrestrict them, give patches which can be applied to at least accommodate your use case. I'm dogfooding here -- I've spent several hours this afternoon doing exactly this with Bind and I'm now doing the same with NSS and NSPR (getting their test suites running with some wrangling).
(In reply to Sam James from comment #3) > It'd be far more valuable to try get test suites in a state where we can run > them, even a subset of. > > Like, in this case, sudo is very much deliberately not a test dependency > because we don't want to run those tests anyway. Any sane subset we might > run would never involve sudo either. > > bytesize & targetcli are valid things which could be added though. > > But really, and I do welcome adding more test suites!, the effort needs to > go into just running some sane subset and fixing up the rest -- or if you > insist on running restricted tests and not making an effort to unrestrict > them, give patches which can be applied to at least accommodate your use > case. > > I'm dogfooding here -- I've spent several hours this afternoon doing exactly > this with Bind and I'm now doing the same with NSS and NSPR (getting their > test suites running with some wrangling). I'm not a programmer so my contributions can't be much other than reports and some digging on why tests fail for various reasons or finding libbytesize[python] and targetcli-fb are missing requirements. That's a least a step towards getting a more sane test suite knowing beforehand things that are missing. My use case for restricted tests is using system wide Clang and the LLVM toolchain. I want to know, does it break compared to GCC? Sometimes restricted or missing tests give me that info. I had to patch in testing to double-conversion, it's a simple patch to switch it on to find out Clang breaks the test suite but GCC doesn't. Haven't had a chance to submit the patch or report it yet. I spend a lot of time doing this, it takes many hours to test, test, and test again. If I get a chance I would be happy to find out what tests can be done with a normal user with the right permissions and which ones require root for libblockdev and help sort a white list of sorts. A patch for that shouldn't be too hard and it seems libblockdev does have the tests broken up to a degree so maybe not all of them requires root. However, right now, I'm just going through my list of failures and if it's simple stuff like missing deps, I'm trying to report that and get it out of the way.
(In reply to unhappy-ending from comment #4) > (In reply to Sam James from comment #3) > > It'd be far more valuable to try get test suites in a state where we can run > > them, even a subset of. > > > > Like, in this case, sudo is very much deliberately not a test dependency > > because we don't want to run those tests anyway. Any sane subset we might > > run would never involve sudo either. > > > > bytesize & targetcli are valid things which could be added though. > > > > But really, and I do welcome adding more test suites!, the effort needs to > > go into just running some sane subset and fixing up the rest -- or if you > > insist on running restricted tests and not making an effort to unrestrict > > them, give patches which can be applied to at least accommodate your use > > case. > > > > I'm dogfooding here -- I've spent several hours this afternoon doing exactly > > this with Bind and I'm now doing the same with NSS and NSPR (getting their > > test suites running with some wrangling). > > I'm not a programmer so my contributions can't be much other than reports > and some digging on why tests fail for various reasons or finding > libbytesize[python] and targetcli-fb are missing requirements. That's a > least a step towards getting a more sane test suite knowing beforehand > things that are missing. > And that's fair, but be up front about what you're doing. Otherwise it looks _really_ confusing. For example, me and somebody else were worried that somehow restricted tests were being run in normal operation. > My use case for restricted tests is using system wide Clang and the LLVM > toolchain. I want to know, does it break compared to GCC? Sometimes > restricted or missing tests give me that info. I had to patch in testing to > double-conversion, it's a simple patch to switch it on to find out Clang > breaks the test suite but GCC doesn't. Haven't had a chance to submit the > patch or report it yet. I spend a lot of time doing this, it takes many > hours to test, test, and test again. Sounds like something we want to fix, yep. > > If I get a chance I would be happy to find out what tests can be done with a > normal user with the right permissions and which ones require root for > libblockdev and help sort a white list of sorts. A patch for that shouldn't > be too hard and it seems libblockdev does have the tests broken up to a > degree so maybe not all of them requires root. However, right now, I'm just > going through my list of failures and if it's simple stuff like missing > deps, I'm trying to report that and get it out of the way. My concern also comes from the fact that the sandbox is not infallible and it's not generally safe to run test suites as root. They're often destructive. I don't need full patches, but a summary of your findings would be good, just be up front about the fact you know you're doing something odd.
Created attachment 785816 [details, diff] add-partial-test-deps.patch We can add some of them now (we really don't want to do sudo). Grubby patch attached. It can't be committed as-is b/c the targetcli-fb dep isn't keyworded on e.g. arm. It'd be useful (but not essential) if you could figure out which tests need it so we can skip them conditionally or something. You might want to play with: - http://storaged.org/libblockdev/ch03.html - ... the --help from run_tests.py. I don't mind doing the wiring up in ebuilds if you can tell me what you've learned from doing it manually.
(In reply to Sam James from comment #5) > And that's fair, but be up front about what you're doing. Otherwise it looks > _really_ confusing. For example, me and somebody else were worried that > somehow restricted tests were being run in normal operation. That's absolutely fair and I didn't consider this point of view. I assumed developers would've known it was a restricted test and the user override, but that's a valid concern I didn't consider. In the future when it comes to these cases, I will state up front the odd use case and overriding a restricted test. (In reply to Sam James from comment #6) > I don't mind doing the wiring up in ebuilds if you can tell me what you've > learned from doing it manually. Thank you! I want to contribute it ways I'm capable of :)
Sounds like a good plan! :)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ca028041dc65001b88300491a73c887d0f0f52a6 commit ca028041dc65001b88300491a73c887d0f0f52a6 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-06-16 23:16:57 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-06-16 23:17:08 +0000 dev-libs/double-conversion: wire up tests Bug: https://bugs.gentoo.org/852635 Thanks-to: unhappy-ending Signed-off-by: Sam James <sam@gentoo.org> dev-libs/double-conversion/double-conversion-3.2.0.ebuild | 10 ++++++++++ 1 file changed, 10 insertions(+)
I think we have waited long enough. USE-masking tests on looong and proceeding.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=35bb28e7bc54e5797075c510acc34cd160b91355 commit 35bb28e7bc54e5797075c510acc34cd160b91355 Author: Marek Szuba <marecki@gentoo.org> AuthorDate: 2022-10-04 18:38:15 +0000 Commit: Marek Szuba <marecki@gentoo.org> CommitDate: 2022-10-04 18:45:24 +0000 sys-libs/libblockdev: add partial test dependencies Suggested-by: unhappy-ending <unhappy-ending@protonmail.com> Closes: https://bugs.gentoo.org/852635 Signed-off-by: Marek Szuba <marecki@gentoo.org> sys-libs/libblockdev/libblockdev-2.28.ebuild | 7 +++++++ 1 file changed, 7 insertions(+)
