Summary: | binpkgs compiled with USE=-test cannot be used with FEATURES=test (and vice versa) | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Eli Schwartz <eschwartz> |
Component: | Binary packages support | Assignee: | Portage team <dev-portage> |
Status: | CONFIRMED --- | ||
Severity: | normal | CC: | eschwartz, esigra, gentoo, mgorny, pms, sam |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | All | ||
See Also: | https://bugs.gentoo.org/show_bug.cgi?id=517658 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 912975 |
Description
Eli Schwartz
2023-08-27 18:07:06 UTC
IMO option 3 is the sensible solution here, although option 2 could have broader usefulness and I wouldn't say no to it. I'm going to hack up something for 1) / 2) at least. Note that per the PMS, USE=test is not in any way special, so a third-party package could actually use it to control installing something. We may also need to set an explicit ::gentoo policy to require USE=test not to affect the installed image — ago filed some bugs about that happening in the past but I don't think we've ever formally defined that. (In reply to Michał Górny from comment #3) > Note that per the PMS, USE=test is not in any way special, so a third-party > package could actually use it to control installing something. I think we should probably figure out some way of defining tests s.t. they're not really a USE flag given they don't affect the image without the hack we currently do. But I know that's not easy to do and people have tried in the past. > We may also > need to set an explicit ::gentoo policy to require USE=test not to affect > the installed image — ago filed some bugs about that happening in the past > but I don't think we've ever formally defined that. Yeah, we should definitely do this. (In reply to Sam James from comment #4) > (In reply to Michał Górny from comment #3) > > Note that per the PMS, USE=test is not in any way special, so a third-party > > package could actually use it to control installing something. > > I think we should probably figure out some way of defining tests s.t. > they're not really a USE flag given they don't affect the image without the > hack we currently do. But I know that's not easy to do and people have tried > in the past. Perhaps "runtime IUSE" proposal could fix this. It doesn't technically affect tests but it covers the generic use case of "USE flags that don't affect the image", so it would solve this immediate use, as well as bunch more. (In reply to Sam James from comment #4) > I think we should probably figure out some way of defining tests s.t. > they're not really a USE flag given they don't affect the image without the > hack we currently do. But I know that's not easy to do and people have tried > in the past. I think the elegant solution here is TDEPEND. Because fundamentally, "the hack we currently do" is to add additional build-time only dependencies that don't affect the image, through the only way that is currently available to conditionally define build-time only dependencies: a USE flag. (In reply to Michał Górny from comment #3) > Note that per the PMS, USE=test is not in any way special, so a third-party > package could actually use it to control installing something. We may also > need to set an explicit ::gentoo policy to require USE=test not to affect > the installed image — ago filed some bugs about that happening in the past > but I don't think we've ever formally defined that. There are certainly cases that do this -- they are bizarre cases, but they do exist... I mentioned it above as an advantage of having TDEPEND to describe things that do NOT affect the image, while allowing USE=test to still be used by such bizarre packages. Aside: the classic example of this sort of thing, which I shall never forgive or forget, is the haskell ecosystem. Merely passing a flag like --enable-tests somehow results in the haskell build system deciding to modify the soname (and of course they use libXXX-{hash}.so because why not). This actually isn't as good of an example as it could be, because my understanding is that basically anything will change that soname hash. Editing a comment line in a source file, adding dependencies, passing a completely unrelated configure argument, the phase of the moon... end result is that you should basically consider it a miracle if haskell libraries compile the same soname twice in a row. Unsurprisingly, it's an absolute disaster for bootstrapping a binary distro... (In reply to Eli Schwartz from comment #6) > (In reply to Sam James from comment #4) > > I think we should probably figure out some way of defining tests s.t. > > they're not really a USE flag given they don't affect the image without the > > hack we currently do. But I know that's not easy to do and people have tried > > in the past. > > I think the elegant solution here is TDEPEND. > > Because fundamentally, "the hack we currently do" is to add additional > build-time only dependencies that don't affect the image, through the only > way that is currently available to conditionally define build-time only > dependencies: a USE flag. > You are right. It also cleanly solves a bunch of the other bugs we have with weird stickiness wrt --usepkg and test deps because we can just treat it like BDEPEND and carve it out as its own option. It's a bit unfortunate in that it doesn't solve some of the other pet peeves I have with it but whatever, I don't actually have any concrete ideas on how to solve that so it is what it is. Perfect enemy of good etc etc. > > (In reply to Michał Górny from comment #3) > [...] > Aside: the classic example of this sort of thing, which I shall never > forgive or forget, is the haskell ecosystem. Merely passing a flag like > --enable-tests somehow results in the haskell build system deciding to > modify the soname (and of course they use libXXX-{hash}.so because why not). > [...] I forgot this is a problem in the wild right now with dev-haskell/* indeed... (In reply to Eli Schwartz from comment #0) > A third possibility is modifying PMS and adding a new T(B)?DEPEND field, > containing test dependencies. This is how other package managers which have > long dealt with binaries usually do things. Other package managers don't have USE flags. AFAICS everything that can be done with a new dependency type can also be done with BDEPEND="test? ( ... )". The price to pay may be that the test flag is somewhat special (which it already is, in practice), but that's still less intrusive than adding yet another dependency type. Also, I don't see how the dependency type would solve the binpkgs problem. A quick scan of the tree shows "use test", "use_enable test" and "use_with test" all over the place, and these won't go away. Not to mention that sometimes there's more than one USE flag that affects tests (USE=test-rust is one example). (In reply to Ulrich Müller from comment #8) > (In reply to Eli Schwartz from comment #0) > > A third possibility is modifying PMS and adding a new T(B)?DEPEND field, > > containing test dependencies. This is how other package managers which have > > long dealt with binaries usually do things. > > Other package managers don't have USE flags. > > AFAICS everything that can be done with a new dependency type can also be > done with BDEPEND="test? ( ... )". The price to pay may be that the test > flag is somewhat special (which it already is, in practice), but that's > still less intrusive than adding yet another dependency type. > > Also, I don't see how the dependency type would solve the binpkgs problem. A > quick scan of the tree shows "use test", "use_enable test" and "use_with > test" all over the place, and these won't go away. Yes, we've already talked about how we're not interested in where the image changes. The point is to not have to re-emerge something simply to get its test dependencies installed. That's where TDEPEND comes in - being able to tell the PM to satisfy a dependency class without it having to "fake satisfy" a USE flag without actually enabling it. (In reply to Sam James from comment #10) > (In reply to Ulrich Müller from comment #8) > > Also, I don't see how the dependency type would solve the binpkgs problem. A > > quick scan of the tree shows "use test", "use_enable test" and "use_with > > test" all over the place, and these won't go away. > > Yes, we've already talked about how we're not interested in where the image > changes. We're talking about packages that need test parts explicitly enabled at configure time but without affecting the installed image. (In reply to Michał Górny from comment #11) > We're talking about packages that need test parts explicitly enabled at > configure time but without affecting the installed image. At this point, I'm not suggesting getting rid of USE=test though, just having some way to get the dependencies it controls easily. (In reply to Ulrich Müller from comment #8) > (In reply to Eli Schwartz from comment #0) > > A third possibility is modifying PMS and adding a new T(B)?DEPEND field, > > containing test dependencies. This is how other package managers which have > > long dealt with binaries usually do things. > > Other package managers don't have USE flags. I am not sure why you're making a big point about this, as the topic of USE flags doesn't IMO tie in to the binary-ness that I'm discussing here, except accidentally. But, if you're going to mention it then it feels apropos to be on the same page about what package managers do and don't do. Specifically, some do have USE flags, some don't. (Whether they call them "USE flags" or not is irrelevant). Here are some examples: Void Linux has both checkdepends (TDEPEND) and build_options (USE flags): - https://github.com/void-linux/void-packages/blob/master/Manual.md#build-options - https://github.com/void-linux/void-packages/blob/master/Manual.md#optional-variables pkgsrc has both TEST_DEPENDS (TDEPEND) and PKG_OPTIONS (USE flags): - https://www.netbsd.org/docs/pkgsrc/pkgsrc.html#selecting-build-options - https://www.netbsd.org/docs/pkgsrc/pkgsrc.html#dependencies rpm has bcond (USE flags) too, but not checkdepends, so they are in the same situation as portage I guess: - https://rpm-software-management.github.io/rpm/manual/conditionalbuilds.html > AFAICS everything that can be done with a new dependency type can also be > done with BDEPEND="test? ( ... )". The price to pay may be that the test > flag is somewhat special (which it already is, in practice), but that's > still less intrusive than adding yet another dependency type. I'm not sure what you mean by intrusive, unless you mean "level of work involved in engineering the implementation". I'm not saying I demand a result tomorrow ;) I'm opening a ticket to discuss what might be nice for the future. And being able to have a technically correct semantic model of what's going on is IMHO a pretty useful outcome. That was... kind of my whole angle in this all along. Per definition, what I'm arguing *against* is having to pay prices such as treating a flag as special. The fact that it already is "in practice" is part and parcel of what I'm proposing to *solve*. > Also, I don't see how the dependency type would solve the binpkgs problem. A > quick scan of the tree shows "use test", "use_enable test" and "use_with > test" all over the place, and these won't go away. I'm not trying to solve all the world's problems in one shot. But I believe we can solve many problems, and thereby reduce the problem space immensely. Many packages do not care about USE=test at all, except inasmuch as this causes the dependencies of a separate manually-invoked testsuite runner to be available when the test phase is run. It would be good to be able to correctly model these packages and assert with confidence that even absent a USE=test special case, the package is "literally identical". Other packages may still need to know at configure time that tests were requested, but at least one should be able to consult the list of test dependencies without fairly confusing state-sensitive indirection. For example: https://packages.gentoo.org/packages/app-shells/bash-completion/dependencies Question: Why does bash-completion BDEPEND on python, pytest, and pexpect? Answer: It doesn't. There are other reasons Sam has discussed with me, to want to be able to semantically model test dependencies without arbitrarily tying it to the activation of a USE flag, such as IIRC making a hash of -o --with-test-deps when USE=test is masked? (In reply to Eli Schwartz from comment #13) > For example: > https://packages.gentoo.org/packages/app-shells/bash-completion/dependencies > > Question: Why does bash-completion BDEPEND on python, pytest, and pexpect? > > Answer: It doesn't. p.g.o doesn't currently show use-conditional dependencies. This is a restriction of that site, but not a problem of dependencies or use flags in general. (In reply to Ulrich Müller from comment #8) > (In reply to Eli Schwartz from comment #0) > > A third possibility is modifying PMS and adding a new T(B)?DEPEND field, > > containing test dependencies. This is how other package managers which have > > long dealt with binaries usually do things. > > Other package managers don't have USE flags. > > AFAICS everything that can be done with a new dependency type can also be > done with BDEPEND="test? ( ... )". The price to pay may be that the test > flag is somewhat special (which it already is, in practice), but that's > still less intrusive than adding yet another dependency type. I just got irritated at the impracticality of testing again today, and while searching to find this bug I also came across the comment at https://bugs.gentoo.org/541938#c2 (In reply to Ulrich Müller from comment #2) > *shrug* This is just another workaround which doesn't address the real > issue. There shouldn't be a test USE flag, in the first place. Also the flag > won't get less magical, unless Portage is changed not to enable it depending > on FEATURES. > > What we really need is a TDEPEND variable (or a generalised DEPENDENCIES > variable) for expressing src_test dependencies. Sounds like a fantastic idea, why don't we do exactly th--- |