PMS says in section 12.1.1 "USE and IUSE Handling": IUSE_EFFECTIVE contains the following values: * All values in the calculated IUSE value. * All possible values for the ARCH variable. * All legal use flag names whose name starts with the lowercase equivalent of any value in the profile USE_EXPAND variable followed by an underscore. Whereas Portage does the following (in portage/package/ebuild/config.py): Some flags are considered to be implicit members of IUSE: * Flags derived from ARCH * Flags derived from USE_EXPAND_HIDDEN variables * Masked flags, such as those from {,package}use.mask * Forced flags, such as those from {,package}use.force * build and bootstrap flags used by bootstrap.sh Since masked and forced USE flags are also widely used throughout the tree without adding them to IUSE (e.g. the "prefix" flag), I think that the spec should mention them.
This has changed considerably between Portage versions, so we can't just document what current Portage does. There was supposed to be a proper fix for this in EAPI 4, which would make implicit flags be listed explicitly in profiles, but it was dropped due to lack of Portage implementation.
(In reply to comment #1) > This has changed considerably between Portage versions, Before filing this bug, I verified that Portage behaviour hasn't changed since 2007, when these checks have been introduced. > so we can't just document what current Portage does. So what do you suggest that we should document? > There was supposed to be a proper fix for this in EAPI 4, which would make > implicit flags be listed explicitly in profiles, but it was dropped due to > lack of Portage implementation. Right, but we should still have a proper spec for lower EAPIs. Or, in your own words: "It's necessary to avoid making large parts of the tree non-compliant." (see bug 176467 comment #4).
It's not that simple. Originally, Portage didn't check at all (and it's still not strict or universal in its checks). The wording in PMS is a nasty compromise from before the checks, based upon what was being done in the tree. Portage's validation came along later. The prefix thing is a recent change -- ebuilds *should* be listing it in IUSE, and until recently people got this right. The original plan was to have EAPI 3 ebuilds list prefix in IUSE, and in EAPI 4 to mark prefix as an implicit flag instead. But since implicit flags got dropped from EAPI 4 due to lack of Portage support, people still need to put it in IUSE. The fix here is to remind people that they need to list prefix explicitly until Portage implements the remainder of the approved-but-dropped-eapi-4 list as EAPI 5.
(In reply to comment #2) > Before filing this bug, I verified that Portage behaviour hasn't changed since > 2007, when these checks have been introduced. As far as I can see this was first added in commit bbff2d8 on 2007-12-18. The rules currently described in PMS were added to the document in da7cbe5 on 2006-12-31, according to what I believe was already considered QA policy at the time (correct me if I'm wrong on that), so Portage really shouldn't have deviated when it added the filtering.
(In reply to comment #3) Thanks for the explanation. Do you happen to have a pointer to a previous discussion about this? > The fix here is to remind people that they need to list prefix explicitly TBH, I don't know how many flags are affected. If prefix is the only one, then it might be feasible to fix its usage in the tree. But I fear that there are more of them... > until Portage implements the remainder of the approved-but-dropped-eapi-4 > list as EAPI 5. @Portage team: Is there anybody working on this?
Paludis checks and moans fairly noisily about these, so we'd probably know if there were lots of other abusers. I'm guessing this bug came about as a result of someone asking about why they'd started getting 'prefix' warnings a few days back... Come to think of it, LINGUAS is abused all over the place in ways that break the Portage rules you've listed (but not the version in PMS, which was written with those abuses in mind), but Portage doesn't check very well so you might not notice.
(In reply to comment #5) > > until Portage implements the remainder of the approved-but-dropped-eapi-4 > > list as EAPI 5. > > @Portage team: Is there anybody working on this? We haven't started any EAPI 5 work yet. EAPI 4 portage was stabilized less than 2 months ago, so it's a bit early to be moving on to another EAPI. (In reply to comment #6) > Come to think of it, LINGUAS is abused all over the place in ways that break > the Portage rules you've listed (but not the version in PMS, which was written > with those abuses in mind), but Portage doesn't check very well so you might > not notice. We have an IUSE.missing check in repoman since portage-2.1.9. There's also an older check in the use() ebuild function that triggers eqawarn logs.
(In reply to comment #7) > > @Portage team: Is there anybody working on this? > > We haven't started any EAPI 5 work yet. EAPI 4 portage was stabilized less than > 2 months ago, so it's a bit early to be moving on to another EAPI. We're talking about stuff that was supposed to be in EAPI 4, not new features. > (In reply to comment #6) > We have an IUSE.missing check in repoman since portage-2.1.9. There's also an > older check in the use() ebuild function that triggers eqawarn logs. Do you still pass in a complete LINGUAS variable that includes things not listed in IUSE(_EFFECTIVE)? As in, if a user has LINGUAS="a b c" in make.conf, and an ebuild has IUSE="linguas_a linguas_a", does 'c' still end up in $LINGUAS? What about if IUSE has no mention at all of linguas?
(In reply to comment #8) > Do you still pass in a complete LINGUAS variable that includes things not > listed in IUSE(_EFFECTIVE)? As in, if a user has LINGUAS="a b c" in make.conf, > and an ebuild has IUSE="linguas_a linguas_a", does 'c' still end up in > $LINGUAS? The ebuild would get LINGUAS="a b" in this case, because the LINGUAS variable is regenerated to be consistent with the filtered USE settings (bug 142909). > What about if IUSE has no mention at all of linguas? In this case, unfiltered LINGUAS is allowed to pass into the ebuild environment. The reasoning is that ebuild itself likely doesn't use the variable in this case, but the underlying build system can use it to control installation of mo files. It's also important to consider that unset LINGUAS means that all available mo files are supposed to be installed (bug 148702).
Ok, so Ulrich's description in comment #0 isn't entirely accurate for USE_EXPAND. As I recall, something related to this is why there's that horrible weasel wording for USE_EXPAND in what's in PMS. We hit this with Paludis because Paludis configs don't have any 'global' settings (merely values that happen to be enabled for */*, or for some combination of wildcards that happens to match everything)...
On the one hand, I agree that it's sort of arbitrary and unsystematic to make an exception for masked and forced flags. On the other hand, I've talked to some people and found that they have very strong opinions about the matter. I've got answers like "prefix is exclusively set from profiles, therefore it's special like ARCH flags" or "users can't toggle the flag so we don't want it being exposed to them". So looks like we're between a rock and a hard place here. It's unlikely that people would add the prefix flag to ebuilds' IUSE. (In reply to comment #10) > Ok, so Ulrich's description in comment #0 isn't entirely accurate for > USE_EXPAND. Not my wording. ;) It's a verbatim copy of a comment in Portage sources.
What happens next week when someone comes along and makes a profile where prefix is optional?
(In reply to comment #12) > What happens next week when someone comes along and makes a profile where > prefix is optional? Are you kidding? If not, why would you want that? Before I added profiles/arch/base to mask all the ARCH flags, we occasionally had reports of people having the x86 flag enabled on amd64 profiles. The prefix flag is similar in that it only makes sense to have if forced or masked, no?
(In reply to comment #13) > (In reply to comment #12) > > What happens next week when someone comes along and makes a profile where > > prefix is optional? > > Are you kidding? If not, why would you want that? I don't presume to assert that the prefix people won't at some point decide that it's possible to do partial prefix installs...
then still - it's undesirable to add "prefix" to IUSE in many places - the user shouldn't control it, but the PM
Why is it undesirable? Besides, undesirable or not, you're violating QA and not following the spec. Implicit IUSE wasn't included in EAPI 4, so you shouldn't be pretending that it was.
Ok, define some means to retrieve instance-specific configuration values (e.g. based on profile), by perhaps a PM provided function get-setup-var(), and make it available in EAPIs 3.1 and 4.1. Define it in such a way that e.g. DEPEND and SRC_URI conditionals based on the available configuration values can be defined. In short, the means must be able to be used in all places where USE-flags can be used now. When such system is in place, all affected ebuilds/eclasses can be fixed to use it. I presume the system will be flexible, such that the current "prefix" flag can be replaced by two or more specific flags, to further improve the affected ebuilds. I'd like such approach over simply adding "prefix" to the list "build" and "bootstrap" also belong to.
We had that in the original EAPI 4 that the Council provisionally approved (at the request of the Prefix team, as I recall). However, Portage didn't implement it, so it was dropped from the EAPI 4 the Council eventually approved. See Comment #7 for Portage progress on implementing the remainder of the features.
I don't recall being involved with any EAPI other than EAPI-3. The "prefix" USE-flag dates back to the EAPI-3 age, when it was commonly accepted that adding it to use.force was ok.
It wasn't OK in EAPI 3. I think you've become confused because of all the mucking around with EAPI orders and the dropped features. Originally, EAPI 3 was going to be what is now EAPI 4 plus a few more features, including IUSE_IMPLICIT. Thus, we had this: http://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg35027.html However, EAPI 3 became EAPI 4, and the final approved EAPI 3 with the prefix thing in it did *not* include implicit IUSE. Thus, like 'build', PMS explicitly says that including 'prefix' in IUSE with EAPI 3 is mandatory. The plan then was to allow you to have it implicitly in EAPI 4, and the provisionally approved EAPI 4 included IUSE_IMPLICIT. However, IUSE_IMPLICIT was dropped from EAPI 4 due to Portage not supporting it. Thus, for EAPI 4, you also still need to explicitly include 'prefix' in IUSE. So far as I know, IUSE_IMPLICIT is still intended for EAPI 5. Thus, once Portage supports the remaining features that got left out of EAPI 4, *then* you'll be able to avoid listing 'prefix' in IUSE. Until that happens, what PMS says goes (and Portage should be updated to follow the spec).
You could also try to accept some people think/thought differently, and then try to improve the situation in upcoming EAPIs. As I indicated before, I'm open to suggestions in that area, and not necessarily on a mission to thwart your quest for soundness.
(In reply to comment #20) > Until that happens, what PMS > says goes (and Portage should be updated to follow the spec). The portage behavior is based on feedback from ebuild developers. As evidenced by all the bugs marked as duplicates of bug 70648, ebuild developers overwhelmingly tend to resist the idea of adding flags to IUSE if those flags should not be adjusted by users. I suspect that if the council were to vote on it, they would probably decide in favor of the current portage behavior.
(In reply to comment #21) > You could also try to accept some people think/thought differently, and then > try to improve the situation in upcoming EAPIs. Yes, we're improving it by IUSE_IMPLICIT, which you'll have whenever Portage gets around to implementing the remainder of the things that were originally in EAPI 4. (In reply to comment #22) > The portage behavior is based on feedback from ebuild developers. Well it shouldn't be. It should be based upon what the Council voted on. > I suspect that if the council were to vote on > it, they would probably decide in favor of the current portage behavior. They already voted on it. Unfortunately, things got screwed up by Portage not implementing what the Council voted on. Why not just finish off the things that got left out of EAPI 4?
> (In reply to comment #22) > > The portage behavior is based on feedback from ebuild developers. > > Well it shouldn't be. It should be based upon what the Council voted on. Well, perhaps the council would like to reconsider this point. Councils do overlook things sometimes. > > I suspect that if the council were to vote on > > it, they would probably decide in favor of the current portage behavior. > > They already voted on it. Unfortunately, things got screwed up by Portage not > implementing what the Council voted on. Fair enough, but now you're talking about EAPI 5. We need to decide how to handle the existing EAPIs. > Why not just finish off the things that > got left out of EAPI 4? We will, but let's focus on the existing EAPIs for now.
For existing EAPIs we should follow the spec, especially since the spec was written and approved before Portage added its slightly-different validation support. As for the rest, is it too much to ask that Portage quickly implements at least the USE stuff that was dropped EAPI 4? Doing a quick EAPI 5 with that (even if Portage still can't do the other dropped features) would keep people happy, and would also mean that use dependency defaults wouldn't be crippled...
(In reply to comment #25) > For existing EAPIs we should follow the spec, especially since the spec was > written and approved before Portage added its slightly-different validation > support. Given my experience with bug 70648, I think we need more developer feedback before we change portage behavior again. > As for the rest, is it too much to ask that Portage quickly implements at least > the USE stuff that was dropped EAPI 4? No it's not too much, but I think the more important question at hand is how we are going to handle the existing EAPIs as long as they are used. > Doing a quick EAPI 5 with that (even if > Portage still can't do the other dropped features) would keep people happy, and > would also mean that use dependency defaults wouldn't be crippled... Again, the handling of existing EAPIs is quite important for the foreseeable future, regardless of how happy EAPI 5 will make people.
We have better than developer feedback. We have a Council decision, and we have a way of giving developers exactly what they're after in a way that doesn't break use dependencies etc in EAPI 5. As for existing EAPIs, it's only an issue if it's going to take Portage a long time to implement the proper solution.
We seem to have a disagreement here. Feel free to submit it to the council.
Before we go back to the Council with this (despite them having already decided), it would help if we know how long it's going to take for Portage to implement the proper, carefully worked out solution that they'd already approved. Also, we can't just throw in forced/masked flags into the existing PMS definition, since that breaks use dependency defaults. If we retroactively make it legal to omit forced and masked flags, someone needs to work out a) a definition for "everywhere forced", and b) an even longer list of things which are illegal to use in use dependencies.
(In reply to comment #29) > Before we go back to the Council with this (despite them having already > decided), it would help if we know how long it's going to take for Portage to > implement the proper, carefully worked out solution that they'd already > approved. I guess 1 week is probably a reasonable estimate, given motivation. If the council approves an EAPI 5 spec, and that should provide the necessary motivation.
Well, I don't imagine we'd have much difficulty getting the Council to approve EAPI 5 as being EAPI 4 plus the USE stuff that was originally in EAPI 4... Ulrich?
(In reply to comment #30) > I guess 1 week is probably a reasonable estimate, given motivation. If the > council approves an EAPI 5 spec, and that should provide the necessary > motivation. I've no doubt that implicit IUSE (basically, revert commit <http://git.overlays.gentoo.org/gitweb/?p=proj/pms.git;a=commit;h=9d2b8ee57bf3be941cfdfe13650952d91b9edfdc>) will be part of EAPI 5. (In reply to comment #31) > Well, I don't imagine we'd have much difficulty getting the Council to > approve EAPI 5 as being EAPI 4 plus the USE stuff that was originally in > EAPI 4... Ulrich? The question is if they would deem this single feature important enough to justify an EAPI bump. It would certainly help if all PM authors agreed that this is the way to go. And do I understand it right: Profiles would simply define some additional variables (like IUSE_IMPLICIT and USE_EXPAND_UNPREFIXED) but wouldn't need to be bumped to EAPI 5?
(In reply to comment #32) > The question is if they would deem this single feature important enough to > justify an EAPI bump. It would certainly help if all PM authors agreed that > this is the way to go. Well, without it, we've got a long and messy process of figuring out how to loosen up the rules enough to allow the abuses people seem to want without breaking use dependencies etc... If we just get a fast EAPI 5 with the use stuff in there, listing prefix in IUSE for EAPIs 3 and 4 as PMS requires suddenly isn't a big deal any more, since there will be very few packages locked to 3 or 4. > And do I understand it right: Profiles would simply define some additional > variables (like IUSE_IMPLICIT and USE_EXPAND_UNPREFIXED) but wouldn't need to > be bumped to EAPI 5? Yup, that's the way it was designed. The only other real difference is that certain [use] and [use(+)] dependencies (mostly on USE_EXPANDed stuff) that are currently illegal (but not enforced by Portage) will become legal.
(In reply to comment #33) > > And do I understand it right: Profiles would simply define some additional > > variables (like IUSE_IMPLICIT and USE_EXPAND_UNPREFIXED) but wouldn't need to > > be bumped to EAPI 5? > > Yup, that's the way it was designed. The only other real difference is that > certain [use] and [use(+)] dependencies (mostly on USE_EXPANDed stuff) that are > currently illegal (but not enforced by Portage) will become legal. Maybe the EAPI 5 bump isn't really necessary if it's feasible to apply this retroactively to all EAPIs.
(In reply to comment #17) > Ok, define some means to retrieve instance-specific configuration values (e.g. > based on profile), by perhaps a PM provided function get-setup-var(), and make > it available in EAPIs 3.1 and 4.1. Define it in such a way that e.g. DEPEND > and SRC_URI conditionals based on the available configuration values can be > defined. In short, the means must be able to be used in all places where > USE-flags can be used now. The prefix flag could perhaps be moved to a new USE_EXPAND variable called PROFILE_FEATURES or something. The spec already allows for USE_EXPAND things to not be listed in IUSE, or alternatively the variable could be put in USE_EXPAND_HIDDEN and then it could be put in IUSE without showing up for users. This could also be used for things like multilib and selinux. I don't think there's been much complaining about listing those in IUSE, but if we have the new mechanism we may as well use it.
(In reply to David Leverton from comment #35) > The prefix flag could perhaps be moved to a new USE_EXPAND variable called > PROFILE_FEATURES or something. .. > This could also be used for things like multilib and selinux. A whitelist variable, profile_features="prefix multilib.." (or some other name) certainly appears to be the simplest method for an implementor to deal with pre-EAPI5 correctly, without giving spurious QA warnings and annoying groups of developers at a time. It's trivial to maintain, and could well be kept internal; after all it should only apply to very specific flags, afaict, without being dubious. And most of those EAPIs have been, or will be phased-out. Sometimes you have to be pragmatic, and support what people use. Or they'll use something else.
This is solved in EAPI 5, therefore closing.
*** Bug 507132 has been marked as a duplicate of this bug. ***
