Bugzilla Bug 133327

ebuilds should be able to simply look like:
IUSE_VIDEO_CARDS="ati nvidia"
IUSE="sdl"

and then portage would expand the values in IUSE_VIDEO_CARDS automatically and
stick it into IUSE

instead, we have to expand IUSE_VIDEO_CARDS ourselves and stick it into IUSE:
IUSE_VIDEO_CARDS="video_cards_ati video_cards_nvidia"
IUSE="sdl ${IUSE_VIDEO_CARDS}"

bugs--

This was gone over on the mailing lists. If you want support for
IUSE_VIDEO_CARDS, the _entire_ cache will need to be invalidated whenever
USE_EXPAND is modified. As well as that, a list of the USE_EXPAND that the
cache was built with will need to be tagged into each cache file causing a
large increase in size.

IOW, it's not possible while a cache is needed.

CANTFIX isnt really acceptable ... just because portage's current design is
unable to cope with this doesnt make the current method correct

at some point this will have to be addressed and portage will have to be less
stupid about it

so, someday when current design aspects suck less to handle this, we'll re-open

(In reply to comment #2)
> This was gone over on the mailing lists. If you want support for
> IUSE_VIDEO_CARDS, the _entire_ cache will need to be invalidated whenever
> USE_EXPAND is modified. As well as that, a list of the USE_EXPAND that the
> cache was built with will need to be tagged into each cache file causing a
> large increase in size.

Well, the cache would really only become invalid when a change in USE_EXPAND
triggers a change in IUSE, and in the given example that would only happen when
VIDEO_CARDS is added or removed from USE_EXPAND.  However, such "spontaneous"
changes in IUSE are actually irrelevant because IUSE really depends on
conditionals that exist in the ebuild/eclasses and those conditionals remain
constant when USE_EXPAND changes.  In our example, if VIDEO_CARDS were removed
from USE_EXPAND, portage would no longer perform automatic expansion of that
variable for the user, but conceivably, the user could manually expand the
variable to achieve the same effect as if it had never been removed.

(In reply to comment #2)
> IOW, it's not possible while a cache is needed.

After chatting with ferringb I'm now aware that some ebuilds may use the list
of user's USE flags to generate new conditionals on the fly, such that the
number of conditionals is not static and can shrink or grow (effectively
causing IUSE to change).  So, apparently we need some type of syntax to express
these "use groups" (or whatever they are) symbolically such that the IUSE cache
remains valid.  For example, an ebuild that uses all possible VIDEO_CARDS would
express that with IUSE="@video_cards", or if it only uses 2 video cards then it
could express it as IUSE="@video_cards(ati,nvidia)".

Changing the literal value of IUSE for USE_EXPAND variables doesn't make much
sense. At least, it makes just as much sense as changing IUSE for USE
variables. There are many cases where certain USE flags are ignored for a
package unless a certain other USE flag is enabled as well. Is there any real
difference between that and this "new" USE_EXPAND case?

Created an attachment (id=87667) [edit]
Automatically add IUSE_${envvar} to IUSE

I think this is essentially what Mike is asking for in this bug. However, there
is one important thing to note which is that additions to USE_EXPAND _must_ be
committed before any ebuilds that make use of the new var are committed.

(In reply to comment #8)
> additions to USE_EXPAND _must_ be committed before any ebuilds that make use of the new var are committed.
> 

USE_EXPAND can be a profile dependent incremental, so the generated metadata
would be profile specific, which is not allowed.  Perhaps we could simply grab
all IUSE_* vars from the env?

(In reply to comment #9)
> (In reply to comment #8)
> > additions to USE_EXPAND _must_ be committed before any ebuilds that make use of the new var are committed.
> 
> USE_EXPAND can be a profile dependent incremental, so the generated metadata
> would be profile specific, which is not allowed.  Perhaps we could simply grab
> all IUSE_* vars from the env?

While it is theoretically possible for profiles to add to USE_EXPAND (and to
reset it) none of the profiles should be doing this. USE_EXPAND only moved to
base/make.defaults from make.globals so that vars could be added without
requiring a newer version of portage.

Having said that, grabbing all IUSE_* from the environment would definitely get
around the guaranteed-to-be-ignored caching rule/issue. I've not sure on the
best way of doing that in bash (MY_IUSE_VARS="$(set | grep '^IUSE_')"?) so if
someone could fix that (and hopefully get rid of awk's tolower() usage) we
might have a sane fix.

Created an attachment (id=87706) [edit]
Expands and adds all IUSE_* variables to IUSE

IUSE_* is found with $(set | grep -E '^IUSE_' | sed 's/=.*//'). Is there a
better way to do this?

The previous `awk '{print tolower($0)}'` has been replaced with `tr '[A-Z]'
'[a-z]'`.

If this is to be implemented, now is the best time. There's no support at all
in 2.0.x and changing it after 2.1.x goes stable will be somewhat difficult.
Ebuild authors would need to be doing something like: 

IUSE_VIDEO_CARDS="ati nvidia"
OLD_IUSE_VIDEO_CARDS="video_cards_ati video_cards_nvidia"
IUSE="${OLD_IUSE_VIDEO_CARDS} sdl"

which is worse than what needs to be done now. That said, IUSE is currently
unique_array()'ed so the above wouldn't cause any display issues.

Mike, you're the resident bash expert. Thoughts on the patch?
Everyone, doing it this way would mean that all of IUSE_* would be reserved for
USE_EXPAND.. Anyone against that?

(In reply to comment #12)
> Mike, you're the resident bash expert. Thoughts on the patch?
Jason,
Just to let you know he is in China for a few weeks. Probably without any 
form of internet till he returns.

Created an attachment (id=87827) [edit]
expand IUSE_* vars only when necessary

This patch does effectively the same thing bug it allows backward compatibility
with existing old-style IUSE_* variables.  I think we should go ahead and do
this then announce it on the gentoo-dev mailing list.

I was just thinking, we can avoid clutter in the vdb IUSE entry if you uniqify
them like this:

IUSE="$(for x in ${IUSE}; do echo ${x}; done | LANG=C sort -u)"

And, we should grep the tree for strange uses of IUSE_* variables before we
deploy this...

Thinking a little more, this feature kind of useless as long as 2.0 is in use
and doesn't support it.  We can distribute the metadata via rsync but that
won't help cvs users.  Maybe it's better just to implement "use groups" or
something similar.

Your patch looks fine to me - glad the set | grep is gone. :)

Not sure what you mean about rsync/cvs/uselessness/use groups though.. Can you
expand on your thoughts a little please?

(In reply to comment #17)
> Not sure what you mean about rsync/cvs/uselessness/use groups though..

Well, I suppose it doesn't matter too much if users running old versions of
portage don't see the new IUSE_* flags in metadata that they generate locally
(though they will be able to see them in metadata generated for rsync
distribution, as soon as I update osprey).  So, I suppose it's useful for
ebuild maintainers right now if they want to drop the old-style manual
expansions.

However, this same functionality could be represented by a new "use groups"
syntax for IUSE.  A possible alternative interim solution to the proposed patch
would be to provide an eclass function (in eutils for example) that allows
ebuilds to perform semi-automatic expansion.  An eclass implementation would
could work with all existing versions of portage right away.

Created an attachment (id=87933) [edit]
add an iuse_expand() function to eutils.eclass

An ebuild that wishes to automatically expand IUSE_* variables into IUSE need
only inherit eutils, define the IUSE_* variables, then call the iuse_expand()
function.    With this approach, there is no need to detect already expanded
USE flags or make the list unique because ebuilds written to use this function
will define the variables such that it is unnecessary.

The class function has many considerable advantages over direct integration
into portage:

1) It works with all existing versions of portage immediately (including third
party package managers such as pkgcore and paludis).
2) There is no global special case for IUSE_* variables, so there is no need to
grep the tree for potential strange uses of IUSE_* varables.  Only the
maintainers of ebuilds that use the iuse_expand() function need to be concerned
about the IUSE_* variables.
3) Similar functionality will likely be provided in the future via "use groups"
or something similar.

Yep, I'm in favour of the eutils extension. 

the problem with that is we get global scope calls to `tr` in the ebuilds now
... wont that cause a QA warning ?

You only get those QA warnings for stuff in QA_INTERCEPTORS and luckily "tr"
isn't in there now.

I found a bash based version at
http://www.codecomments.com/archive287-2005-2-392096.html. Apparently, it would
suffice for short strings but some testing would be probably be warranted.

On my system, for 5000 runs echoing to tr and assigning the result takes about
14.6s. With the function listed in the previous URL, I get the following
results.

 1 char    4.7s
 5 chars   6.6s
10 chars   9.5s
15 chars  12.2s

Looking at the function, it could be optimized a fair bit for our specific case
though.

*** Bug 156565 has been marked as a duplicate of this bug. ***

So what's the status on this?

*** Bug 167252 has been marked as a duplicate of this bug. ***

*** Bug 170748 has been marked as a duplicate of this bug. ***

*** Bug 172598 has been marked as a duplicate of this bug. ***

*** Bug 172601 has been marked as a duplicate of this bug. ***

Anyone going for this?

*** Bug 136953 has been marked as a duplicate of this bug. ***

*** Bug 136953 has been marked as a duplicate of this bug. ***