Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 133327

Summary: add USE_EXPAND flags to IUSE via IUSE_* variables
Product: Portage Development Reporter: SpanKY <vapier>
Component: Enhancement/Feature RequestsAssignee: PMS/EAPI <pms>
Status: RESOLVED OBSOLETE    
Severity: normal CC: avuton, boltomli, bugzilla-gentoo, dkarasik, esigra, hanno, kensington, masterdriverz, matrixhax0r, mattepiu, mike, mr_bones_, pacho, seventhguardian, write2David
Priority: High    
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 174380, 228907    
Attachments: Automatically add IUSE_${envvar} to IUSE
Expands and adds all IUSE_* variables to IUSE
expand IUSE_* vars only when necessary
add an iuse_expand() function to eutils.eclass

Description SpanKY gentoo-dev 2006-05-14 13:46:57 UTC
ebuilds should be able to simply look like:
IUSE_VIDEO_CARDS="ati nvidia"
IUSE="sdl"

and then portage would expand the values in IUSE_VIDEO_CARDS automatically and stick it into IUSE

instead, we have to expand IUSE_VIDEO_CARDS ourselves and stick it into IUSE:
IUSE_VIDEO_CARDS="video_cards_ati video_cards_nvidia"
IUSE="sdl ${IUSE_VIDEO_CARDS}"
Comment 1 Alec Warner archtester Gentoo Infrastructure gentoo-dev Security 2006-05-14 14:17:57 UTC
bugs--
Comment 2 Jason Stubbs (RETIRED) gentoo-dev 2006-05-15 06:36:36 UTC
This was gone over on the mailing lists. If you want support for IUSE_VIDEO_CARDS, the _entire_ cache will need to be invalidated whenever USE_EXPAND is modified. As well as that, a list of the USE_EXPAND that the cache was built with will need to be tagged into each cache file causing a large increase in size.

IOW, it's not possible while a cache is needed.
Comment 3 SpanKY gentoo-dev 2006-05-16 23:14:17 UTC
CANTFIX isnt really acceptable ... just because portage's current design is unable to cope with this doesnt make the current method correct

at some point this will have to be addressed and portage will have to be less stupid about it
Comment 4 SpanKY gentoo-dev 2006-05-16 23:14:48 UTC
so, someday when current design aspects suck less to handle this, we'll re-open
Comment 5 Zac Medico gentoo-dev 2006-05-17 00:31:53 UTC
(In reply to comment #2)
> This was gone over on the mailing lists. If you want support for
> IUSE_VIDEO_CARDS, the _entire_ cache will need to be invalidated whenever
> USE_EXPAND is modified. As well as that, a list of the USE_EXPAND that the
> cache was built with will need to be tagged into each cache file causing a
> large increase in size.

Well, the cache would really only become invalid when a change in USE_EXPAND triggers a change in IUSE, and in the given example that would only happen when VIDEO_CARDS is added or removed from USE_EXPAND.  However, such "spontaneous" changes in IUSE are actually irrelevant because IUSE really depends on conditionals that exist in the ebuild/eclasses and those conditionals remain constant when USE_EXPAND changes.  In our example, if VIDEO_CARDS were removed from USE_EXPAND, portage would no longer perform automatic expansion of that variable for the user, but conceivably, the user could manually expand the variable to achieve the same effect as if it had never been removed.

Comment 6 Zac Medico gentoo-dev 2006-05-26 04:09:20 UTC
(In reply to comment #2)
> IOW, it's not possible while a cache is needed.

After chatting with ferringb I'm now aware that some ebuilds may use the list of user's USE flags to generate new conditionals on the fly, such that the number of conditionals is not static and can shrink or grow (effectively causing IUSE to change).  So, apparently we need some type of syntax to express these "use groups" (or whatever they are) symbolically such that the IUSE cache remains valid.  For example, an ebuild that uses all possible VIDEO_CARDS would express that with IUSE="@video_cards", or if it only uses 2 video cards then it could express it as IUSE="@video_cards(ati,nvidia)".
Comment 7 Jason Stubbs (RETIRED) gentoo-dev 2006-05-27 09:04:37 UTC
Changing the literal value of IUSE for USE_EXPAND variables doesn't make much sense. At least, it makes just as much sense as changing IUSE for USE variables. There are many cases where certain USE flags are ignored for a package unless a certain other USE flag is enabled as well. Is there any real difference between that and this "new" USE_EXPAND case?
Comment 8 Jason Stubbs (RETIRED) gentoo-dev 2006-05-27 09:07:18 UTC
Created attachment 87667 [details, diff]
Automatically add IUSE_${envvar} to IUSE

I think this is essentially what Mike is asking for in this bug. However, there is one important thing to note which is that additions to USE_EXPAND _must_ be committed before any ebuilds that make use of the new var are committed.
Comment 9 Zac Medico gentoo-dev 2006-05-27 13:24:50 UTC
(In reply to comment #8)
> additions to USE_EXPAND _must_ be committed before any ebuilds that make use of the new var are committed.
> 

USE_EXPAND can be a profile dependent incremental, so the generated metadata would be profile specific, which is not allowed.  Perhaps we could simply grab all IUSE_* vars from the env?

Comment 10 Jason Stubbs (RETIRED) gentoo-dev 2006-05-27 22:26:16 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > additions to USE_EXPAND _must_ be committed before any ebuilds that make use of the new var are committed.
> 
> USE_EXPAND can be a profile dependent incremental, so the generated metadata
> would be profile specific, which is not allowed.  Perhaps we could simply grab
> all IUSE_* vars from the env?

While it is theoretically possible for profiles to add to USE_EXPAND (and to reset it) none of the profiles should be doing this. USE_EXPAND only moved to base/make.defaults from make.globals so that vars could be added without requiring a newer version of portage.

Having said that, grabbing all IUSE_* from the environment would definitely get around the guaranteed-to-be-ignored caching rule/issue. I've not sure on the best way of doing that in bash (MY_IUSE_VARS="$(set | grep '^IUSE_')"?) so if someone could fix that (and hopefully get rid of awk's tolower() usage) we might have a sane fix.
Comment 11 Jason Stubbs (RETIRED) gentoo-dev 2006-05-28 00:48:14 UTC
Created attachment 87706 [details, diff]
Expands and adds all IUSE_* variables to IUSE

IUSE_* is found with $(set | grep -E '^IUSE_' | sed 's/=.*//'). Is there a better way to do this?

The previous `awk '{print tolower($0)}'` has been replaced with `tr '[A-Z]' '[a-z]'`.
Comment 12 Jason Stubbs (RETIRED) gentoo-dev 2006-05-29 10:56:31 UTC
If this is to be implemented, now is the best time. There's no support at all in 2.0.x and changing it after 2.1.x goes stable will be somewhat difficult. Ebuild authors would need to be doing something like: 

IUSE_VIDEO_CARDS="ati nvidia"
OLD_IUSE_VIDEO_CARDS="video_cards_ati video_cards_nvidia"
IUSE="${OLD_IUSE_VIDEO_CARDS} sdl"

which is worse than what needs to be done now. That said, IUSE is currently unique_array()'ed so the above wouldn't cause any display issues.

Mike, you're the resident bash expert. Thoughts on the patch?
Everyone, doing it this way would mean that all of IUSE_* would be reserved for USE_EXPAND.. Anyone against that?
Comment 13 solar (RETIRED) gentoo-dev 2006-05-29 12:02:10 UTC
(In reply to comment #12)
> Mike, you're the resident bash expert. Thoughts on the patch?
Jason,
Just to let you know he is in China for a few weeks. Probably without any 
form of internet till he returns.

Comment 14 Zac Medico gentoo-dev 2006-05-29 14:56:28 UTC
Created attachment 87827 [details, diff]
expand IUSE_* vars only when necessary

This patch does effectively the same thing bug it allows backward compatibility with existing old-style IUSE_* variables.  I think we should go ahead and do this then announce it on the gentoo-dev mailing list.
Comment 15 Zac Medico gentoo-dev 2006-05-29 15:31:13 UTC
I was just thinking, we can avoid clutter in the vdb IUSE entry if you uniqify them like this:

IUSE="$(for x in ${IUSE}; do echo ${x}; done | LANG=C sort -u)"

And, we should grep the tree for strange uses of IUSE_* variables before we deploy this...
Comment 16 Zac Medico gentoo-dev 2006-05-29 15:50:12 UTC
Thinking a little more, this feature kind of useless as long as 2.0 is in use and doesn't support it.  We can distribute the metadata via rsync but that won't help cvs users.  Maybe it's better just to implement "use groups" or something similar.
Comment 17 Jason Stubbs (RETIRED) gentoo-dev 2006-05-29 17:26:43 UTC
Your patch looks fine to me - glad the set | grep is gone. :)

Not sure what you mean about rsync/cvs/uselessness/use groups though.. Can you expand on your thoughts a little please?
Comment 18 Zac Medico gentoo-dev 2006-05-29 20:10:43 UTC
(In reply to comment #17)
> Not sure what you mean about rsync/cvs/uselessness/use groups though..

Well, I suppose it doesn't matter too much if users running old versions of portage don't see the new IUSE_* flags in metadata that they generate locally (though they will be able to see them in metadata generated for rsync distribution, as soon as I update osprey).  So, I suppose it's useful for ebuild maintainers right now if they want to drop the old-style manual expansions.

However, this same functionality could be represented by a new "use groups" syntax for IUSE.  A possible alternative interim solution to the proposed patch would be to provide an eclass function (in eutils for example) that allows ebuilds to perform semi-automatic expansion.  An eclass implementation would could work with all existing versions of portage right away.
Comment 19 Zac Medico gentoo-dev 2006-05-30 18:14:51 UTC
Created attachment 87933 [details, diff]
add an iuse_expand() function to eutils.eclass

An ebuild that wishes to automatically expand IUSE_* variables into IUSE need only inherit eutils, define the IUSE_* variables, then call the iuse_expand() function.    With this approach, there is no need to detect already expanded USE flags or make the list unique because ebuilds written to use this function will define the variables such that it is unnecessary.

The class function has many considerable advantages over direct integration into portage:

1) It works with all existing versions of portage immediately (including third party package managers such as pkgcore and paludis).
2) There is no global special case for IUSE_* variables, so there is no need to grep the tree for potential strange uses of IUSE_* varables.  Only the maintainers of ebuilds that use the iuse_expand() function need to be concerned about the IUSE_* variables.
3) Similar functionality will likely be provided in the future via "use groups" or something similar.
Comment 20 Jason Stubbs (RETIRED) gentoo-dev 2006-05-31 19:02:20 UTC
Yep, I'm in favour of the eutils extension. 
Comment 21 SpanKY gentoo-dev 2006-06-07 06:28:24 UTC
the problem with that is we get global scope calls to `tr` in the ebuilds now ... wont that cause a QA warning ?
Comment 22 Zac Medico gentoo-dev 2006-06-07 07:12:03 UTC
You only get those QA warnings for stuff in QA_INTERCEPTORS and luckily "tr" isn't in there now.
Comment 23 Jason Stubbs (RETIRED) gentoo-dev 2006-06-07 07:46:56 UTC
I found a bash based version at http://www.codecomments.com/archive287-2005-2-392096.html. Apparently, it would suffice for short strings but some testing would be probably be warranted.
Comment 24 Jason Stubbs (RETIRED) gentoo-dev 2006-06-07 08:06:17 UTC
On my system, for 5000 runs echoing to tr and assigning the result takes about 14.6s. With the function listed in the previous URL, I get the following results.

 1 char    4.7s
 5 chars   6.6s
10 chars   9.5s
15 chars  12.2s

Looking at the function, it could be optimized a fair bit for our specific case though.
Comment 25 SpanKY gentoo-dev 2006-11-28 17:44:48 UTC
*** Bug 156565 has been marked as a duplicate of this bug. ***
Comment 26 Marius Mauch (RETIRED) gentoo-dev 2007-01-10 02:33:21 UTC
So what's the status on this?
Comment 27 Mike Kelly (RETIRED) gentoo-dev 2007-02-17 01:41:13 UTC
*** Bug 167252 has been marked as a duplicate of this bug. ***
Comment 28 Jakub Moc (RETIRED) gentoo-dev 2007-03-13 19:09:56 UTC
*** Bug 170748 has been marked as a duplicate of this bug. ***
Comment 29 Jakub Moc (RETIRED) gentoo-dev 2007-03-28 21:24:11 UTC
*** Bug 172598 has been marked as a duplicate of this bug. ***
Comment 30 Jakub Moc (RETIRED) gentoo-dev 2007-03-28 21:24:44 UTC
*** Bug 172601 has been marked as a duplicate of this bug. ***
Comment 31 Marius Mauch (RETIRED) gentoo-dev 2007-06-07 14:22:01 UTC
Anyone going for this?
Comment 32 SpanKY gentoo-dev 2007-11-28 07:32:13 UTC
*** Bug 136953 has been marked as a duplicate of this bug. ***
Comment 33 SpanKY gentoo-dev 2008-01-15 09:55:26 UTC
*** Bug 136953 has been marked as a duplicate of this bug. ***
Comment 34 Zac Medico gentoo-dev 2011-05-10 17:04:30 UTC
*** Bug 70648 has been marked as a duplicate of this bug. ***
Comment 35 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-09-06 21:32:04 UTC
Does anyone want this anymore? I think most ebuilds are using 'for' loop-based solution nowadays. The idea sounds certainly doable but it requires fresh discussion on gentoo-dev@. If you want it, please start it. If you don't, we'll just WONTFIX it.
Comment 36 Ulrich Müller gentoo-dev 2014-09-06 21:48:25 UTC
No progress since 7 years. Closing.