Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 640318 - sys-apps/portage: binary packages should use internal IUSE_EFFECTIVE metadata for independence from profile IUSE_IMPLICIT state
Summary: sys-apps/portage: binary packages should use internal IUSE_EFFECTIVE metadata...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: .tbz2 support (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 651804
  Show dependency tree
 
Reported: 2017-12-08 18:37 UTC by Zac Medico
Modified: 2018-07-02 18:41 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2017-12-08 18:37:22 UTC
For binary packages, currently the Package.is_valid_flag method calls the config._iuse_effective_match method which uses information from the current profile. This is the faulty code in the _use_class._init_use method:

	is_valid_flag = self.iuse.is_valid_flag
	enabled_flags = [x for x in use_str.split() if is_valid_flag(x)]

If any IUSE_IMPLICIT values have been removed from the profile since the package was built, then those values will be erroneously filtered from enabled_flags.

It would be better to use the binary package's internal IUSE_EFFECTIVE metadata, so that evaluation of USE for binary packages is independent of the current profile state.
Comment 1 Zac Medico gentoo-dev 2017-12-08 18:43:20 UTC
Once this bug is fixed, we can eliminate the [[ ${EMERGE_FROM} != binary ]] condition where the use() function dies for flags not in IUSE_EFFECTIVE in bin/phase-helpers.sh:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=fe09b49fd09f42b9b6956f6c299bd325c62f0e83
Comment 2 Zac Medico gentoo-dev 2017-12-08 19:01:12 UTC
The ${PKGDIR}/Packages file header contains the settings of IUSE_IMPLICIT and USE_EXPAND_IMPLICIT variables, but those values come directly from the current profile settings, so they are vulnerable to profile changes. The ${PKGDIR}/Packages file does not include IUSE_EFFECTIVE metadata for each package, which means this value is not known when instantiating Package instance for remote packages. In order to solve this problem, any member of USE that is not found in IUSE can be assumed to be a member of IUSE_EFFECTIVE (for EAPIs that support it).
Comment 3 Zac Medico gentoo-dev 2017-12-11 02:08:09 UTC
Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/83ab249e6721189c1cd79545e62bad0e
https://github.com/gentoo/portage/pull/235

(In reply to Zac Medico from comment #1)
> Once this bug is fixed, we can eliminate the [[ ${EMERGE_FROM} != binary ]]
> condition where the use() function dies for flags not in IUSE_EFFECTIVE in
> bin/phase-helpers.sh:
> 
> https://gitweb.gentoo.org/proj/portage.git/commit/
> ?id=fe09b49fd09f42b9b6956f6c299bd325c62f0e83

It's not save to die for binary packages unless we use the embedded IUSE_EFFECTIVE metadata, and I haven't implemented that yet.
Comment 4 Larry the Git Cow gentoo-dev 2017-12-11 03:33:32 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=2382082dcfce5e6c2be7c4dd6aed7c32e1d20616

commit 2382082dcfce5e6c2be7c4dd6aed7c32e1d20616
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2017-12-11 01:12:41 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2017-12-11 02:31:45 +0000

    Handle binary package IUSE_IMPLICIT divergence (bug 640318)
    
    Since profile IUSE_IMPLICIT settings may diverge from those
    that a binary package was built with, consider members of
    built USE settings to be members of IUSE_EFFECTIVE. Only do
    this for EAPIs that support IUSE_EFFECTIVE, since built USE
    settings for earlier EAPIs may contain a large number of
    irrelevant flags.
    
    Note that the binary package may be remote, so it's only
    possible to rely on metadata that is available in the remote
    Packages file, and the IUSE_IMPLICIT header in the Packages
    file is vulnerable to mutation.
    
    Bug: https://bugs.gentoo.org/640318

 pym/_emerge/Package.py                    | 32 +++++++++++++++++++++++++++++++
 pym/portage/dbapi/__init__.py             |  5 +++++
 pym/portage/package/ebuild/config.py      |  6 ++++++
 pym/portage/tests/dbapi/test_fakedbapi.py | 20 +++++++++++++++++++
 4 files changed, 63 insertions(+)}
Comment 5 Zac Medico gentoo-dev 2017-12-11 04:55:10 UTC
There's still the remaining issue of how to handle the missing IUSE_EFFECTIVE data in $PKGDIR/Packages. Without that, USE dependencies won't match if the corresponding flag is not enabled in USE and it is no longer a member of IUSE_EFFECTIVE from the current profile settings.
Comment 6 Zac Medico gentoo-dev 2017-12-12 18:15:17 UTC
In order to cover the flags that were in IUSE_EFFECTIVE but were not enabled when the package was built, it is probably good enough to rely on the IUSE_IMPLICIT header from the Packages file. The IUSE_IMPLICIT setting from a remote binhost certainly does not propagate to the local ${PKGDIR}/Packages header, so maybe we should use a union local elements and remote elements.
Comment 7 Zac Medico gentoo-dev 2018-04-07 02:19:53 UTC
I'm working on this in the following branch:

https://github.com/zmedico/portage/tree/bug_640318_binary_iuse_implicit

I'm planning to make binarytree update the IUSE_IMPLICIT settings in ${PKGDIR}/Packages only when the current profile has a non-empty IUSE_IMPLICIT setting. That should protect these settings from being accidentally discarded if the profile is in a temporarily invalid state. These settings might also be used for installed packages when there are no other settings available.
Comment 8 Larry the Git Cow gentoo-dev 2018-04-09 05:21:11 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=bfe7892202b85c46ff127048ca1cc752c54b186c

commit bfe7892202b85c46ff127048ca1cc752c54b186c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-06 20:05:25 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-09 05:19:55 +0000

    _pkg_str: add _db attribute (bug 640318)
    
    In order to propagate information from dbapi to Package instance,
    it's useful for each _pkg_str instance to have a _db attribute
    which references the dbapi instance that instantiated it. Use a new
    dbapi._iuse_implicit_cnstr method to delegate implicit IUSE logic to
    the dbapi instance, in order to make the behavior customizable at the
    dbapi level for the purposes of bug 640318. This patch consists only
    of refactoring, with no behavioral changes.
    
    Bug: https://bugs.gentoo.org/640318

 pym/_emerge/Package.py                     | 53 +++++++++---------------------
 pym/_emerge/depgraph.py                    | 11 ++++++-
 pym/_emerge/resolver/DbapiProvidesIndex.py |  3 +-
 pym/portage/dbapi/__init__.py              | 44 +++++++++++++++++++++----
 pym/portage/dbapi/bintree.py               | 15 +++++----
 pym/portage/dbapi/porttree.py              |  4 +--
 pym/portage/dbapi/vartree.py               |  5 +--
 pym/portage/dbapi/virtual.py               |  4 +--
 pym/portage/versions.py                    |  6 ++--
 9 files changed, 84 insertions(+), 61 deletions(-)}
Comment 9 Larry the Git Cow gentoo-dev 2018-04-10 01:33:28 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=232aae7f1fd79573426c0da9c1cbd79324994099

commit 232aae7f1fd79573426c0da9c1cbd79324994099
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-10 01:32:04 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-10 01:32:04 +0000

    binarytree._update_pkgindex_header: skip update if profile invalid (bug 640318)
    
    Return silently if the current profile is invalid or does not have
    an IUSE_IMPLICIT variable, since it's useful to maintain a cache of
    implicit IUSE settings for use with binary packages.
    
    Bug: https://bugs.gentoo.org/640318

 pym/portage/dbapi/bintree.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)}
Comment 10 Larry the Git Cow gentoo-dev 2018-04-10 22:17:36 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=1689911cf9aeb363b23e0bbd4d357cab5ce78538

commit 1689911cf9aeb363b23e0bbd4d357cab5ce78538
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-10 22:16:07 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-10 22:16:07 +0000

    binarytree._update_pkgindex_header: include USE_EXPAND_UNPREFIXED vars (bug 640318)
    
    Values of USE_EXPAND_UNPREFIXED variables are required for implicit
    IUSE calculations.
    
    Bug: https://bugs.gentoo.org/640318

 pym/portage/dbapi/bintree.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)}
Comment 11 Zac Medico gentoo-dev 2018-04-18 01:17:46 UTC
I've posted a patch to propagate implicit IUSE and USE_EXPAND for emerge --usepkgonly mode:

https://archives.gentoo.org/gentoo-portage-dev/message/c544f506fd3d4bfa0476d46ae03f28bc
https://github.com/gentoo/portage/pull/305
Comment 12 Larry the Git Cow gentoo-dev 2018-04-18 22:06:31 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=cab78dba98c4309504e077c52a2ab0f0b7e27cc8

commit cab78dba98c4309504e077c52a2ab0f0b7e27cc8
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-17 20:56:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-18 21:55:58 +0000

    emerge --usepkgonly: propagate implicit IUSE and USE_EXPAND (bug 640318)
    
    For emerge --usepkgonly mode, it's useful to be able to operate without
    dependence on a local ebuild repository and profile. Therefore,
    merge Packages header settings from the binary package database
    (including binhost(s) if enabled) into the profile configuration,
    in order to propagate implicit IUSE and USE_EXPAND settings for use
    with binary and installed packages. Values are appended, so the result
    is a union of elements.
    
    Also use ARCH from the binary package database if it's not defined by
    the profile, since ARCH is used for validation by emerge's profile_check
    function, and also for KEYWORDS logic in the _getmaskingstatus function.
    
    All changes are currently confined to --usepkgonly mode, since this is
    the mode where it is needed the most, and this ensures that behavior of
    source-based builds is completely unaffected.
    
    The changes only affect dependency calculations (where implicit IUSE
    plays a role) and the user interface (display of USE_EXPAND flags).
    The bash execution environment for binary and installed packages is
    completely unaffected, since that environment relies on variables loaded
    from environment.bz2 files.
    
    Bug: https://bugs.gentoo.org/640318

 pym/_emerge/actions.py                     | 48 +++++++++++++++++--------
 pym/_emerge/main.py                        |  4 ---
 pym/portage/dbapi/bintree.py               | 56 ++++++++++++++++++++++++++++++
 pym/portage/package/ebuild/config.py       |  7 ++--
 pym/portage/package/ebuild/profile_iuse.py | 32 +++++++++++++++++
 5 files changed, 127 insertions(+), 20 deletions(-)}
Comment 13 Zac Medico gentoo-dev 2018-04-18 22:35:49 UTC
The --usepkgonly case is probably the only one that anyone cares about, so I think we can consider this fixed in portage-2.3.31.
Comment 14 Zac Medico gentoo-dev 2018-07-02 18:41:06 UTC
Fixed in portage-2.3.40-r1.