| Summary: | sys-apps/portage: binary packages should use internal IUSE_EFFECTIVE metadata for independence from profile IUSE_IMPLICIT state | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Zac Medico <zmedico> |
| Component: | Binary packages support | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | alexander, vivo75 |
| Priority: | Normal | Keywords: | InVCS |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | All | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=644990 https://bugs.gentoo.org/show_bug.cgi?id=912648 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 651804 | ||
|
Description
Zac Medico
2017-12-08 18:37:22 UTC
Once this bug is fixed, we can eliminate the [[ ${EMERGE_FROM} != binary ]] condition where the use() function dies for flags not in IUSE_EFFECTIVE in bin/phase-helpers.sh:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=fe09b49fd09f42b9b6956f6c299bd325c62f0e83
The ${PKGDIR}/Packages file header contains the settings of IUSE_IMPLICIT and USE_EXPAND_IMPLICIT variables, but those values come directly from the current profile settings, so they are vulnerable to profile changes. The ${PKGDIR}/Packages file does not include IUSE_EFFECTIVE metadata for each package, which means this value is not known when instantiating Package instance for remote packages. In order to solve this problem, any member of USE that is not found in IUSE can be assumed to be a member of IUSE_EFFECTIVE (for EAPIs that support it).
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/83ab249e6721189c1cd79545e62bad0e https://github.com/gentoo/portage/pull/235 (In reply to Zac Medico from comment #1) > Once this bug is fixed, we can eliminate the [[ ${EMERGE_FROM} != binary ]] > condition where the use() function dies for flags not in IUSE_EFFECTIVE in > bin/phase-helpers.sh: > > https://gitweb.gentoo.org/proj/portage.git/commit/ > ?id=fe09b49fd09f42b9b6956f6c299bd325c62f0e83 It's not save to die for binary packages unless we use the embedded IUSE_EFFECTIVE metadata, and I haven't implemented that yet. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2382082dcfce5e6c2be7c4dd6aed7c32e1d20616 commit 2382082dcfce5e6c2be7c4dd6aed7c32e1d20616 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2017-12-11 01:12:41 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2017-12-11 02:31:45 +0000 Handle binary package IUSE_IMPLICIT divergence (bug 640318) Since profile IUSE_IMPLICIT settings may diverge from those that a binary package was built with, consider members of built USE settings to be members of IUSE_EFFECTIVE. Only do this for EAPIs that support IUSE_EFFECTIVE, since built USE settings for earlier EAPIs may contain a large number of irrelevant flags. Note that the binary package may be remote, so it's only possible to rely on metadata that is available in the remote Packages file, and the IUSE_IMPLICIT header in the Packages file is vulnerable to mutation. Bug: https://bugs.gentoo.org/640318 pym/_emerge/Package.py | 32 +++++++++++++++++++++++++++++++ pym/portage/dbapi/__init__.py | 5 +++++ pym/portage/package/ebuild/config.py | 6 ++++++ pym/portage/tests/dbapi/test_fakedbapi.py | 20 +++++++++++++++++++ 4 files changed, 63 insertions(+)} There's still the remaining issue of how to handle the missing IUSE_EFFECTIVE data in $PKGDIR/Packages. Without that, USE dependencies won't match if the corresponding flag is not enabled in USE and it is no longer a member of IUSE_EFFECTIVE from the current profile settings. In order to cover the flags that were in IUSE_EFFECTIVE but were not enabled when the package was built, it is probably good enough to rely on the IUSE_IMPLICIT header from the Packages file. The IUSE_IMPLICIT setting from a remote binhost certainly does not propagate to the local ${PKGDIR}/Packages header, so maybe we should use a union local elements and remote elements.
I'm working on this in the following branch: https://github.com/zmedico/portage/tree/bug_640318_binary_iuse_implicit I'm planning to make binarytree update the IUSE_IMPLICIT settings in ${PKGDIR}/Packages only when the current profile has a non-empty IUSE_IMPLICIT setting. That should protect these settings from being accidentally discarded if the profile is in a temporarily invalid state. These settings might also be used for installed packages when there are no other settings available. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=bfe7892202b85c46ff127048ca1cc752c54b186c commit bfe7892202b85c46ff127048ca1cc752c54b186c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-06 20:05:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-09 05:19:55 +0000 _pkg_str: add _db attribute (bug 640318) In order to propagate information from dbapi to Package instance, it's useful for each _pkg_str instance to have a _db attribute which references the dbapi instance that instantiated it. Use a new dbapi._iuse_implicit_cnstr method to delegate implicit IUSE logic to the dbapi instance, in order to make the behavior customizable at the dbapi level for the purposes of bug 640318. This patch consists only of refactoring, with no behavioral changes. Bug: https://bugs.gentoo.org/640318 pym/_emerge/Package.py | 53 +++++++++--------------------- pym/_emerge/depgraph.py | 11 ++++++- pym/_emerge/resolver/DbapiProvidesIndex.py | 3 +- pym/portage/dbapi/__init__.py | 44 +++++++++++++++++++++---- pym/portage/dbapi/bintree.py | 15 +++++---- pym/portage/dbapi/porttree.py | 4 +-- pym/portage/dbapi/vartree.py | 5 +-- pym/portage/dbapi/virtual.py | 4 +-- pym/portage/versions.py | 6 ++-- 9 files changed, 84 insertions(+), 61 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=232aae7f1fd79573426c0da9c1cbd79324994099 commit 232aae7f1fd79573426c0da9c1cbd79324994099 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-10 01:32:04 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-10 01:32:04 +0000 binarytree._update_pkgindex_header: skip update if profile invalid (bug 640318) Return silently if the current profile is invalid or does not have an IUSE_IMPLICIT variable, since it's useful to maintain a cache of implicit IUSE settings for use with binary packages. Bug: https://bugs.gentoo.org/640318 pym/portage/dbapi/bintree.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=1689911cf9aeb363b23e0bbd4d357cab5ce78538 commit 1689911cf9aeb363b23e0bbd4d357cab5ce78538 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-10 22:16:07 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-10 22:16:07 +0000 binarytree._update_pkgindex_header: include USE_EXPAND_UNPREFIXED vars (bug 640318) Values of USE_EXPAND_UNPREFIXED variables are required for implicit IUSE calculations. Bug: https://bugs.gentoo.org/640318 pym/portage/dbapi/bintree.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)} I've posted a patch to propagate implicit IUSE and USE_EXPAND for emerge --usepkgonly mode: https://archives.gentoo.org/gentoo-portage-dev/message/c544f506fd3d4bfa0476d46ae03f28bc https://github.com/gentoo/portage/pull/305 The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=cab78dba98c4309504e077c52a2ab0f0b7e27cc8 commit cab78dba98c4309504e077c52a2ab0f0b7e27cc8 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-17 20:56:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-18 21:55:58 +0000 emerge --usepkgonly: propagate implicit IUSE and USE_EXPAND (bug 640318) For emerge --usepkgonly mode, it's useful to be able to operate without dependence on a local ebuild repository and profile. Therefore, merge Packages header settings from the binary package database (including binhost(s) if enabled) into the profile configuration, in order to propagate implicit IUSE and USE_EXPAND settings for use with binary and installed packages. Values are appended, so the result is a union of elements. Also use ARCH from the binary package database if it's not defined by the profile, since ARCH is used for validation by emerge's profile_check function, and also for KEYWORDS logic in the _getmaskingstatus function. All changes are currently confined to --usepkgonly mode, since this is the mode where it is needed the most, and this ensures that behavior of source-based builds is completely unaffected. The changes only affect dependency calculations (where implicit IUSE plays a role) and the user interface (display of USE_EXPAND flags). The bash execution environment for binary and installed packages is completely unaffected, since that environment relies on variables loaded from environment.bz2 files. Bug: https://bugs.gentoo.org/640318 pym/_emerge/actions.py | 48 +++++++++++++++++-------- pym/_emerge/main.py | 4 --- pym/portage/dbapi/bintree.py | 56 ++++++++++++++++++++++++++++++ pym/portage/package/ebuild/config.py | 7 ++-- pym/portage/package/ebuild/profile_iuse.py | 32 +++++++++++++++++ 5 files changed, 127 insertions(+), 20 deletions(-)} The --usepkgonly case is probably the only one that anyone cares about, so I think we can consider this fixed in portage-2.3.31. Fixed in portage-2.3.40-r1. |