Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 605650 - [dlang] Ebuild failures occuring in global scope
Summary: [dlang] Ebuild failures occuring in global scope
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Overlays (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Marco Leise
URL: https://qa-reports.gentoo.org/output/...
Whiteboard:
Keywords:
Depends on:
Blocks: repository-qa-issues
  Show dependency tree
 
Reported: 2017-01-14 09:40 UTC by Michał Górny
Modified: 2017-04-06 21:21 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-01-14 09:40:30 UTC
Our automated repository checks [1] have detected that the 'dlang'
repository contains ebuilds that trigger fatal errors during the cache
regeneration. This usually means that the ebuilds call 'die' in global
scope indicating serious issues.

Global-scope failures prevent the ebuild not only from being installed
but also from being properly processed by the Package Manager. Since
metadata can not be obtained for those ebuilds, no cache entries are
created for them and the Package Manager needs to retry running them
every time it stumbles upon them. This involves both a serious slowdown
and repeating error output while performing dependency resolution.

The most common cause of global-scope failures is use of removed or
banned APIs in old ebuilds. In particular, this includes eclasses being
removed or removing support for old EAPIs. Nonetheless there are also
other issues such as performing illegal operations in global scope
(external program calls), malformed bash in ebuilds or malformed
metadata.xml.

The error log for the repository can be found at:

  https://qa-reports.gentoo.org/output/repos/dlang.html

In particular, please look for highlighted '!!! ERROR' and '!!! caught
exception' lines. The former usually mean failures coming from eclasses
and the ebuild itself, while exceptions usually mean malformed ebuilds
or metadata.xml.

While at it, please consider fixing global-scope 'use' call warnings (if
any). They are not fatal but are considered a serious QA violation.
'use' functions must not ever be called outside of phase functions.

Please fix the issue ASAP, possibly via removing unmaintained, old
ebuilds. We reserve the right to remove the repository from our list if
we do not receive any reply within 4 weeks.

[1]:https://wiki.gentoo.org/wiki/Project:Repository_mirror_and_CI
Comment 1 Marco Leise 2017-01-20 01:47:05 UTC
It is due to my reliance on KEYWORDS in the bash environment. As is known, a package may only depend on other packages that are at least as stable on each target architecture. One dependency of a Dlang package is a Dlang compiler with its custom tailored runtime and standard library Phobos. So instead of listing and maintaining each option manually, I let dlang.eclass generate the dependencies.

It takes a list of Dlang compilers and their stability on each architecture from dlang-compilers.eclass (and that file is kept in sync with KEYWORDS on the three Dlang compilers) and matches that against the package's KEYWORDS and optional compatible language versions (DLANG_VERSION_RANGE). The result are computed IUSE, REQUIRED_USE and (R)DEPEND variables.

Of particular interest for me is the logic that automatically removes potential compilers from a stable package's dependency list when the compiler/Phobos combination is unstable, since then I don't have to make sure all compilers are stable before marking a package stable. The package will simply drop offending compilers itself and offer some of the other choices.
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-04-05 08:10:03 UTC
Something is seriously broken in that logic. It's too complex for me to even try to figure out where or why it's failing. It's breaking metadata cache. Please fix it or the repository will be removed.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-04-05 08:10:20 UTC
I should also point out that stable keywords don't belong in unofficial repositories.
Comment 4 Daniel M. Weeks 2017-04-06 02:32:01 UTC
Marco, I think I understand what you've done and it's clever. This die call shouldn't happen unless a package has an ARCH+DLANG_VERSION_RANGE combination that doesn't exist in the dlang-compilers eclass table. (Then it really would be a bad package that needs QA attention.) Additionally, I think I tracked down exactly what's going wrong when the dlang eclass in inherited by an ebuild inside pmaint (pkgcore) - at least in regards to it affecting #605654. I have a fix but I want to understand the real underlying problem first, which is not in the dlang eclass!!!

Michał, the behavior of pkgcore and egencache is drastically different for this case. I believe egencache --update-use-local-desc --update-pkg-desc-index is effectively the same ("the old way") and works perfectly fine for ebuilds inheriting the dlang eclass. It seems to come down to the dlang eclass using the EAPI 5 variable USE_EXPAND_VALUES_ARCH which is not set in the environment by pmaint/pkgcore. I think the principal of least surprise would suggest pmaint should set USE_EXPAND_VALUES_ARCH since it's trying to do what egencache has done (without problem) for a long time.

Perhaps the pkgcore page on the Gentoo wiki is outdated but it contains the warning: "pkgcore support for EAPI 5 is still experimental." Personally, I don't think pkgcore should be used for QA testing if it doesn't fully implement all the EAPI features or if it's not behaving the same way as long-standing, stable, standard, recommended tools. (See also #613564)
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-04-06 05:29:07 UTC
USE_EXPAND_VALUES_ARCH is not supposed to be exposed to ebuild environment. Relying on that is incorrect. Portage accidentally exposing random configuration variables is not a feature but a long-standing mis-design.

pkgcore is much faster than Portage, and in many cases it's also cleaner wrt QA (i.e. lacks many of the bugs Portage has). It's the best tool for QA that we have at the moment.
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-04-06 21:09:14 UTC
The bug seems to be fixed in the repository. Closing.
Comment 7 Marco Leise 2017-04-06 21:21:38 UTC
I copied the values from USE_EXPAND_VALUES_ARCH into a global variable in dlang-compilers.eclass and added a warning for when USE_EXPAND_VALUES_ARCH is updated.