My use case for this enhancement proposal is building application containers (e.g. docker or OCI images) using Portage. I use a two phase build for this: In phase 1 the host system creates an image with a minimal Gentoo installation from a stage3 tarball (a "builder" image). In phase 2 the builder creates an image containing the desired package (e.g. app-shells/bash) and all necessary runtime dependencies of the package (e.g. readline, ncurses, libc, …). The resulting image should be as small as possible. Therefore, build-time dependencies are installed in the builder environment (via emerge --onlydeps …) and all runtime dependencies go into the final image (via emerge --root=… --root-deps=rdeps …). The build times for this process are quite long even when using binary packages for caching. Trying to reduce the build time, I stumbled into the following issue: There is no way to tell emerge to "install all packages required to build a specified package". Emerge does have the --onlydeps flag which includes all dependencies of a package. Calling "emerge --onlydeps dev-util/foo" will work for the described use case, but the set of installed packages is larger than necessary, because it includes all runtime dependencies of dev-util/foo. Calling "emerge --onlydeps --onlydeps-with-rdeps=n" ignores all runtime dependencies of the package. However, this flag combination also ignores the build-time dependencies of the runtime dependencies. This potentially causes a subsequent "emerge --root=… --root-deps=rdeps" to fail (e.g. when dev-util/foo depends on a package written in Go, but dev-lang/go is not installed in the builder). I assume that the behavior of "--onlydeps --onlydeps-with-rdeps=n" is intentional, because it behaves as documented and the commit message that introduced --onlydeps-with-rdeps states that the goal was to set up for "emerge --buildpkgonly" afterwards. Is it worthwhile to support a use case like this in Portage? If so, what could the emerge call look like? Reproducible: Always
(In reply to Michael Seifert from comment #0) > Calling "emerge --onlydeps --onlydeps-with-rdeps=n" ignores all runtime > dependencies of the package. However, this flag combination also ignores the > build-time dependencies of the runtime dependencies. This potentially causes > a subsequent "emerge --root=… --root-deps=rdeps" to fail (e.g. when > dev-util/foo depends on a package written in Go, but dev-lang/go is not > installed in the builder). Does it work if you add --with-bdeps=y along with --onlydeps and --onlydeps-with-rdeps=n?
Created attachment 526560 [details] Ebuild for reproducing the behavior The ebuild has a runtime dependency on dev-go/sanitized-anchor-name which in turn has a build-time dependency on dev-lang/go (via the inherited eclass).
I stuck together a minimal example and tested again. Adding --with-bdeps=y seems to have no effect: # emerge --pretend --onlydeps --onlydeps-with-rdeps=y dev-util/foo [ebuild N ] dev-lang/go-1.10.1 [ebuild N ] dev-go/sanitized-anchor-name-0_pre20151027 # emerge --pretend --onlydeps --onlydeps-with-rdeps=n dev-util/foo <no console output> # emerge --pretend --onlydeps --onlydeps-with-rdeps=n --with-bdeps=y dev-util/foo <no console output>
Tested with sys-apps/portage-2.3.28.
(In reply to Michael Seifert from comment #2) > Created attachment 526560 [details] > Ebuild for reproducing the behavior > > The ebuild has a runtime dependency on dev-go/sanitized-anchor-name which in > turn has a build-time dependency on dev-lang/go (via the inherited eclass). With --onlydeps-with-rdeps=n, it's intended to ignore the dev-go/sanitized-anchor-name, because it's in RDEPEND. When we ignore the dev-go/sanitized-anchor-name dependency, it means that the dependencies of dev-go/sanitized-anchor-name are considered irrelevant. In fact we don't even to check to see if a dev-go/sanitized-anchor-name ebuild is available in this case, which means that we also don't examine its dependencies. It seems that the main problem is that you're using --root-deps=rdeps, but you aren't ensuring that the build time dependencies are satisfied. In the upcoming EAPI 7 (EAPI 7_pre1 is available now for testing), we have a BDEPEND variable, which will install build time dependencies regardless of the --root-deps option. In the absence of BDEPEND, with --root-deps you have to manually satisfy the build time dependencies somehow (I think Chromium OS uses a meta-package to pull in these dependencies).
Created attachment 526590 [details] Test ebuild for dependency resolution with EAPI 7 BDEPEND dev-util/foo has an RDEPEND on dev-util/bar and a DEPEND on dev-util/baz. dev-util/bar has a BDEPEND on dev-lang/go. dev-util/baz has no dependencies.
(In reply to Zac Medico from comment #5) > With --onlydeps-with-rdeps=n, it's intended to ignore the > dev-go/sanitized-anchor-name, because it's in RDEPEND. When we ignore the > dev-go/sanitized-anchor-name dependency, it means that the dependencies of > dev-go/sanitized-anchor-name are considered irrelevant. In fact we don't > even to check to see if a dev-go/sanitized-anchor-name ebuild is available > in this case, which means that we also don't examine its dependencies. Thanks for the clarification. This is more or less how I understood it. > It seems that the main problem is that you're using --root-deps=rdeps, but > you aren't ensuring that the build time dependencies are satisfied. Correct. The purpose of this issue was to start a discussion for adding a command line option for exactly this task: "Make sure that all dependencies required to build package X are satisfied". For example, --onlydeps ensures that I can subsequently run emerge --nodeps without getting an error. --onlydeps-with-rdeps=n ensures that I can run emerge --buildpkgonly. I am looking for something that sets everything up for emerge --root=… --root-deps=rdeps. > In the upcoming EAPI 7 (EAPI 7_pre1 is available now for testing), we have a > BDEPEND variable, which will install build time dependencies regardless of > the --root-deps option. In the absence of BDEPEND, with --root-deps you have > to manually satisfy the build time dependencies somehow (I think Chromium OS > uses a meta-package to pull in these dependencies). This initially sounded like the solution for my problem so I ran a small test (see attachment for the ebuilds I used): # emerge --pretend --onlydeps dev-util/foo [ebuild N ] dev-lang/go-1.10.1 [ebuild N ] dev-util/baz-1.0.0 [ebuild N ] dev-util/bar-1.0.0 # emerge --pretend --onlydeps --onlydeps-with-rdeps=n dev-util/foo [ebuild N ] dev-util/baz-1.0.0 --onlydeps and --onlydeps-with-rdeps are behaving as usual. # emerge --pretend --root=/tmp/rootfs dev-util/foo [ebuild N ] dev-util/baz-1.0.0 to /tmp/rootfs/ [ebuild N ] dev-lang/go-1.10.1 [ebuild N ] dev-util/bar-1.0.0 to /tmp/rootfs/ [ebuild N ] dev-util/foo-1.0.0 to /tmp/rootfs/ dev-lang/go is installed into "/", because it specified in BDEPEND. All other dependencies are installed into /tmp/rootfs. Everything behaves as expected. # emerge --pretend --root=/tmp/rootfs --root-deps=rdeps dev-util/foo [ebuild N ] dev-util/baz-1.0.0 to /tmp/rootfs/ [ebuild N ] dev-lang/go-1.10.1 [ebuild N ] dev-util/bar-1.0.0 to /tmp/rootfs/ [ebuild N ] dev-util/foo-1.0.0 to /tmp/rootfs/ When adding --root-deps=rdeps Go is still installed into "/", which is correct. However, dev-util/baz is still scheduled for installation in the new ROOT, even though it is declared in DEPEND of dev-util/foo. I would expect that either dev-util/baz is installed into "/" OR that dev-util/baz does not appear at all in this console output. (If baz was installed to /, my problem was solved, because all build dependencies went to "/" and all run time deps to "/tmp/rootfs". If it was not installed at all, this left me with my original problem that I have to find a way to satisfy all dependencies for building foo :) By the looks of it, the new EAPI 7 features do not address the use case described in this ticket.
(In reply to Michael Seifert from comment #7) > # emerge --pretend --root=/tmp/rootfs --root-deps=rdeps dev-util/foo > [ebuild N ] dev-util/baz-1.0.0 to /tmp/rootfs/ > [ebuild N ] dev-lang/go-1.10.1 > [ebuild N ] dev-util/bar-1.0.0 to /tmp/rootfs/ > [ebuild N ] dev-util/foo-1.0.0 to /tmp/rootfs/ > > When adding --root-deps=rdeps Go is still installed into "/", which is > correct. However, dev-util/baz is still scheduled for installation in the > new ROOT, even though it is declared in DEPEND of dev-util/foo. For EAPI 7_pre1, DEPEND always installs into ROOT. You have to put dev-util/baz in BDEPEND if you want it installed into "/".
I see. I read up a bit on the intentions behind BDEPEND/DEPEND. It's not exactly what I was looking for. Yet, EAPI 7 seems like a huge improvement for my specific use case and dependency management in Portage in general. Thanks for helping out and for being on top of this topic!
