Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 563034 - sys-apps/portage-2.2.20.1: subslot deps (or all deps?) broken when ROOT != /
Summary: sys-apps/portage-2.2.20.1: subslot deps (or all deps?) broken when ROOT != /
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Dependencies (show other bugs)
Hardware: All Linux
: Normal major (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on: 317337
Blocks: 137867
  Show dependency tree
 
Reported: 2015-10-14 04:30 UTC by Daniel Robbins
Modified: 2016-09-15 15:19 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Robbins 2015-10-14 04:30:31 UTC
Portage 2.2.20.1 does not properly implement subslot deps when ROOT!=/. Here is an example. Using a stage3 with ncurses-5 installed (both 5.9-r5 with SLOT 0/5 and 5.9-r99 with SLOT 5/5) to generate a new stage1, a bash ebuild containing DEPEND and RDEPEND of >=sys-libs/ncurses-5.2-r2:0= is merged with ROOT set to /tmp/stage1root via "ROOT=/tmp/stage1root emerge bash". What happens is that bash is emerged to /tmp/stage1root along with ncurses-6 to /tmp/stage1root. but ncurses-6 is not installed to / first. So bash links against ncurses-5 but there is no ncurses-5 library in /tmp/stage1root. This is incorrect behavior.

The "ROOT=/tmp/stage1root emerge -u bash" case is broken as well. It will emerge ncurses:0/6 to /tmp/stage1root, along with bash, which will also violate the sub-slot deps and result in a broken bash installation.

What I believe should happen is as follows, which not only respects the existing definition of sub-slot dependencies (in other words, this isn't a new feature but just ensuring that things work as already specified in the man page) but also has the wonderful side-effect of installing non-broken binaries: "ROOT=/tmp/stage1root emerge bash" would find that ncurses:0/5 is installed and this would satisfy bash's DEPEND of >=sys-libs/ncurses-5.2-r2:0=. bash would be built and link against this installed version of ncurses, and emerge would then ensure that an ncurses:0/5 is merged to /tmp/stage1root due to >=sys-libs/ncurses-5.2-r2:0= in RDEPEND (and per the existing definition of sub-slot deps, emerge will ensure that "a  matching  package  with slot and sub-slot equal to the slot and sub-slot of the best installed version at the time the package was installed is available." This would be ncurses:0/5.)

"ROOT=/tmp/stage1root emerge -u bash" would behave slightly differently, by emerging the bestmatch for >=sys-libs/ncurses-5.2-r2:0= as a DEPEND to /, which would be ncurses-6 (SLOT 0/6), and then merge bash to /tmp/stage1root, along with ncurses 6 to /tmp/stage1root to satisfy >=sys-libs/ncurses-5.2-r2:0= in bash's RDEPEND (ensuring that the RDEPEND matched the sub-slot of the bestmatch of >=sys-libs/ncurses-5.2-r2:0= (looking at /) at the time that bash was installed, per the definition of sub-slot deps.

Reproducible: Always
Comment 1 Alexis Ballier gentoo-dev 2015-10-14 07:29:21 UTC
(In reply to Daniel Robbins from comment #0)
> Portage 2.2.20.1 does not properly implement subslot deps when ROOT!=/. Here
> is an example. Using a stage3 with ncurses-5 installed (both 5.9-r5 with
> SLOT 0/5 and 5.9-r99 with SLOT 5/5) to generate a new stage1, a bash ebuild
> containing DEPEND and RDEPEND of >=sys-libs/ncurses-5.2-r2:0= is merged with
> ROOT set to /tmp/stage1root via "ROOT=/tmp/stage1root emerge bash". What
> happens is that bash is emerged to /tmp/stage1root along with ncurses-6 to
> /tmp/stage1root. but ncurses-6 is not installed to / first. So bash links
> against ncurses-5 but there is no ncurses-5 library in /tmp/stage1root. This
> is incorrect behavior.


I think portage is correct, but the bug is in bash linking against ncurses from /: portage uses the ncurses subslot from $ROOT vdb, which is 6, but has no way to check that bash properly linked against it.

You should probably have a look at the sysroot option of gcc, or add proper -I/-L${ROOT}/... to your CFLAGS/LDFLAGS when building stages, so that packages link against what is in $ROOT rather than /.


OTOH, since you're only doing native builds, just upgrade / before using 'ROOT=... emerge' so that libs in / and ROOT are the same.

[...]
> What I believe should happen is as follows, which not only respects the
> existing definition of sub-slot dependencies (in other words, this isn't a
> new feature but just ensuring that things work as already specified in the
> man page) but also has the wonderful side-effect of installing non-broken
> binaries: "ROOT=/tmp/stage1root emerge bash" would find that ncurses:0/5 is
> installed and this would satisfy bash's DEPEND of
> >=sys-libs/ncurses-5.2-r2:0=. bash would be built and link against this
> installed version of ncurses, and emerge would then ensure that an
> ncurses:0/5 is merged to /tmp/stage1root due to >=sys-libs/ncurses-5.2-r2:0=
> in RDEPEND (and per the existing definition of sub-slot deps, emerge will
> ensure that "a  matching  package  with slot and sub-slot equal to the slot
> and sub-slot of the best installed version at the time the package was
> installed is available." This would be ncurses:0/5.)


This won't work when cross-compiling, which can be somewhat reduced to 'ROOT=/somewhere CHOST=somethingelse emerge'.
Comment 2 Rick Farina (Zero_Chaos) gentoo-dev 2015-10-14 15:08:23 UTC
this doesn't seem to have anything to do with violating deps, it looks like bash is just linking to / instead of ROOT
Comment 3 Daniel Robbins 2015-10-14 15:33:58 UTC
When ROOT!=/, build deps should be installed to / and runtime deps should be installed to ROOT. Build deps are not always libraries so trying to address this in the bash ebuild by changing how it links does not seem like an appropriate way to fix this bug. Unless there has been some major change to what ROOT means, the deps are misbehaving.
Comment 4 Alexis Ballier gentoo-dev 2015-10-14 17:04:09 UTC
Yeah, there are unclear things about ROOT vs SYSROOT I think.

In your case, SYSROOT=/, ROOT!=/

You seem to want: Link against SYSROOT, install in ROOT, which I think is the correct meaning of those variables. However, for this to work, portage needs to install runtime deps matching those of SYSROOT: e.g. take the ebuilds from SYSROOT's vdb, not the portage tree, otherwise you get what you describe: ncurses6 installed in ROOT but bash linked against ncurses5 from SYSROOT.

I think I faced this problem in the past, and afaik, portage has always been doing that way. The only "new" problem is that bash will record ROOT's vdb subslot for ncurses while it has been built against SYSROOT's one, but IMHO this is rather minor in comparison since you'd still get a broken stage1 :)
Comment 5 James Le Cuirot gentoo-dev 2015-12-02 23:26:02 UTC
(In reply to Rick Farina (Zero_Chaos) from comment #2)
> this doesn't seem to have anything to do with violating deps, it looks like
> bash is just linking to / instead of ROOT

drobbins is right, we would not expect bash to link against ROOT in this case. You would only do this when cross-compiling and possibly when PORTAGE_CONFIGROOT != /. In the latter case, / might be entirely incompatible with ROOT, even if you're not cross-compiling. You don't want to confuse that use case with this one because it introduces a whole set of trickier problems. Keeping / up to date is sufficient here and that's much easier to do. I've just been discussing this on gentoo-dev if anyone's interested.
Comment 6 Steve L 2015-12-03 09:21:11 UTC
(In reply to Alexis Ballier from comment #4)
> Yeah, there are unclear things about ROOT vs SYSROOT I think.
> 
> In your case, SYSROOT=/, ROOT!=/
> 
> You seem to want: Link against SYSROOT, install in ROOT, which I think is
> the correct meaning of those variables. 

As a point of information, from the workingset/toolchain perspective: No, that is incorrect.

ROOT is what we link against; SYSROOT came later, and is for toolchain linkage (like gcc runtime): defaults to ROOT.
Occasionally people put other things in SYSROOT; glibc is the usual example.

We install to DESTDIR (or STAGEDIR for BSD ports.)

So yes, your "install" is in ROOT; but it's not where the end makefile is going to install to. Conceptually, it is the pre-extant installation, before the admin installs the current package.

SYSROOT is effectively a subsidiary ROOT, used because toolchain packages are a pita, so we separate them out, and we always need to link against the same SYSROOT for the particular cross-target (= CHOST for every package but toolchain, or board-support.)
Comment 7 Zac Medico gentoo-dev 2016-08-07 04:45:57 UTC
(In reply to Daniel Robbins from comment #0)
> What happens is that bash is emerged to /tmp/stage1root along with ncurses-6 to
> /tmp/stage1root. but ncurses-6 is not installed to / first. So bash links
> against ncurses-5 but there is no ncurses-5 library in /tmp/stage1root. This
> is incorrect behavior.

I don't think it's possible to properly distinguish the dependencies without an EAPI extension as discussed in bug 317337.
Comment 8 Daniel Robbins 2016-08-07 04:57:35 UTC
(In reply to Zac Medico from comment #7)
> (In reply to Daniel Robbins from comment #0)
> > What happens is that bash is emerged to /tmp/stage1root along with ncurses-6 to
> > /tmp/stage1root. but ncurses-6 is not installed to / first. So bash links
> > against ncurses-5 but there is no ncurses-5 library in /tmp/stage1root. This
> > is incorrect behavior.
> 
> I don't think it's possible to properly distinguish the dependencies without
> an EAPI extension as discussed in bug 317337.

Implementing my solution as described in the original bug report would fix the current behavior, which is a bug, without requiring new functionality intended for cross-compilers. This bug occurs in non-cross environments so it should not require a cross-specific feature to fix. Without commenting on 317337 (don't want to get distracted) it really doesn't seem to make sense to conflate these two issues.
Comment 9 Zac Medico gentoo-dev 2016-08-07 06:10:19 UTC
(In reply to Daniel Robbins from comment #8)
> This bug occurs in non-cross environments so
> it should not require a cross-specific feature to fix.

The description in comment #0 seems cross-specific, so can you re-frame the bug description in a way that doesn't involve ROOT?

Note that the slot-operator evaluation code already has some special HDEPEND handling here:

https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/dep/_slot_operator.py?h=portage-2.3.0#n68
Comment 10 Daniel Robbins 2016-08-07 20:24:40 UTC
It has nothing to do with cross-compiling and nowhere do I say it is related to cross-compiling. It is related to how Portage currently handles dependencies when ROOT is not / -- specifically, stage1 building with a non-/ ROOT. Please read my original description a bit more slowly. I have a lot of detailed information in there.
Comment 11 Zac Medico gentoo-dev 2016-08-07 21:00:12 UTC
I agree with comment #2. I prefer that this be considered as a case of cross-compiling where CHOST and CTARGET are equal. Doing otherwise introduces a special case that is redundant.
Comment 12 Daniel Robbins 2016-08-07 21:07:00 UTC
I replied directly to comment #2 in comment #3. And I think what I stated in comment #3 is correct and addresses the points made in comment #2. 

It is not cross-compilation. It is just a matter of consistently applying our definition of what DEPEND is. In a non-cross situation, a "build dependency" (ie. DEPEND) should always be installed to /, because / is what contains the currently-running instance of Gentoo. This includes both libraries and headers, but could also include *commands* (such as cmake) that are build dependencies.

I personally don't care specifically how you fix it in the code, with one condition -- I would consider it wrong to require bash or ncurses to be made "cross-compile aware" in order to have a stage1 build correctly. That is just wrong.

As long as you fix the behavior of DEPEND, which is currently not functioning properly with sub-slot dependencies, without introducing the requirement of using new *DEPEND variables, I would be fine with the solution.
Comment 13 Zac Medico gentoo-dev 2016-08-07 21:12:31 UTC
(In reply to Daniel Robbins from comment #12)
> As long as you fix the behavior of DEPEND, which is currently not
> functioning properly with sub-slot dependencies, without introducing the
> requirement of using new *DEPEND variables, I would be fine with the
> solution.

I feel that that doing as you suggest would lead to incorrect behavior for ebuilds that do the "right" thing and link against the ncurses inside ROOT=/tmp/stage1root.
Comment 14 Daniel Robbins 2016-08-07 21:19:00 UTC
Aha. I don't consider that to be the 'right' thing based on the definition of DEPEND.
That has not been the historical definition of how DEPEND is supposed to work.

The building is happening on /, so the DEPENDs should be satisfied on /. As I pointed out -- and I think you are totally missing this point -- some of the DEPENDs may be executables needed for compilation, like 'cmake'. This would be listed in DEPEND, if an ebuild requires it. This would ALWAYS need to be installed to the current root filesystem, regardless of the setting of ROOT. Because you actually need to execute this command.

The most consistent way to interpret DEPEND is how I described it -- DEPEND stuff gets installed to /, unconditionally. If you require other behavior, such as linking to what is in your ROOT, then introduce new functionality for *that*.

Let's zoom out a moment and also consider that non-/ ROOT is used very infrequently, and in a non-cross situation is used almost exclusively for building stage1's. We need to make sure that there is sane behavior in a non-cross situation. For stage1 building, the behavior I described is much more sane than trying to link against what is in ROOT. 

And again, remember that DEPEND is used for more things than just libraries.
Comment 15 Zac Medico gentoo-dev 2016-08-07 21:33:56 UTC
(In reply to Daniel Robbins from comment #0)
> What I believe should happen is as follows, which not only respects the
> existing definition of sub-slot dependencies (in other words, this isn't a
> new feature but just ensuring that things work as already specified in the
> man page) but also has the wonderful side-effect of installing non-broken
> binaries: "ROOT=/tmp/stage1root emerge bash" would find that ncurses:0/5 is
> installed and this would satisfy bash's DEPEND of
> >=sys-libs/ncurses-5.2-r2:0=. bash would be built and link against this
> installed version of ncurses, and emerge would then ensure that an
> ncurses:0/5 is merged to /tmp/stage1root due to >=sys-libs/ncurses-5.2-r2:0=
> in RDEPEND (and per the existing definition of sub-slot deps, emerge will
> ensure that "a  matching  package  with slot and sub-slot equal to the slot
> and sub-slot of the best installed version at the time the package was
> installed is available." This would be ncurses:0/5.)

This bit about ensuring that the same version is selected for both DEPEND and RDEPEND is a bit ugly. I would be much nicer to have a separate *DEPEND variable (or labels as discussed in bug 201499) to indicate dependencies that should have this sort of property.
Comment 16 Daniel Robbins 2016-08-07 22:11:31 UTC
It seems quite elegant to me, and also seems to actually be correct.

Just have evaluation of sub-slots for DEPEND (ie. "ncurses:0/5") always stick to / in a ROOT!=/ situation. Keep evaluation of sub-slots for RDEPEND as-is, where ROOT affects where they are resolved. Problem solved. One fix. No biggie. This will implement the suggested behavior I described in comment #0.

This fix, unlike current behavior, does not contradict the historical and current definition of DEPEND and sub-slot dependencies.

Then, if you need ebuilds to link against differing locations for cross-compiling, implement *DEPEND functionality to specifically support this use case properly.

You will do much better to respect the historical definition of DEPEND rather than redefine its behavior to be cross-friendly. It just creates problems to do otherwise. DEPEND and RDEPEND were not created for a cross-compile situation.
Comment 17 Zac Medico gentoo-dev 2016-08-07 22:55:29 UTC
Basically, the code will have to intersect DEPEND and RDEPEND, and classify the intersection as a special type of dependency which stipulates that whatever version it matches must be identical in both ROOTs. What about USE flags? Must the USE settings also be identical?
Comment 18 Daniel Robbins 2016-08-07 22:58:27 UTC
I don't think what you are suggesting is actually necessary.

Just bind DEPEND, including sub-slot resolution for DEPEND, to /. Remember my example had RDEPEND also set to the same value which will take care of the non-/ dependencies.

You should not need a special kind of dependency. You should be able to alter the DEPEND behavior based on the setting of ROOT.
Comment 19 Daniel Robbins 2016-08-07 22:59:11 UTC
And yes, if you are 'looking to /' for things, then you would also want to 'look to /' when matching USE deps.
Comment 20 Zac Medico gentoo-dev 2016-08-07 23:03:29 UTC
(In reply to Daniel Robbins from comment #18)
> Just bind DEPEND, including sub-slot resolution for DEPEND, to /. Remember
> my example had RDEPEND also set to the same value which will take care of
> the non-/ dependencies.

But there's no rule which says that the same version has to be selected for DEPEND and RDEPEND. So, you can end up having bash installed together ncurses-6 inside ROOT=/tmp/stage1root, but built against ncurses-5 and having a broken soname dependency at runtime.
Comment 21 Daniel Robbins 2016-08-07 23:06:42 UTC
I don't think that's necessary. Let me step through the scenarios and confirm. Will get back to you in the next few days (dinner time and work distractions for the next days.)
Comment 22 Alexis Ballier gentoo-dev 2016-08-11 21:07:03 UTC
Maybe it'd be simpler that 'ROOT=/somewhere emerge ...' updates any DEPEND to the best visible version in /. This would also make packages lacking := deps work, shouldn't hurt cross compilation (granted / is replaced by SYSROOT), and is what I have experienced with 'ROOT=/somewhere emerge -uDN --with-bdeps=y ...'.