Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 449094

Summary: Support for "passthru subslots" to rebuild all the reverse dependency graph.
Product: Gentoo Hosted Projects Reporter: Sergei Trofimovich (RETIRED) <slyfox>
Component: PMS/EAPIAssignee: PMS/EAPI <pms>
Status: CONFIRMED ---    
Severity: normal CC: aballier, anton.kochkov, axs, bog, bugs+gentoo, crockabiscuit, dlan, esigra, haskell, hendrik, jj, kentnl, mrueg, nikoli, pacho, sam, solpeth, tomwij, toralf, zerochaos
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 174380, 454430, 456712    

Description Sergei Trofimovich (RETIRED) gentoo-dev 2012-12-28 21:28:10 UTC
[ long description, but I hope to describe some problems ]

1. Currently subslot dependencies allow rebuilding
only one step further in reverse dependencies:

     It is handly for C-like libraries when they change ABI.
         Example 1:
             libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so
              libbar.ebuild: RDEPEND=libfoo:=

     Once you upgrade libfoo you get libbar forced rebuild.

2. It does not work so well when you provide such library as a virtual.

    Example 2:
        libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so
         virt.ebuild: RDEPEND=libfoo:=
          libbar.ebuild: RDEPEND=virt:=

    Here we need exactly two steps to lookup package breakages.

In this case virt itself defines if it has the same subslot as
it's RDEPEND, thus we need some special SLOT for it.

    virt.ebuild: SLOT=0/${PV}-sum-rdepend's-subslots-operator

3. And the worst case (the dev-haskell/* case), when package's
   ABI is a sum of ABIs of used packages (recursively).

    Example 3:
        dev-vcs/darcs: RDEPEND="dev-haskell/text:= ghc:="
         dev-haskell/text: RDEPEND="dev-haskell/random:= ghc:="
          dev-haskell/random: RDEPEND="ghc:="

In this case when upgrading dev-haskell/random I'd like to schedule for
rebuilding all the reverse depgraph: dev-haskell/text dev-vcs/darcs (and all other transitive reverse RDEPENDs of dev-haskell/text).

In haskell case it's dev-haskell/random defines whether
dev-haskell/text and dev-vcs/darcs will break
after dev-haskell/random upgrade.

Thus we need to somehow mark it's SLOT= as transitively fragile.

    dev-haskell/random: SLOT=0/${PV}-transitively-fragile-operator

in summary, cases 2. and 3. are a bit different:

- in 2. we need to mark intermediate nodes as special
- in 3. we need to mark root nodes as special
Comment 1 Ian Stakenvicius (RETIRED) gentoo-dev 2013-01-30 17:42:00 UTC
(In reply to comment #0)
> 2. It does not work so well when you provide such library as a virtual.
> 
>     Example 2:
>         libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so
>          virt.ebuild: RDEPEND=libfoo:=
>           libbar.ebuild: RDEPEND=virt:=
> 
>     Here we need exactly two steps to lookup package breakages.
> 
> In this case virt itself defines if it has the same subslot as
> it's RDEPEND, thus we need some special SLOT for it.
> 
>     virt.ebuild: SLOT=0/${PV}-sum-rdepend's-subslots-operator
> 
> 3. And the worst case (the dev-haskell/* case), when package's
>    ABI is a sum of ABIs of used packages (recursively).
> [...] 
> in summary, cases 2. and 3. are a bit different:
> 
> - in 2. we need to mark intermediate nodes as special
> - in 3. we need to mark root nodes as special


I may need some clarification on #3 as I'm not sure I understand the needs of that case, however I've been mulling how to deal with this in my head for some time.

I think a solution that might address this would be a flag that could be added to both *DEPEND atoms -and- SLOT= in the ebuild -- something that would essentially append each flagged dep's slot change to this ebuild's subslot (therefore propagating the change).

Say we use '%' for this.  Examples:

     Example 2:
         libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so
          virt.ebuild: 
                  SLOT="0/%"
                  RDEPEND="libfoo:=% libfoobar:=%"
           libbar.ebuild: RDEPEND=virt:=



    Example 3:
        dev-vcs/darcs: RDEPEND="dev-haskell/text:= ghc:="
         dev-haskell/text: 
                  SLOT="0/whatever%"
                  RDEPEND="dev-haskell/random:=% ghc:="
          dev-haskell/random: RDEPEND="ghc:="


In terms of Example-3, I believe the desired result could also be established by adding 'dev-haskell/random:=' to RDEPEND of dev-vcs/darcs , correct?  (except of course this isn't an actual direct RDEPEND and so is not something we want to do)

I can't forsee there being a way to somehow mark dev-haskell/random as special such that dev-vcs/darcs would be rebuilt automatically when it changes, WITHOUT putting marks in dev-haskell/text (or -something- that dev-vcs/darcs directly depends on).  However, since dev-haskell/text will be triggered for rebuild, passing through the subslot changes in this manner I think will suffice, correct?
Comment 2 Sergei Trofimovich (RETIRED) gentoo-dev 2013-01-31 09:09:18 UTC
(In reply to comment #1)
> (In reply to comment #0)
> > 2. It does not work so well when you provide such library as a virtual.
> > 
> >     Example 2:
> >         libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so
> >          virt.ebuild: RDEPEND=libfoo:=
> >           libbar.ebuild: RDEPEND=virt:=
> > 
> >     Here we need exactly two steps to lookup package breakages.
> > 
> > In this case virt itself defines if it has the same subslot as
> > it's RDEPEND, thus we need some special SLOT for it.
> > 
> >     virt.ebuild: SLOT=0/${PV}-sum-rdepend's-subslots-operator
> > 
> > 3. And the worst case (the dev-haskell/* case), when package's
> >    ABI is a sum of ABIs of used packages (recursively).
> > [...] 
> > in summary, cases 2. and 3. are a bit different:
> > 
> > - in 2. we need to mark intermediate nodes as special
> > - in 3. we need to mark root nodes as special
> 
> 
> I may need some clarification on #3 as I'm not sure I understand the needs
> of that case, however I've been mulling how to deal with this in my head for
> some time.
> 
> I think a solution that might address this would be a flag that could be
> added to both *DEPEND atoms -and- SLOT= in the ebuild -- something that
> would essentially append each flagged dep's slot change to this ebuild's
> subslot (therefore propagating the change).
> 
> Say we use '%' for this.  Examples:
> 
>      Example 2:
>          libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so
>           virt.ebuild: 
>                   SLOT="0/%"
>                   RDEPEND="libfoo:=% libfoobar:=%"
>            libbar.ebuild: RDEPEND=virt:=
> 
> 
> 
>     Example 3:
>         dev-vcs/darcs: RDEPEND="dev-haskell/text:= ghc:="
>          dev-haskell/text: 
>                   SLOT="0/whatever%"
>                   RDEPEND="dev-haskell/random:=% ghc:="
>           dev-haskell/random: RDEPEND="ghc:="
> 
> 
> In terms of Example-3, I believe the desired result could also be
> established by adding 'dev-haskell/random:=' to RDEPEND of dev-vcs/darcs ,
> correct?  (except of course this isn't an actual direct RDEPEND and so is
> not something we want to do)

Yeah.

> I can't forsee there being a way to somehow mark dev-haskell/random as
> special such that dev-vcs/darcs would be rebuilt automatically when it
> changes, WITHOUT putting marks in dev-haskell/text (or -something- that
> dev-vcs/darcs directly depends on).  However, since dev-haskell/text will be
> triggered for rebuild, passing through the subslot changes in this manner I
> think will suffice, correct?

It means your plan is to adjust depends like that (i've build deeper dep chain):

>         something-using-darcs-binary:
>                   SLOT="0"
>                   RDEPEND="dev-vcs/darcs"
>         something-using-darcs-library:
>                   SLOT="0/${PV}%"
>                   RDEPEND="dev-vcs/darcs:=% ghc:="
>         dev-vcs/darcs:
>                   SLOT="0/${PV}%"
>                   RDEPEND="dev-haskell/text:=% ghc:="
>          dev-haskell/text: 
>                   SLOT="0/${PV}%"
>                   RDEPEND="dev-haskell/random:=% ghc:="
>           dev-haskell/random:
>                   SLOT="0/${PV}%"
>                   RDEPEND="ghc:="

In thins case upgrade of any of ['dev-haskell/random', 'dev-haskell/text', 'dev-vcs/darcs'] will trigger rebuild of 'something-using-darcs-library',
but not 'something-using-darcs-binary', right?

If yes, then it sounds great.
Comment 3 Ian Stakenvicius (RETIRED) gentoo-dev 2013-01-31 13:55:21 UTC
(In reply to comment #2)
> 
> It means your plan is to adjust depends like that (i've build deeper dep
> chain):
> 
> >         something-using-darcs-binary:
> >                   SLOT="0"
> >                   RDEPEND="dev-vcs/darcs"
> >         something-using-darcs-library:
> >                   SLOT="0/${PV}%"
> >                   RDEPEND="dev-vcs/darcs:=% ghc:="
> >         dev-vcs/darcs:
> >                   SLOT="0/${PV}%"
> >                   RDEPEND="dev-haskell/text:=% ghc:="
> >          dev-haskell/text: 
> >                   SLOT="0/${PV}%"
> >                   RDEPEND="dev-haskell/random:=% ghc:="
> >           dev-haskell/random:
> >                   SLOT="0/${PV}%"
> >                   RDEPEND="ghc:="
> 
> In thins case upgrade of any of ['dev-haskell/random', 'dev-haskell/text',
> 'dev-vcs/darcs'] will trigger rebuild of 'something-using-darcs-library',
> but not 'something-using-darcs-binary', right?
> 
> If yes, then it sounds great.

In the example above:

- dev-vcs/darcs will be rebuilt.  

- something-using-darcs-library will be rebuilt.  FURTHER, because something-using-darcs-library has % in SLOT, anything that :='s on something-using-darcs-library would also rebuild.  If this is unnecessary/unintended you could just do:
        something-using-darcs-library:
                   SLOT="0/${PV}" <-- don't know if you need to subslot?
                   RDEPEND="dev-vcs/darcs:= ghc:="

- something-using-darcs-binary would not rebuild.

So in summary, yes.

-----

The one thing i'm not sure of is whether this new syntax might possibly be invalid due to muddling with the invariantness of SLOT= ...  I *THINK* that the '%' will only need to expand out in the SLOT= values saved within the VDB (and of course at emerge deptree resolution time) and so it won't break metadata generation or anything like that.  Will need to discuss with Zac to confirm though.
Comment 4 Ciaran McCreesh 2013-01-31 13:58:41 UTC
It'll need new special behaviour for implementation. Just shoving in an arbitrary magic token on the PM side won't be enough.
Comment 5 Ian Stakenvicius (RETIRED) gentoo-dev 2013-01-31 14:07:56 UTC
(In reply to comment #4)
> It'll need new special behaviour for implementation. Just shoving in an
> arbitrary magic token on the PM side won't be enough.

Oh yes -- there will definitely be a rather significant implementation cost to this.  However, perculating the slot-operator behaviour from root to leaf (when desirable) is worth it imo.  

My concern right now is related to legality of this new behaviour compared to other parts of the PMS spec.  IE, if in current PMS the SLOT value *needs* to be its final value in metadata (ie, PMs depend on this and it's defined as such), then the whole idea is illegal.
Comment 6 Ciaran McCreesh 2013-01-31 14:10:56 UTC
Well... The way things are currently, yes, you'll be breaking various invariants. But new features quite often do...

What I've not seen addressed with this kind of approach is how binary packages will work.
Comment 7 Ian Stakenvicius (RETIRED) gentoo-dev 2013-01-31 14:24:41 UTC
(In reply to comment #6)
> What I've not seen addressed with this kind of approach is how binary
> packages will work.

Binary packages as in firefox-bin or binary packages as in quickpkg'd items?  (firefox-bin is easy enough, it just won't have any slot-operators on its deps)

Quickpkg'd items is another story..  I expect that for proper resolution the full SLOT value (as would be saved in the vdb) would need to be used to tag the package (same thing as slot/sublsot now).  

Given that cases requiring the slot-operator values to perculate up would theoretically only need to occur when there is a binary dependency on that relationship, it would make sense that two binary packages with different fully-rolled-out SLOT='s would actually be different and therefore only the properly matching one could be installed and be expected to work.

It would be interesting to see how quickpkg'd dev-haskell stuff works now -- Taking the example above, I expect if dev-haskell/random is bumped then the previous binpkg's for both dev-haskell/text and dev-cvs/darcs would be invalid against the new dev-haskell/random, correct?  ie, all three would need to be quickpkg'd and their binpkg's reinstalled.
Comment 8 Ciaran McCreesh 2013-01-31 15:45:58 UTC
The latter (and not just quickpkg, but binaries being distributed in general).

It's not necessarily a problem. We'd have to be sure nothing assumed that "% appears in installable packages, and an expanded value appears in installed packages".
Comment 9 Sergei Trofimovich (RETIRED) gentoo-dev 2013-01-31 18:01:56 UTC
> It would be interesting to see how quickpkg'd dev-haskell stuff works now --
> Taking the example above, I expect if dev-haskell/random is bumped then the
> previous binpkg's for both dev-haskell/text and dev-cvs/darcs would be
> invalid against the new dev-haskell/random, correct?  ie, all three would
> need to be quickpkg'd and their binpkg's reinstalled.

Haskell binpkgs are very fragile, yes.
Comment 10 Rick Farina (Zero_Chaos) gentoo-dev 2013-05-27 21:57:09 UTC
Honestly this was a limitation of the original SLOT implementation as well not just sub-slots.

One of the many reasons why I need slot/sub-slot passthrough is for binpkgs actually.  There is a major issue when I compile kernel modules against kernel version 3.x.x and the user has kernel 3.x.y installed.  Unfortunately it's not possible to make kernel modules dep on the kernel they were built against since right now they only dep virtual/linux-sources and that never really changes slot/subslot.

that's a very short description but I hope it makes sense.
Comment 11 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-09-26 18:42:55 UTC
Yes, I've been thinking through kernel build packages some time ago with other people (including Zero_Chaos) and "passthru (sub)slots" is what is kind of missing to make it work in a sane way to have reverse dependency modules automatically rebuild against a virtual for the kernel build packages.

I'm not sure how binary packages are relevant here, they can't get rebuild?
Comment 12 Rick Farina (Zero_Chaos) gentoo-dev 2013-09-27 00:26:10 UTC
(In reply to Tom Wijsman (TomWij) from comment #11)
> I'm not sure how binary packages are relevant here, they can't get rebuild?

emerge @preserved-rebuild will cause a manual rebuild of things unless you have binpkg-only set, I would expect this to work the same.

Forcing binary packages only has a lot of limitations, I would expect most users don't want this, although --rebuilt-binaries=y will reinstall if the binary package is newer so the build host rebuilds and the binary users just pull a fresh package.
Comment 13 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-02 09:01:34 UTC
This can't work for virtual packages and there's no point in even trying to use it there. It's a step backwards, toward PROVIDES= and other magic.

Usually virtual specifies a few packages that satisfy it using any-of dep, like:

  || ( foo bar )

Now, if you switch the provider from foo to bar, how should it all work? Should it force rebuilding all the revdeps? Or maybe the 'passthru slot' should move transparently from subslot of one provider to those of another provider?

What if different providers have different subslots? What if you have a binary package that works with different subslots of different providers?

While subslot (and slot) operator rebuilds cause trouble already, with users being confused and portage being buggy, adding indefinite number of layers of indirection is going to be even worse.

I can understand that you may need it for other stuff. But please don't make virtuals a mess with it. A much simpler solution would be having versioned virtuals for SONAMEs (we often need to version them anyway for versioned deps), with subslot deps on the deps, e.g.:

  SLOT="0/11"
  RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )"
Comment 14 Sergei Trofimovich (RETIRED) gentoo-dev 2013-10-02 17:21:13 UTC
(In reply to Michał Górny from comment #13)
> This can't work for virtual packages and there's no point in even trying to
> use it there. It's a step backwards, toward PROVIDES= and other magic.
> 
> Usually virtual specifies a few packages that satisfy it using any-of dep,
> like:
> 
>   || ( foo bar )
>
> Now, if you switch the provider from foo to bar, how should it all work?
> Should it force rebuilding all the revdeps? Or maybe the 'passthru slot'
> should move transparently from subslot of one provider to those of another
> provider?
> 
> What if different providers have different subslots? What if you have a
> binary package that works with different subslots of different providers?
> 
> While subslot (and slot) operator rebuilds cause trouble already, with users
> being confused and portage being buggy, adding indefinite number of layers
> of indirection is going to be even worse.
>
> I can understand that you may need it for other stuff. But please don't make
> virtuals a mess with it. A much simpler solution would be having versioned
> virtuals for SONAMEs (we often need to version them anyway for versioned
> deps), with subslot deps on the deps, e.g.:
> 
>   SLOT="0/11"
>   RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )"

I don't understand that example. What does it do?
I guess those 11 and 12 are SONAME versions.
 "bar:0/11 -> bar:0/12" upgrade will break virtual users w/o them noticing.
(for those who happened to link against "bar:0/11" and have that virtual installed)

Seems your "problem" is's not specific to virtuals. Nobody forbids you
to put exactly that into usual ebuild.

    || ( foo:= bar:= ) # to make syntax more valid for current portage

How does portage handle that? It needs to know exactly what alternative
is taken by built package to make correct rebuild decisions.

The bug is about the packages which ABIs are defined by their depends.
Comment 15 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-02 17:35:21 UTC
(In reply to Sergei Trofimovich from comment #14)
> (In reply to Michał Górny from comment #13)
> >   SLOT="0/11"
> >   RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )"
> 
> I don't understand that example. What does it do?
> I guess those 11 and 12 are SONAME versions.
>  "bar:0/11 -> bar:0/12" upgrade will break virtual users w/o them noticing.
> (for those who happened to link against "bar:0/11" and have that virtual
> installed)

No, it won't. Simply because '11' and '12' are subslots in which some other library did change (e.g. libsystemd* in systemd for virtual/udev).
Comment 16 Ian Stakenvicius (RETIRED) gentoo-dev 2013-10-02 17:40:20 UTC
(In reply to Michał Górny from comment #13)
> This can't work for virtual packages and there's no point in even trying to
> use it there. It's a step backwards, toward PROVIDES= and other magic.
> 
> Usually virtual specifies a few packages that satisfy it using any-of dep,
> like:
> 
>   || ( foo bar )
> 
> Now, if you switch the provider from foo to bar, how should it all work?
> Should it force rebuilding all the revdeps? Or maybe the 'passthru slot'
> should move transparently from subslot of one provider to those of another
> provider?

Given that "changing providers" could mean many different things:

1- if emerge -C foo && emerge bar , yes I expect that the only way this would work in a stable fashion would be for the passthrough-subslot to change in the virtual, and therefore rdeps of the virtual would be triggered for rebuild.  ESPECIALLY, since most virtuals do not have version controls determining ABI equivalence throughout (rather, just API equivalence, if even that)

2- if just emerging another one in the list (because for whatever reason they don't block eachother), then I agree that it shouldn't trigger a rebuild.  I'd need to test this, but I expect that emerging a second package within an RDEPEND="||( a:= b:= )" right now would not trigger a rebuild, so if the virtual itself isn't re-emerged then the recorded "passthrough" subslot wouldn't change and so it'd all work out.  As I said though, more research is needed.


> What if different providers have different subslots? What if you have a
> binary package that works with different subslots of different providers?

subslots are just indicators for change.  They are not a version in and of themselves and cannot be relied upon as such.  So, what subslot value any given dep of a virtual has, really doesn't matter in terms of what the other deps have.

A binary package only gets to work with what it's built against, but the subslot doesn't determine that any better than a virtual without versions determines it now.

 
> I can understand that you may need it for other stuff. But please don't make
> virtuals a mess with it. A much simpler solution would be having versioned
> virtuals for SONAMEs (we often need to version them anyway for versioned
> deps), with subslot deps on the deps, e.g.:
> 
>   SLOT="0/11"
>   RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )"

The issue with this is that we would then need to have an ebuild for every virtual that specifies all valid combinations of packages via their subslot values.  That's a lot more ebuilds to have to work with.

Not that this is a bad thing, mind you -- it would be nice for a virtual to have an easily specified upper-bound on the packages that provide it, and revbumping to add new versions doesn't seem like a bad idea to me.  Of course there's the other advantage that it is doable now, without any new PM features or EAPI additions.
Comment 17 Alexis Ballier gentoo-dev 2017-01-31 09:31:34 UTC
*** Bug 607780 has been marked as a duplicate of this bug. ***
Comment 18 Alexis Ballier gentoo-dev 2017-01-31 09:32:18 UTC
*** Bug 607782 has been marked as a duplicate of this bug. ***
Comment 19 Alexis Ballier gentoo-dev 2017-01-31 09:32:52 UTC
*** Bug 607784 has been marked as a duplicate of this bug. ***
Comment 20 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-02-11 21:20:58 UTC
Did anyone come up with a reasonable idea of how this could actually be made working? With growing number of ABI-incompatible library forks, a solution to the problem of writing sane dependencies would be quite welcome, and I think some kind of 'passthru subslots' on virtuals would be quite readable on developer side.

However, for that to work we really need a good plan. If we aren't going to have one in near time, I think we ought to look for other solutions.
Comment 21 crocket 2019-01-31 03:45:02 UTC
Is this fixed with EAPI=7?
Comment 22 Sergei Trofimovich (RETIRED) gentoo-dev 2019-01-31 07:19:17 UTC
(In reply to crocket from comment #21)
> Is this fixed with EAPI=7?

No.
Comment 23 Sergei Trofimovich (RETIRED) gentoo-dev 2019-11-28 07:17:50 UTC
*** Bug 701374 has been marked as a duplicate of this bug. ***
Comment 24 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2019-11-29 09:06:00 UTC
(In reply to Michał Górny from comment #20)
> Did anyone come up with a reasonable idea of how this could actually be made
> working? With growing number of ABI-incompatible library forks, a solution
> to the problem of writing sane dependencies would be quite welcome, and I
> think some kind of 'passthru subslots' on virtuals would be quite readable
> on developer side.
> 
> However, for that to work we really need a good plan. If we aren't going to
> have one in near time, I think we ought to look for other solutions.

I don't think it can be solved entirely in DEPEND* syntax.

I get the impression that an ebuild needs some sort of mechanism to semi-programmatically declare "What is my ABI" in terms of its parts.

subslots are limited primarily in that they have to be determined statically. 

I'm just spit-balling here, but maybe something like:

VARY="
  use: foo,bar,baz
  depend: dev-lang/foo
"

Which could also be written:

VARY="
   use: *
   depend: *
"

Both of which indicate "my ABI changes when these values change"

These aspects could be combined to produce a unique SHA1 ( or even MD5, its not a security aspect, and the collision space is unique to a package ) that represents the variable aspects of the ABI.

At least, that approach maps to how this works under-the-hood in a plethora of modern languages.

Then the "rebuild" condition is determined by detecting if a packages SHA1 would change.