[ long description, but I hope to describe some problems ] 1. Currently subslot dependencies allow rebuilding only one step further in reverse dependencies: It is handly for C-like libraries when they change ABI. Example 1: libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so libbar.ebuild: RDEPEND=libfoo:= Once you upgrade libfoo you get libbar forced rebuild. 2. It does not work so well when you provide such library as a virtual. Example 2: libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so virt.ebuild: RDEPEND=libfoo:= libbar.ebuild: RDEPEND=virt:= Here we need exactly two steps to lookup package breakages. In this case virt itself defines if it has the same subslot as it's RDEPEND, thus we need some special SLOT for it. virt.ebuild: SLOT=0/${PV}-sum-rdepend's-subslots-operator 3. And the worst case (the dev-haskell/* case), when package's ABI is a sum of ABIs of used packages (recursively). Example 3: dev-vcs/darcs: RDEPEND="dev-haskell/text:= ghc:=" dev-haskell/text: RDEPEND="dev-haskell/random:= ghc:=" dev-haskell/random: RDEPEND="ghc:=" In this case when upgrading dev-haskell/random I'd like to schedule for rebuilding all the reverse depgraph: dev-haskell/text dev-vcs/darcs (and all other transitive reverse RDEPENDs of dev-haskell/text). In haskell case it's dev-haskell/random defines whether dev-haskell/text and dev-vcs/darcs will break after dev-haskell/random upgrade. Thus we need to somehow mark it's SLOT= as transitively fragile. dev-haskell/random: SLOT=0/${PV}-transitively-fragile-operator in summary, cases 2. and 3. are a bit different: - in 2. we need to mark intermediate nodes as special - in 3. we need to mark root nodes as special
(In reply to comment #0) > 2. It does not work so well when you provide such library as a virtual. > > Example 2: > libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so > virt.ebuild: RDEPEND=libfoo:= > libbar.ebuild: RDEPEND=virt:= > > Here we need exactly two steps to lookup package breakages. > > In this case virt itself defines if it has the same subslot as > it's RDEPEND, thus we need some special SLOT for it. > > virt.ebuild: SLOT=0/${PV}-sum-rdepend's-subslots-operator > > 3. And the worst case (the dev-haskell/* case), when package's > ABI is a sum of ABIs of used packages (recursively). > [...] > in summary, cases 2. and 3. are a bit different: > > - in 2. we need to mark intermediate nodes as special > - in 3. we need to mark root nodes as special I may need some clarification on #3 as I'm not sure I understand the needs of that case, however I've been mulling how to deal with this in my head for some time. I think a solution that might address this would be a flag that could be added to both *DEPEND atoms -and- SLOT= in the ebuild -- something that would essentially append each flagged dep's slot change to this ebuild's subslot (therefore propagating the change). Say we use '%' for this. Examples: Example 2: libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so virt.ebuild: SLOT="0/%" RDEPEND="libfoo:=% libfoobar:=%" libbar.ebuild: RDEPEND=virt:= Example 3: dev-vcs/darcs: RDEPEND="dev-haskell/text:= ghc:=" dev-haskell/text: SLOT="0/whatever%" RDEPEND="dev-haskell/random:=% ghc:=" dev-haskell/random: RDEPEND="ghc:=" In terms of Example-3, I believe the desired result could also be established by adding 'dev-haskell/random:=' to RDEPEND of dev-vcs/darcs , correct? (except of course this isn't an actual direct RDEPEND and so is not something we want to do) I can't forsee there being a way to somehow mark dev-haskell/random as special such that dev-vcs/darcs would be rebuilt automatically when it changes, WITHOUT putting marks in dev-haskell/text (or -something- that dev-vcs/darcs directly depends on). However, since dev-haskell/text will be triggered for rebuild, passing through the subslot changes in this manner I think will suffice, correct?
(In reply to comment #1) > (In reply to comment #0) > > 2. It does not work so well when you provide such library as a virtual. > > > > Example 2: > > libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so > > virt.ebuild: RDEPEND=libfoo:= > > libbar.ebuild: RDEPEND=virt:= > > > > Here we need exactly two steps to lookup package breakages. > > > > In this case virt itself defines if it has the same subslot as > > it's RDEPEND, thus we need some special SLOT for it. > > > > virt.ebuild: SLOT=0/${PV}-sum-rdepend's-subslots-operator > > > > 3. And the worst case (the dev-haskell/* case), when package's > > ABI is a sum of ABIs of used packages (recursively). > > [...] > > in summary, cases 2. and 3. are a bit different: > > > > - in 2. we need to mark intermediate nodes as special > > - in 3. we need to mark root nodes as special > > > I may need some clarification on #3 as I'm not sure I understand the needs > of that case, however I've been mulling how to deal with this in my head for > some time. > > I think a solution that might address this would be a flag that could be > added to both *DEPEND atoms -and- SLOT= in the ebuild -- something that > would essentially append each flagged dep's slot change to this ebuild's > subslot (therefore propagating the change). > > Say we use '%' for this. Examples: > > Example 2: > libfoo.ebuild: SLOT=0/0.1.so -> SLOT=0/0.2.so > virt.ebuild: > SLOT="0/%" > RDEPEND="libfoo:=% libfoobar:=%" > libbar.ebuild: RDEPEND=virt:= > > > > Example 3: > dev-vcs/darcs: RDEPEND="dev-haskell/text:= ghc:=" > dev-haskell/text: > SLOT="0/whatever%" > RDEPEND="dev-haskell/random:=% ghc:=" > dev-haskell/random: RDEPEND="ghc:=" > > > In terms of Example-3, I believe the desired result could also be > established by adding 'dev-haskell/random:=' to RDEPEND of dev-vcs/darcs , > correct? (except of course this isn't an actual direct RDEPEND and so is > not something we want to do) Yeah. > I can't forsee there being a way to somehow mark dev-haskell/random as > special such that dev-vcs/darcs would be rebuilt automatically when it > changes, WITHOUT putting marks in dev-haskell/text (or -something- that > dev-vcs/darcs directly depends on). However, since dev-haskell/text will be > triggered for rebuild, passing through the subslot changes in this manner I > think will suffice, correct? It means your plan is to adjust depends like that (i've build deeper dep chain): > something-using-darcs-binary: > SLOT="0" > RDEPEND="dev-vcs/darcs" > something-using-darcs-library: > SLOT="0/${PV}%" > RDEPEND="dev-vcs/darcs:=% ghc:=" > dev-vcs/darcs: > SLOT="0/${PV}%" > RDEPEND="dev-haskell/text:=% ghc:=" > dev-haskell/text: > SLOT="0/${PV}%" > RDEPEND="dev-haskell/random:=% ghc:=" > dev-haskell/random: > SLOT="0/${PV}%" > RDEPEND="ghc:=" In thins case upgrade of any of ['dev-haskell/random', 'dev-haskell/text', 'dev-vcs/darcs'] will trigger rebuild of 'something-using-darcs-library', but not 'something-using-darcs-binary', right? If yes, then it sounds great.
(In reply to comment #2) > > It means your plan is to adjust depends like that (i've build deeper dep > chain): > > > something-using-darcs-binary: > > SLOT="0" > > RDEPEND="dev-vcs/darcs" > > something-using-darcs-library: > > SLOT="0/${PV}%" > > RDEPEND="dev-vcs/darcs:=% ghc:=" > > dev-vcs/darcs: > > SLOT="0/${PV}%" > > RDEPEND="dev-haskell/text:=% ghc:=" > > dev-haskell/text: > > SLOT="0/${PV}%" > > RDEPEND="dev-haskell/random:=% ghc:=" > > dev-haskell/random: > > SLOT="0/${PV}%" > > RDEPEND="ghc:=" > > In thins case upgrade of any of ['dev-haskell/random', 'dev-haskell/text', > 'dev-vcs/darcs'] will trigger rebuild of 'something-using-darcs-library', > but not 'something-using-darcs-binary', right? > > If yes, then it sounds great. In the example above: - dev-vcs/darcs will be rebuilt. - something-using-darcs-library will be rebuilt. FURTHER, because something-using-darcs-library has % in SLOT, anything that :='s on something-using-darcs-library would also rebuild. If this is unnecessary/unintended you could just do: something-using-darcs-library: SLOT="0/${PV}" <-- don't know if you need to subslot? RDEPEND="dev-vcs/darcs:= ghc:=" - something-using-darcs-binary would not rebuild. So in summary, yes. ----- The one thing i'm not sure of is whether this new syntax might possibly be invalid due to muddling with the invariantness of SLOT= ... I *THINK* that the '%' will only need to expand out in the SLOT= values saved within the VDB (and of course at emerge deptree resolution time) and so it won't break metadata generation or anything like that. Will need to discuss with Zac to confirm though.
It'll need new special behaviour for implementation. Just shoving in an arbitrary magic token on the PM side won't be enough.
(In reply to comment #4) > It'll need new special behaviour for implementation. Just shoving in an > arbitrary magic token on the PM side won't be enough. Oh yes -- there will definitely be a rather significant implementation cost to this. However, perculating the slot-operator behaviour from root to leaf (when desirable) is worth it imo. My concern right now is related to legality of this new behaviour compared to other parts of the PMS spec. IE, if in current PMS the SLOT value *needs* to be its final value in metadata (ie, PMs depend on this and it's defined as such), then the whole idea is illegal.
Well... The way things are currently, yes, you'll be breaking various invariants. But new features quite often do... What I've not seen addressed with this kind of approach is how binary packages will work.
(In reply to comment #6) > What I've not seen addressed with this kind of approach is how binary > packages will work. Binary packages as in firefox-bin or binary packages as in quickpkg'd items? (firefox-bin is easy enough, it just won't have any slot-operators on its deps) Quickpkg'd items is another story.. I expect that for proper resolution the full SLOT value (as would be saved in the vdb) would need to be used to tag the package (same thing as slot/sublsot now). Given that cases requiring the slot-operator values to perculate up would theoretically only need to occur when there is a binary dependency on that relationship, it would make sense that two binary packages with different fully-rolled-out SLOT='s would actually be different and therefore only the properly matching one could be installed and be expected to work. It would be interesting to see how quickpkg'd dev-haskell stuff works now -- Taking the example above, I expect if dev-haskell/random is bumped then the previous binpkg's for both dev-haskell/text and dev-cvs/darcs would be invalid against the new dev-haskell/random, correct? ie, all three would need to be quickpkg'd and their binpkg's reinstalled.
The latter (and not just quickpkg, but binaries being distributed in general). It's not necessarily a problem. We'd have to be sure nothing assumed that "% appears in installable packages, and an expanded value appears in installed packages".
> It would be interesting to see how quickpkg'd dev-haskell stuff works now -- > Taking the example above, I expect if dev-haskell/random is bumped then the > previous binpkg's for both dev-haskell/text and dev-cvs/darcs would be > invalid against the new dev-haskell/random, correct? ie, all three would > need to be quickpkg'd and their binpkg's reinstalled. Haskell binpkgs are very fragile, yes.
Honestly this was a limitation of the original SLOT implementation as well not just sub-slots. One of the many reasons why I need slot/sub-slot passthrough is for binpkgs actually. There is a major issue when I compile kernel modules against kernel version 3.x.x and the user has kernel 3.x.y installed. Unfortunately it's not possible to make kernel modules dep on the kernel they were built against since right now they only dep virtual/linux-sources and that never really changes slot/subslot. that's a very short description but I hope it makes sense.
Yes, I've been thinking through kernel build packages some time ago with other people (including Zero_Chaos) and "passthru (sub)slots" is what is kind of missing to make it work in a sane way to have reverse dependency modules automatically rebuild against a virtual for the kernel build packages. I'm not sure how binary packages are relevant here, they can't get rebuild?
(In reply to Tom Wijsman (TomWij) from comment #11) > I'm not sure how binary packages are relevant here, they can't get rebuild? emerge @preserved-rebuild will cause a manual rebuild of things unless you have binpkg-only set, I would expect this to work the same. Forcing binary packages only has a lot of limitations, I would expect most users don't want this, although --rebuilt-binaries=y will reinstall if the binary package is newer so the build host rebuilds and the binary users just pull a fresh package.
This can't work for virtual packages and there's no point in even trying to use it there. It's a step backwards, toward PROVIDES= and other magic. Usually virtual specifies a few packages that satisfy it using any-of dep, like: || ( foo bar ) Now, if you switch the provider from foo to bar, how should it all work? Should it force rebuilding all the revdeps? Or maybe the 'passthru slot' should move transparently from subslot of one provider to those of another provider? What if different providers have different subslots? What if you have a binary package that works with different subslots of different providers? While subslot (and slot) operator rebuilds cause trouble already, with users being confused and portage being buggy, adding indefinite number of layers of indirection is going to be even worse. I can understand that you may need it for other stuff. But please don't make virtuals a mess with it. A much simpler solution would be having versioned virtuals for SONAMEs (we often need to version them anyway for versioned deps), with subslot deps on the deps, e.g.: SLOT="0/11" RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )"
(In reply to Michał Górny from comment #13) > This can't work for virtual packages and there's no point in even trying to > use it there. It's a step backwards, toward PROVIDES= and other magic. > > Usually virtual specifies a few packages that satisfy it using any-of dep, > like: > > || ( foo bar ) > > Now, if you switch the provider from foo to bar, how should it all work? > Should it force rebuilding all the revdeps? Or maybe the 'passthru slot' > should move transparently from subslot of one provider to those of another > provider? > > What if different providers have different subslots? What if you have a > binary package that works with different subslots of different providers? > > While subslot (and slot) operator rebuilds cause trouble already, with users > being confused and portage being buggy, adding indefinite number of layers > of indirection is going to be even worse. > > I can understand that you may need it for other stuff. But please don't make > virtuals a mess with it. A much simpler solution would be having versioned > virtuals for SONAMEs (we often need to version them anyway for versioned > deps), with subslot deps on the deps, e.g.: > > SLOT="0/11" > RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )" I don't understand that example. What does it do? I guess those 11 and 12 are SONAME versions. "bar:0/11 -> bar:0/12" upgrade will break virtual users w/o them noticing. (for those who happened to link against "bar:0/11" and have that virtual installed) Seems your "problem" is's not specific to virtuals. Nobody forbids you to put exactly that into usual ebuild. || ( foo:= bar:= ) # to make syntax more valid for current portage How does portage handle that? It needs to know exactly what alternative is taken by built package to make correct rebuild decisions. The bug is about the packages which ABIs are defined by their depends.
(In reply to Sergei Trofimovich from comment #14) > (In reply to Michał Górny from comment #13) > > SLOT="0/11" > > RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )" > > I don't understand that example. What does it do? > I guess those 11 and 12 are SONAME versions. > "bar:0/11 -> bar:0/12" upgrade will break virtual users w/o them noticing. > (for those who happened to link against "bar:0/11" and have that virtual > installed) No, it won't. Simply because '11' and '12' are subslots in which some other library did change (e.g. libsystemd* in systemd for virtual/udev).
(In reply to Michał Górny from comment #13) > This can't work for virtual packages and there's no point in even trying to > use it there. It's a step backwards, toward PROVIDES= and other magic. > > Usually virtual specifies a few packages that satisfy it using any-of dep, > like: > > || ( foo bar ) > > Now, if you switch the provider from foo to bar, how should it all work? > Should it force rebuilding all the revdeps? Or maybe the 'passthru slot' > should move transparently from subslot of one provider to those of another > provider? Given that "changing providers" could mean many different things: 1- if emerge -C foo && emerge bar , yes I expect that the only way this would work in a stable fashion would be for the passthrough-subslot to change in the virtual, and therefore rdeps of the virtual would be triggered for rebuild. ESPECIALLY, since most virtuals do not have version controls determining ABI equivalence throughout (rather, just API equivalence, if even that) 2- if just emerging another one in the list (because for whatever reason they don't block eachother), then I agree that it shouldn't trigger a rebuild. I'd need to test this, but I expect that emerging a second package within an RDEPEND="||( a:= b:= )" right now would not trigger a rebuild, so if the virtual itself isn't re-emerged then the recorded "passthrough" subslot wouldn't change and so it'd all work out. As I said though, more research is needed. > What if different providers have different subslots? What if you have a > binary package that works with different subslots of different providers? subslots are just indicators for change. They are not a version in and of themselves and cannot be relied upon as such. So, what subslot value any given dep of a virtual has, really doesn't matter in terms of what the other deps have. A binary package only gets to work with what it's built against, but the subslot doesn't determine that any better than a virtual without versions determines it now. > I can understand that you may need it for other stuff. But please don't make > virtuals a mess with it. A much simpler solution would be having versioned > virtuals for SONAMEs (we often need to version them anyway for versioned > deps), with subslot deps on the deps, e.g.: > > SLOT="0/11" > RDEPEND="|| ( foo:0/11 bar:0/12 bar:0/11 )" The issue with this is that we would then need to have an ebuild for every virtual that specifies all valid combinations of packages via their subslot values. That's a lot more ebuilds to have to work with. Not that this is a bad thing, mind you -- it would be nice for a virtual to have an easily specified upper-bound on the packages that provide it, and revbumping to add new versions doesn't seem like a bad idea to me. Of course there's the other advantage that it is doable now, without any new PM features or EAPI additions.
*** Bug 607780 has been marked as a duplicate of this bug. ***
*** Bug 607782 has been marked as a duplicate of this bug. ***
*** Bug 607784 has been marked as a duplicate of this bug. ***
Did anyone come up with a reasonable idea of how this could actually be made working? With growing number of ABI-incompatible library forks, a solution to the problem of writing sane dependencies would be quite welcome, and I think some kind of 'passthru subslots' on virtuals would be quite readable on developer side. However, for that to work we really need a good plan. If we aren't going to have one in near time, I think we ought to look for other solutions.
Is this fixed with EAPI=7?
(In reply to crocket from comment #21) > Is this fixed with EAPI=7? No.
*** Bug 701374 has been marked as a duplicate of this bug. ***
(In reply to Michał Górny from comment #20) > Did anyone come up with a reasonable idea of how this could actually be made > working? With growing number of ABI-incompatible library forks, a solution > to the problem of writing sane dependencies would be quite welcome, and I > think some kind of 'passthru subslots' on virtuals would be quite readable > on developer side. > > However, for that to work we really need a good plan. If we aren't going to > have one in near time, I think we ought to look for other solutions. I don't think it can be solved entirely in DEPEND* syntax. I get the impression that an ebuild needs some sort of mechanism to semi-programmatically declare "What is my ABI" in terms of its parts. subslots are limited primarily in that they have to be determined statically. I'm just spit-balling here, but maybe something like: VARY=" use: foo,bar,baz depend: dev-lang/foo " Which could also be written: VARY=" use: * depend: * " Both of which indicate "my ABI changes when these values change" These aspects could be combined to produce a unique SHA1 ( or even MD5, its not a security aspect, and the collision space is unique to a package ) that represents the variable aspects of the ABI. At least, that approach maps to how this works under-the-hood in a plethora of modern languages. Then the "rebuild" condition is determined by detecting if a packages SHA1 would change.
Is it the only issue that prevents gentoo linux from dropping haskell-updater?
(In reply to amano.kenji from comment #25) > Is it the only issue that prevents gentoo linux from dropping > haskell-updater? No, bug 680496 would be needed too.
I suspect we'd also need a way to arbitrarily derive some sort of 'compatibility string' (a generalization of a subslot), which can take both USE state and dependencies' compat strings as inputs, or something. this would facilitate checking for satisfaction in binary packages (at least, I'm not sure if, in a binary package context, we can detect for subslot updates or such, because I see no way for a binary package to prevent being updated without updating its deps or such (imagine if A->B, B gets an update and A gets a subslot rebuild, what if the client updates solely A to the new build#?)) currently just waffling on without too much worked out, but I'd like to nudge this discussion a bit given the new binpkg context
What about adopting the style of functional package management done by nix and guix? gnu guix wouldn't have this issue. Actually, with gnu guix, you can install multiple versions of a haskell hackage library. I think it's time that we should at least look into functional package management.
Someone almost finished implementing package parameters for GNU guix. There are global package parameters and local ones. They are almost identical to USE flags, but they will be implemented as higher-order guile scheme functions. See https://notabug.org/cel7t/guix-parameters
yes, the idea I mentioned above is rather similar to it, but, unlike guix, will not depend on *all* bytes of a generated dependency, but rather a composable 'compatibility string'. this leads to significant benefits over the naive approach as it reduces the coupling between otherwise independent sw components (which is the majority case, as in, all C and most C++ libraries)
Is it like computing a hash over USE flags and dependencies?
