Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 516612 - sys-apps/portage: please disable dynamic-deps by default
Summary: sys-apps/portage: please disable dynamic-deps by default
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Dependencies (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL: http://comments.gmane.org/gmane.linux...
Whiteboard:
Keywords:
Depends on: 486778
Blocks:
  Show dependency tree
 
Reported: 2014-07-07 13:52 UTC by Michał Górny
Modified: 2017-02-12 08:23 UTC (History)
11 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-07 13:52:22 UTC
I've got one more potential screwup possible due to dynamic-deps.

1. I have =A-1 installed that depends on B.

2. Someone removes the dependency on B in-place in the ebuild.

3. Now dynamic-deps causes portage to depclean B from my system.

4. Someone lastrites and removes B from the tree since no ebuild depends on it anymore.

5. Someone removes ebuild for =A-1.

6. Now, portage no longer has reference for dynamic-deps and starts using deps from vdb. So installed package suddenly starts requiring package that can no longer be installed.

Could we finally kill the thing and start doing things properly?
Comment 1 Sebastian Luther (few) 2014-07-07 16:15:59 UTC
(In reply to Michał Górny from comment #0)
> Could we finally kill the thing and start doing things properly?

+1, but that's more complicated than just changing the default for emerge.

Sometimes people perform mass changes to ebuild deps and want them reflected in installed packages' deps.

The questions are
a) Are those uses valid at all? and if they are,
b) Who could the a similar effect be achieved?

A slotmove like mechanism comes to mind, but does that really scale to hundreds of ebuilds?
Comment 2 Martin Väth 2014-07-07 17:44:47 UTC
Quite the opposite. Please, let me explain why I think that reading dependencies from /var/db at all is problematic.

It happens very often that dependencies are updated, either for fixes (e.g. if somebody reports that some feature is not working without a certain dependency) or to reflect freshly added USE-flags, project splits (of the dependent projects), or whatever.

So the DEPEND data in /var/db is usually in an outdated, often in a broken state.
When doing emerge -NaDu @world, users (well, at least me) want to see the "true" current dependencies, not some years old obsolete state which had been valid when some package had been installed.

One way to fix this is to copy DEPEND to /var/db after every sync, but then what is the point of having DEPEND data in /var/db at all?

A similar thing holds for .tbz2: Who cares about the outdated dependencies and keywords which were valid many years ago when the file had been built? When I (re)install the .tbz2 file after some years, I want it to reflect the current tree, not an obsolete tree which is many years old.

The latter is an actual problem: I usually build .tbz2 files for a client machine which is updated only very rarely (say, once a year). Generally running stable, however, I sometimes need some ~ARCH package. After some months that ~ARCH package is stabilized, and so I remove the package from /etc/portage/package.accept_keywords again. Now the client, with the same /etc/portage/package.accept_keywords file refuses to install the .tbz2. Reason: In the moment when the file was built (many months ago), the package was ~ARCH. This years-old and long obsolete information is still stored in .tbz2 and used...

In your example, what is the "problem" is actually the last point: After

> 5. Someone removes ebuild for =A-1.

the user should either remove A-1, too, or upgrade to the current version:
It is a very bad idea to keep a package version installed without having the appropriate ebuild - not only due to dynamic deps. A rotting installed package of a nonexisting ebuild (hence unmaintained) will after a while never reflect the correct dependencies of the package in the current tree (unless the user has good luck, of course, but we should not be talking about the "good luck" case).

Portage should severely warn about this and either recommend upgrading or removal (IIRC it already does).
Comment 3 Sebastian Luther (few) 2014-07-07 18:03:12 UTC
There is no need for an ebuild to correspond to an installed version. For example an application may optionally depend on a library and the library is later found to be broken and gets masked. When the mask happens, the corresponding use flag of the application gets removed and the dependency is removed.

Now with dynamic-deps emerge will happily uninstall the library with -c, because the application's updated deps say that it doesn't depend of the library. But that's wrong and the application with silently be broken.

You could argue that the application should be rev-bumped, but that doesn't happen often enough.
Comment 4 Martin Väth 2014-07-07 18:32:15 UTC
(In reply to Sebastian Luther (few) from comment #3)
> and the library is later found to be broken and gets masked.
> 
> Now with dynamic-deps emerge will happily uninstall the library with -c

This is a good thing and will be cought with revdep-rebuild:
When a broken library is masked, I would *want* dependent packages to be rebuilt without this library and not tacitly continue to use the broken library, instead of portage not informing me that there is a serious problem.
Comment 5 Sebastian Luther (few) 2014-07-07 18:50:10 UTC
(In reply to Martin Väth from comment #4)
> (In reply to Sebastian Luther (few) from comment #3)
> > and the library is later found to be broken and gets masked.
> > 
> > Now with dynamic-deps emerge will happily uninstall the library with -c
> 
> This is a good thing and will be cought with revdep-rebuild:

revdep-rebuild is a hack from the past. Silent breakage is the worst thing a package manager can deliver.

> When a broken library is masked, I would *want* dependent packages to be
> rebuilt without this library and not tacitly continue to use the broken
> library, instead of portage not informing me that there is a serious problem.

That's your opinion, I have seen users with the opinion that stuff that works, should just keep working. There's also no problem with this example, instead of silent breakage you'll get a warning about masked packages now. You can then act accordingly.
Comment 6 Martin Väth 2014-07-07 19:36:05 UTC
(In reply to Sebastian Luther (few) from comment #5)
> revdep-rebuild is a hack from the past.

With @revdep-rebuild it works as well.
And calling revdep-rebuild a hack does not make using obsolete and broken dependency information a non-hack.

> Silent breakage is the worst thing a package manager can deliver.

One can discuss whether it is better to keep packages known to use broken libraries.

Anyway, having obsoleted broken dependencies is certainly not a good concept and will only create more problems than it solves.

> > When a broken library is masked, I would *want* dependent packages to be
> > rebuilt without this library and not tacitly continue to use the broken
> > library, instead of portage not informing me that there is a serious
> > problem.
> 
> That's your opinion, I have seen users with the opinion that stuff that
> works, should just keep working.

Of course, that's why gentoo is about choice: They have the choice to keep the old library.
In contrast, when portage just switches to using obsolete dependencies, I have no choice, and I get silent breakage through these obsolete dependencies by portage.
For instance, if this broken library is replaced by a new fork which is perhaps split into several libraries, portage does not suggest to upgrade - in fact, nothing depends on these new shiny libraries, because the years-old obsolete information is used instead.
Also, it contradicts the principle of least surprise: On a system which is rather similar to the first system, but with package A not being installed for some reason (or being reemerged for some reason), suddenly completely different dependencies appear.
Comment 7 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-07 20:23:20 UTC
(CC-ing Ciaran for the rescue!)
Comment 8 Ciaran McCreesh 2014-07-07 20:30:08 UTC
Dynamic deps is a hack, and it only "works" under very restricted circumstances (the 'associated' ebuild can be correctly determined, it still exists, and its updated dependency information happens to match whatever was installed). This is a long string of coincidences that have to hold. Removing an ebuild from the tree shouldn't cause random breakage to installed systems, and occasional minor convenience isn't a good excuse for erratic behaviour. The only sensible long-term fix is to abandon the idea entirely.
Comment 9 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-07 20:30:54 UTC
Let's get a few things clear:

1. Most of software written in C/C++ links to the libraries that are in the system at the time of build. Nowadays we can express this with slot/subslot operator :=. dynamic-deps simply can't work with that since it needs the exact slot/subslot from install time, not the current atom in ebuild. You need to use the dependency from vardb for that.

2. If ebuild is removed and portage can no longer apply the dependency from ebuild, the resulting dependency tree may be so broken that you won't be even be able to sanely upgrade the package in question. Portage will simply prevent you from doing that due to, say, circular dependency with package that has a dependency that can't be satisfied.

3. I feel like you're saying 'visible breakage is better because it forces user to fix it'. Though I'd rather have an extra library installed till next upgrade rather than system that does not boot anymore.
Comment 10 Martin Väth 2014-07-08 07:47:29 UTC
(In reply to Ciaran McCreesh from comment #8)
> Dynamic deps is a hack, and it only "works" under very restricted
> circumstances

Both holds for static rotten dependencies, even in a way which the aminstrator cannot control well.

Instead of throwing terms like "hack" let's face how it is:
In both cases, things work quite often, but in certain situations one has the potential of breaking installed packages if the adminstrator is lazy (see below), the other has the potential of breaking new packages by keeping working alternatives from being pulled in.
I consider it dangerous to remove the user's choice about what risk he prefers, by calling the alternative a hack and removing it.

> (the 'associated' ebuild can be correctly determined, it still
> exists, and its updated dependency information happens to match whatever was
> installed). This is a long string of coincidences that have to hold.

No, this is the normal requirement one should expect from a responsible administrator:
If an ebuild is being removed and the administrator wants to keep it installed anyway, he should put the ebuild into his overlay and feel obliged to maintain it himself.

Checking and keeping dependencies updated by editing mainly undocumented /var/db and *.tbz2 files - this is not something one should expect from an adminstrator to keep his system in a sane state for the next update command to work properly...

> Removing an ebuild from the tree shouldn't cause random breakage to
> installed systems, and occasional minor convenience isn't a good excuse for
> erratic behaviour.

Excatly: The minor convenience that you keep a packet installed without maintainig it properly in your personal overlay - hopefully the adminstrator is wise enough to do this only for a limited period - this minor convenience is not something for which you should omit reflecting the *current* tree in the dependencies.

Note that without dynamic dependencies the actual dependency tree used is then a mixture of various dependencies of rather different ages of the gentoo tree, typically over many years - nobody can have an overview what unexcepted problems this causes. I cannot imagine that you really want to advocate this as a state desirable for user's systems.

> If ebuild is removed and portage can no longer apply the dependency from > ebuild, the resulting dependency tree may be so broken

(In reply to Michał Górny from comment #9)
> 
> 1. Nowadays we can express this with slot/subslot operator :=

No discussion needed about that ('though IIRC there were some cases where this cannot replace revdep-rebuild - I must admit that I cannot recall these cases in the moment.)

> 2. If ebuild is removed and portage can no longer apply the dependency from
> ebuild, the resulting dependency tree may be so broken that

This is exactly one of my worries:
The user was running emerge -NaDu @world, everything was reported as working, and so the user thinks he is running a current up-to-date safe system, though actually he is running a system with dependency bugs which perhaps have already been reported and fixed years ago (which in the worst caswe might even be implicit security issues!)
Moreover, some day he decides for whatever reason to reemerge a certain package and then suddenly a whole bulk of new packages is pulled in. In the worst case, portage even reports circular dependencies and the user has a hard time to get his system really to a working state, although portage claimed that it was working OK all of the time.

> 3. I feel like you're saying 'visible breakage is better because it forces
> user to fix it'.

Yes, in the moment when the state of the gentoo tree changes in such a manner that I would need to emerge new dependencies (or emerge dependencies with different USE-flag settings), I would rather like portage to break with an error and allow me to fix things instead of proceeeding tacitly with an inappropriate dependency tree.

> Though I'd rather have an extra library installed till next
> upgrade rather than system that does not boot anymore.

When the administrator fixes things in the correct order - by first re-adding a removed ebuild if he wants to keep that version and only then removing random packages - nothing is ever broken.
Moreover, concerning a non-booting system: If the administrator does not care enough about his boot-critical stuff by keeping boot-critical packages without ebuilds and removing random packages they might rely on, he really cannot be helped.
Comment 11 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-08 08:29:49 UTC
(In reply to Martin Väth from comment #10)
> (In reply to Michał Górny from comment #9)
> > 
> > 1. Nowadays we can express this with slot/subslot operator :=
> 
> No discussion needed about that ('though IIRC there were some cases where
> this cannot replace revdep-rebuild - I must admit that I cannot recall these
> cases in the moment.)

Did you read that paragraph to the end? It said that dynamic-deps *can't work*. Realistically saying, over time most of C/C++ packages will have them implicitly disabled.

> > Though I'd rather have an extra library installed till next
> > upgrade rather than system that does not boot anymore.
> 
> When the administrator fixes things in the correct order - by first
> re-adding a removed ebuild if he wants to keep that version and only then
> removing random packages - nothing is ever broken.

Re-adding ebuilds was never supported and never will be. Don't even try doing that. Please also remember that eclasses and profiles change, and old ebuild may no longer work.

Moreover, how is the administrator supposed to know something random is being removed? Portage doesn't say 'removing X, Y may be broken'. In fact, portage doesn't even say that the ebuild no longer exists.

> Moreover, concerning a non-booting system: If the administrator does not
> care enough about his boot-critical stuff by keeping boot-critical packages
> without ebuilds and removing random packages they might rely on, he really
> cannot be helped.

Computer is a tool, and is supposed to serve people. Following your reasoning, we could kill all the package management tools since sysadmin is pretty much responsible for doing all the hard work himself anyway. Not having semi-capable tools would at least avoid implicit breakage.
Comment 12 Martin Väth 2014-07-08 09:40:46 UTC
(In reply to Michał Górny from comment #11)
> Did you read that paragraph to the end?

I did, but I did not comment on the rest, because IMHO the problem you mention there is unrelated.

> It said that dynamic-deps *can't work*.

Why? Portage looks up the deps from the current tree, see some := and in this case looks up some information stored in /var/db to fill in the appropriate value of "=" for that package. Whether portage looks up this information from the file /var/db/*/*/DEPEND or possibly from another file in /var/db is irrelevant and has nothing to do with our discussion.

> > When the administrator fixes things in the correct order - by first
> > re-adding a removed ebuild if he wants to keep that version and only then
> > removing random packages - nothing is ever broken.
> 
> Re-adding ebuilds was never supported and never will be.

Actually, running packages removed from the portage tree is not supported and *can* never be supported. If the administrator has a serious reason to do this, he is responsible for it, and he has to take the pieces if it breaks.
Re-adding the ebuild and maintaining the ebuild by the administrator is the best what he can do if he so seriously needs to keep a removed package.

All of the discussion and breaking on other points we are talking about is only to support this very exceptional situation which can never have a satisfactory solution, anyway.

> Please also remember that eclasses and profiles change, and old
> ebuild may no longer work.

Exactly. That's why either the administrator must take full responsibility of the ebuild (including adapting to such changes) or he must expect breakage when running a package removed from the tree. Pretending anything else is a lie, and basing the concept of a PM to deal with this situation conveniently at the price of breaking other situations is not a good idea.

> Moreover, how is the administrator supposed to know something random is
> being removed?

Because he omits an emerge -c or emerge depclean without updating the packages with emerge -NDu first.

> In fact, portage doesn't even say that the ebuild no longer exists.

It doesn't? I remember warnings that e.g. installed packages have been masked, and I would expect the same if the user unmasked anyway but the package eventually gets treecleaned.
If there are really no messages in this case, I strongly suggest that this should be fixed.

> sysadmin is pretty much responsible for doing all the hard work himself

sysadmin is pretty much on his own when he runs packages removed from the tree - there is not much pm can do to support him except allowing him easily to maintain the ebuild in his own overlay. Without dynamic deps he would even have to modify /var/db by himself...
Comment 13 Alexander Berntsen (RETIRED) gentoo-dev 2014-07-08 09:45:25 UTC
(In reply to Martin Väth from comment #12)
> > In fact, portage doesn't even say that the ebuild no longer exists.
> It doesn't? I remember warnings that e.g. installed packages have been
> masked, and I would expect the same if the user unmasked anyway but the
> package eventually gets treecleaned.
> If there are really no messages in this case, I strongly suggest that this
> should be fixed.

Friends,

can someone please establish verity or falsity on the claim that emerge emits no warning if masked-but-unmasked packages are treecleaned? This would be very helpful! Please file a separate bug if it is the case.
Comment 14 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-08 10:25:32 UTC
(In reply to Martin Väth from comment #12)
> (In reply to Michał Górny from comment #11)
> > Did you read that paragraph to the end?
> 
> I did, but I did not comment on the rest, because IMHO the problem you
> mention there is unrelated.
> 
> > It said that dynamic-deps *can't work*.
> 
> Why? Portage looks up the deps from the current tree, see some := and in
> this case looks up some information stored in /var/db to fill in the
> appropriate value of "=" for that package. Whether portage looks up this
> information from the file /var/db/*/*/DEPEND or possibly from another file
> in /var/db is irrelevant and has nothing to do with our discussion.

Except that it doesn't. It is unable to mix the two dependency sets, and therefore disables dynamic-deps completely.

Additionally, how would you solve the issue when a new dependency is added with :=? Portage can not know which version was used during the build.

> > Moreover, how is the administrator supposed to know something random is
> > being removed?
> 
> Because he omits an emerge -c or emerge depclean without updating the
> packages with emerge -NDu first.

There's no guarantee that the @world upgrade will upgrade packages. '<' dependencies common in some places are one reason for that. Not saying they're good but they happen.

Furthermore, the dependency resolver in portage is so much broken nowadays that it sometimes silently omits upgrades for various reasons (subslots mostly). I mean, it looks like you have nothing to upgrade. But if you add '--ignore-built-slot-operator-deps', then boom -- a lot of upgrades missed...

> > sysadmin is pretty much responsible for doing all the hard work himself
> 
> sysadmin is pretty much on his own when he runs packages removed from the
> tree - there is not much pm can do to support him except allowing him easily
> to maintain the ebuild in his own overlay. Without dynamic deps he would
> even have to modify /var/db by himself...

Which means that every sysadmin is on his own *daily*. Because daily packages are replaced by newer revisions with old revisions being removed instantly (due to severe issues, for example). Between --sync and world update, he is *running removed package*, and we need to make sure he can upgrade to the newer version without issues.
Comment 15 Martin Väth 2014-07-08 13:50:17 UTC
(In reply to Michał Górny from comment #14)
> Except that it doesn't. It is unable to mix the two dependency sets

doesn't != cannot.
I am surprised that it does not (I admit that I never looked at the resolver code, only at the pms specification), and indeed I think that this should be fixed. If the "=" data would not be stored in /var/db/***/DEPEND but in a different place, this would perhaps be trivial to fix...

> Additionally, how would you solve the issue when a new dependency is added
> with :=? Portage can not know which version was used during the build.

I think the answer should be clear in that case: Portage has not stored the slot information, so some illegal slot should be used, i.e. a rebuild of the package should be forced.
This is also what I would expect as a user in such a case...

> Furthermore, the dependency resolver in portage is so much broken nowadays
> that it sometimes silently omits upgrades for various reasons

Bad enough that this happens, but the existence of one bug is not a reason to introduce yet another bug which might break even more things.

> Which means that every sysadmin is on his own *daily*. Because daily
> packages are replaced by newer revisions with old revisions being removed
> instantly (due to severe issues, for example).

I think portage has to make sure that in the case of removed versions, if a newer version is available, the upgrade is happening, and if this cannot happen a big fat warning or even an error should be omitted: I think in such a situation the administrator must be informed and is obliged to do anything to either make this upgrade happen or remove the package (or live in an unsupported state).

Perhaps some poll should be made: I cannot remember any situation since many years where after an "emerge -NDu @world" I have an installed package without a corresponding ebuild in the main tree (with the obvious exceptions of packages needing to be depcleaned); an I really checks this with eix -uc and eix-test-obsolete after every sync/emerge -NDu @world.
I do not doubt that such a problem *might* happen, but one should work hard to not *make* it happen (and if it does happen omit a big warning).
This, however, is somewhat independent of the bug discussed here...
Comment 16 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-08 14:00:20 UTC
(In reply to Martin Väth from comment #15)
> (In reply to Michał Górny from comment #14)
> > Except that it doesn't. It is unable to mix the two dependency sets
> 
> doesn't != cannot.
> I am surprised that it does not (I admit that I never looked at the resolver
> code, only at the pms specification), and indeed I think that this should be
> fixed. If the "=" data would not be stored in /var/db/***/DEPEND but in a
> different place, this would perhaps be trivial to fix...

It's never trivial. Please also understand that any change to the resolver at the moment is likely to break another pile of hacks that is already there.

> > Furthermore, the dependency resolver in portage is so much broken nowadays
> > that it sometimes silently omits upgrades for various reasons
> 
> Bad enough that this happens, but the existence of one bug is not a reason
> to introduce yet another bug which might break even more things.

Exactly. So why are you telling us to enable all the bugs involved with dynamic-deps?


Please also understand that developing ebuilds for dynamic-deps is completely different than developing ebuilds for 'static deps'.

If you don't have dynamic-deps, you need to revbump ebuilds in order to get dependency changes applied. However, if you make a non-important change that affects build only, you don't need to revbump since people will grab it over time or with other bump.

On the other hand, if you have dynamic-deps, you don't need to revbump to get dependency changes applied. But you actually have to revbump to avoid breakage when something affecting build is changed (e.g. unnecessary linking without -Wl,--as-needed is fixed). So... people have to rebuild stuff to get no actual change.


The real issue is that you can't rely on either. Dynamic-deps is on by default but it isn't used:

1) by non-portage package managers,

2) if user disabled it because of the past screwups,

3) if ebuild uses :=,

4) possibly in more random cases, depending on the phase of the moon etc.

So in the end, developers would have to revbump all the stuff twice as much to satisfy both systems. So instead of extra rebuilds because of lack of dynamic-deps, you get twice the amount of extra rebuilds to avoid issues involved with having them on or off randomly.
Comment 17 Martin Väth 2014-07-08 15:17:38 UTC
(In reply to Michał Górny from comment #16)
> Please also understand that developing ebuilds for dynamic-deps is
> completely different than developing ebuilds for 'static deps'.
> [...]
> If you don't have dynamic-deps, you need to revbump ebuilds in order to get
> dependency changes applied.

Maybe I misunderstood:
Do you suggest to change the way the portage tree is currently handled and that revbumps are made on *any* change of dependendencies?
Indeed, in this case dynamic deps are not needed, but people will have to recompile a lot for now actual change.

> On the other hand, if you have dynamic-deps, you don't need to revbump to
> get dependency changes applied.

Which is how it is done now. Otherwise, we would see with any nontrivial change of python.eclass (e.g.) a revbump of hundreds of packages.

> But you actually have to revbump to avoid breakage when something
> affecting build is changed (e.g. unnecessary linking
> without -Wl,--as-needed is fixed).

In such a case, a revbump should happen in any case. Otherwise the dependency will proceed in the user's tree, and the user will have a hard time to get rid of this problem.

> So... people have to rebuild stuff to get no actual change.

It seems you are talking about the non-dynamic variant now...

> So in the end, developers would have to revbump all the stuff twice as much
> to satisfy both systems.

The second example (a serious fix in the build system having effects on the compilation of further packages) should be revbumped in any case.
With non-dynamic variants much more revbumps have to be made than is the case currently. If you really want to change to the policy to bump every dependency change, I do not see any breakage with static deps, although things will then get tough for users who do not have a powerful machine or a huge ccache directory.
Comment 18 Ciaran McCreesh 2014-07-08 15:23:09 UTC
(In reply to Martin Väth from comment #10)
> Instead of throwing terms like "hack" let's face how it is:

How it actually is: dynamic-deps is broken by design, but sometimes you don't notice the breakage, and sometimes it hides developer mistakes. So the only argument in favour of it is that maybe you think developers are going to keep on making mistakes, even if the issue is brought to their attention and made more visible.
Comment 19 Martin Väth 2014-07-08 17:24:22 UTC
(In reply to Ciaran McCreesh from comment #18)
> (In reply to Martin Väth from comment #10)
> > Instead of throwing terms like "hack" let's face how it is:
> 
> How it actually is: dynamic-deps is broken by design

Do you seriously want to prove this claim by repeating it in an attacking manner without any factual arguments and by ignoring my examples in which not using dynamic-deps causes breakage?
It is exactly that I want to stick to the facts instead of emotions why I had written the cited sentence. Please argue by facts.

I might very well be convinced by facts. Currently, I see breakage in both approaches, and one has to decide which breakage is more severe - obviously, this depends on what you expect from your PM, and it is the reason why I strongly argue that both approaches should be kept in portage. (I do not mind changing defaults, but I am afraid that this is the first step to cancel the dynamic approach.)

A "solution" would be to change the policy to bump for every single dependency change. With the latter, the two approaches should hardly differ in practice, but of course, this "solution" has other obvious disadvantages as well.
Comment 20 Ciaran McCreesh 2014-07-08 17:33:59 UTC
(In reply to Martin Väth from comment #19)
> (In reply to Ciaran McCreesh from comment #18)
> > (In reply to Martin Väth from comment #10)
> > > Instead of throwing terms like "hack" let's face how it is:
> > 
> > How it actually is: dynamic-deps is broken by design
> 
> Do you seriously want to prove this claim by repeating it in an attacking
> manner without any factual arguments and by ignoring my examples in which
> not using dynamic-deps causes breakage?

That is not a claim to be justified. It is a simple fact, and there is nothing more to be said on the matter. The long-term fix for the issue is to get rid of dynamic deps entirely, and to ensure that developers handle dependencies correctly. The only issue to discuss is how long it's worth prolonging the current broken state of affairs, given that it might take a while to fix bad developer habits.
Comment 21 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-08 17:35:03 UTC
(In reply to Martin Väth from comment #17)
> (In reply to Michał Górny from comment #16)
> > Please also understand that developing ebuilds for dynamic-deps is
> > completely different than developing ebuilds for 'static deps'.
> > [...]
> > If you don't have dynamic-deps, you need to revbump ebuilds in order to get
> > dependency changes applied.
> 
> Maybe I misunderstood:
> Do you suggest to change the way the portage tree is currently handled and
> that revbumps are made on *any* change of dependendencies?
> Indeed, in this case dynamic deps are not needed, but people will have to
> recompile a lot for now actual change.

On any important change of dependencies that need to be applied instantly. Minor changes or build-time changes can be delayed safely.

In fact, developers have to do that for compliance with the PMS and other package managers anyway. And pretty much for compliance with the lot of cases when dynamic-deps don't work.

The only problem is that they don't do it, usually covering behind dynamic-deps. Even when dynamic-deps doesn't work. They just don't know their change doesn't work...

> > On the other hand, if you have dynamic-deps, you don't need to revbump to
> > get dependency changes applied.
> 
> Which is how it is done now. Otherwise, we would see with any nontrivial
> change of python.eclass (e.g.) a revbump of hundreds of packages.

Well, if you mean python-r1 then it's already how it is now. python-r1 eclasses use := operator, so every Python ebuild already breaks dynamic-deps.

> > But you actually have to revbump to avoid breakage when something
> > affecting build is changed (e.g. unnecessary linking
> > without -Wl,--as-needed is fixed).
> 
> In such a case, a revbump should happen in any case. Otherwise the
> dependency will proceed in the user's tree, and the user will have a hard
> time to get rid of this problem.

This has two sides to it. Having extra library installed for a few days (until more important change required bump, or a version bump) is a minor issue. Having to rebuild LibreOffice because dep change broke linking is a major issue.

> > So... people have to rebuild stuff to get no actual change.
> 
> It seems you are talking about the non-dynamic variant now...

No, I'm talking about dynamic variant. It assumes that every change in ebuild needs to be applied immediately. It doesn't allow you to change build stuff without affecting already-installed packages. So you need to revbump and force everyone to rebuild to prevent dynamic-deps from happening... while normally they wouldn't apply at all.

> > So in the end, developers would have to revbump all the stuff twice as much
> > to satisfy both systems.
> 
> The second example (a serious fix in the build system having effects on the
> compilation of further packages) should be revbumped in any case.

It's not a serious fix. It's killing extra direct dependency in a rare scenario (non-default, uncommon LDFLAGS).

> With non-dynamic variants much more revbumps have to be made than is the
> case currently. If you really want to change to the policy to bump every
> dependency change, I do not see any breakage with static deps, although
> things will then get tough for users who do not have a powerful machine or a
> huge ccache directory.

To bump every important dependency change, to be more correct. Weigh it.
Comment 22 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-08 17:36:52 UTC
(In reply to Martin Väth from comment #19)
> I might very well be convinced by facts. Currently, I see breakage in both
> approaches, and one has to decide which breakage is more severe - obviously,
> this depends on what you expect from your PM, and it is the reason why I
> strongly argue that both approaches should be kept in portage. (I do not
> mind changing defaults, but I am afraid that this is the first step to
> cancel the dynamic approach.)

Keeping both mean supporting both. Which means we end up with issues with both, and no advantages. If we have to support non-dynamic deps, dynamic-deps don't do anything. So by supporting it we actually force another set of issues for us.
Comment 23 Arfrever Frehtes Taifersar Arahesis 2014-07-08 17:42:02 UTC
Improvements in Portage should be implemented to make dynamic dependencies work with := operator (as much as possible).

I do not care if ebuilds in gentoo-x86 are allowed to rely on dynamic dependencies, but other repositories must have ability to choose to rely on dynamic dependencies.
Comment 24 Ciaran McCreesh 2014-07-08 17:48:34 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #23)
> Improvements in Portage should be implemented to make dynamic dependencies
> work with := operator (as much as possible).

:= is just one of the many issues. Even if := wasn't involved, dynamic deps are still broken. But "improvements in Portage" can't fix this: your statement is just like saying "improvements in Portage should be implemented to make it automatically patch all bugs in the software it's installing".

> I do not care if ebuilds in gentoo-x86 are allowed to rely on dynamic
> dependencies, but other repositories must have ability to choose to rely on
> dynamic dependencies.

By their very design, there are no circumstances in which it is safe to rely upon dynamic dependencies.
Comment 25 Arfrever Frehtes Taifersar Arahesis 2014-07-08 18:00:09 UTC
Potential solution for problem described in comment #0:

Make Portage dependency resolver store dynamically calculated dependencies for all packages, for which they have been calculated, in separate files in VDB:
  /var/db/pkg/${CATEGORY}/${PF}/DEPEND.updated
  /var/db/pkg/${CATEGORY}/${PF}/PDEPEND.updated
  /var/db/pkg/${CATEGORY}/${PF}/RDEPEND.updated
When ebuild for an installed package is deleted, then last calculated dependencies stored in above files will be used.
Comment 26 Ciaran McCreesh 2014-07-08 18:05:37 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #25)
> Potential solution for problem described in comment #0:

Not a solution: if an ebuild is updated and then removed, and the system is not synced in the middle, then it fails.
Comment 27 Ulrich Müller gentoo-dev 2014-07-08 18:06:16 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #25)
> Potential solution for problem described in comment #0:
> 
> Make Portage dependency resolver store dynamically calculated dependencies
> for all packages, for which they have been calculated, in separate files in
> VDB:
>   /var/db/pkg/${CATEGORY}/${PF}/DEPEND.updated
>   /var/db/pkg/${CATEGORY}/${PF}/PDEPEND.updated
>   /var/db/pkg/${CATEGORY}/${PF}/RDEPEND.updated
> When ebuild for an installed package is deleted, then last calculated
> dependencies stored in above files will be used.

This looks like a horrible hack. The VDB should contain information about the packages installed on a system (at the time when they have been installed), which isn't the case for your *.updated files.
Comment 28 Arfrever Frehtes Taifersar Arahesis 2014-07-08 18:12:05 UTC
(In reply to Ulrich Müller from comment #27)
> (In reply to Arfrever Frehtes Taifersar Arahesis from comment #25)
> > Potential solution for problem described in comment #0:
> > 
> > Make Portage dependency resolver store dynamically calculated dependencies
> > for all packages, for which they have been calculated, in separate files in
> > VDB:
> >   /var/db/pkg/${CATEGORY}/${PF}/DEPEND.updated
> >   /var/db/pkg/${CATEGORY}/${PF}/PDEPEND.updated
> >   /var/db/pkg/${CATEGORY}/${PF}/RDEPEND.updated
> > When ebuild for an installed package is deleted, then last calculated
> > dependencies stored in above files will be used.
> 
> This looks like a horrible hack. The VDB should contain information about
> the packages installed on a system (at the time when they have been
> installed), which isn't the case for your *.updated files.

It is better to have a solution for majority of a problem than to wait forever for a perfect solution.

${repository_location}/profiles/updates/* files already cause updating of data stored in VDB.
Comment 29 Ulrich Müller gentoo-dev 2014-07-08 18:17:00 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #28)
> > This looks like a horrible hack. The VDB should contain information about
> > the packages installed on a system (at the time when they have been
> > installed), which isn't the case for your *.updated files.
> 
> It is better to have a solution for majority of a problem than to wait
> forever for a perfect solution.

Then where do we stop? Certainly we also need IUSE.updated and LICENSE.updated, because these could have changed, too. Or the ebuild could have acquired a pkg_postrm() function, so there should be environment.updated.bz2.
Comment 30 Ciaran McCreesh 2014-07-08 18:18:28 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #28)
> It is better to have a solution for majority of a problem than to wait
> forever for a perfect solution.

Good news! There is no need to wait to get the perfect solution. We can have it right now. All it takes is for Portage to remove the feature, and for QA to remind developers to do dependencies properly.
Comment 31 Arfrever Frehtes Taifersar Arahesis 2014-07-08 18:22:49 UTC
(In reply to Ulrich Müller from comment #29)
> (In reply to Arfrever Frehtes Taifersar Arahesis from comment #28)
> > > This looks like a horrible hack. The VDB should contain information about
> > > the packages installed on a system (at the time when they have been
> > > installed), which isn't the case for your *.updated files.
> > 
> > It is better to have a solution for majority of a problem than to wait
> > forever for a perfect solution.
> 
> Then where do we stop? Certainly we also need IUSE.updated and
> LICENSE.updated, because these could have changed, too. Or the ebuild could
> have acquired a pkg_postrm() function, so there should be
> environment.updated.bz2.

Even when an ebuild for an installed package is still available, pkg_postrm() from original environment will be used, so there is no need for environment.updated.bz2.
Comment 32 Ciaran McCreesh 2014-07-08 18:27:52 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #31)
> Even when an ebuild for an installed package is still available,
> pkg_postrm() from original environment will be used, so there is no need for
> environment.updated.bz2.

But the pkg_postrm in environment assumes that the old dependencies are available, not whatever the dynamic dependencies might be.
Comment 33 Martin Väth 2014-07-08 18:28:42 UTC
Another idea (perhaps already suggested somewhere?):

It seems that all issues would be solved if one would revbump every dependency change (not distinguishing between "minor" or "major" which can always be wrong for certain use cases and which is probably the reason why opinions about the correct behaviour of the PM differ).
Currently, the disadvantage of this is the recompilation which is really serious: One cannot expect the user to recompile libreoffice every few days.

But how about introducing a feature to tell the PM that dependencies are the only change and that he could just update them? (Maybe this could eventually be extended for certain other minor changes which certainly do not cause recompilation but might deserve a revision bump anyway.)

There are probably many ways to implement this. Something obvious might be a secondary revision number like -r1.2 (which ages ago had been used in prefix-portage for similar purposes). Of course, this would need a new EAPI and thus would start to take effect only very slowly, but we are talking about a long-term fix, don't we?

It would not be a trivial project since it would require some other things to be implemented as well (e.g. updating of *.tbz2 files comes to my mind).
Comment 34 Alexander Berntsen (RETIRED) gentoo-dev 2014-07-09 14:33:14 UTC
Friends,

I agree with Michał & Sebastian.


Michał, you seem to have a good grip on both sides of the story here. What I would like for you to do is to make a subpage here: <https://wiki.gentoo.org/wiki/Project:Portage> where you tersely present problems in a "What is the problem" section. Then you list the two solutions: "Remove dynamic deps" and "Do dynamic deps properly", with a poignant note on why each of them are plausible strategies. Also add a non-solution, "Do nothing", and what that implies. I can help you with this -- and so can anyone else with wiki access.

When this is done, I will call in a Portage meeting to discuss this (or add it to a meeting agenda if there is a meeting coming up). There we will take votes.

When we have decided on a strategy, we document it on its own subpage in the wiki, where we sketch a path to achieve it. This means a list of what needs to be done, the order to do them in, and so forth.
Comment 35 Alexander Berntsen (RETIRED) gentoo-dev 2014-07-09 18:11:19 UTC
Three cheers for Michał, who added a wiki page: <https://wiki.gentoo.org/wiki/Project:Portage/Dynamic_dependencies>.
Comment 36 Samuli Suominen (RETIRED) gentoo-dev 2014-07-22 08:41:07 UTC
(In reply to Martin Väth from comment #33)
> But how about introducing a feature to tell the PM that dependencies are the
> only change and that he could just update them? (Maybe this could eventually
> be extended for certain other minor changes which certainly do not cause
> recompilation but might deserve a revision bump anyway.)

Something like this should be mandatory requirement before disabling dynamic deps by default. And use of the feature should be enforced by a policy.
Comment 37 Michael Haubenwallner (RETIRED) gentoo-dev 2014-07-23 18:51:24 UTC
(In reply to Martin Väth from comment #33)
> There are probably many ways to implement this. Something obvious might be a
> secondary revision number like -r1.2 (which ages ago had been used in
> prefix-portage for similar purposes).

For secondary revision numbers: When using some Gentoo portage tree as a meta-distribution, it is important to have another level of revision numbers - as can be seen with Gentoo/Prefix already, using Gentoo/Linux as meta-distro. And myself uses Gentoo/Prefix as meta-distro for another distro. 

Actually, I would have loved to have more revision number levels than currently supported by prefix-portage, one per meta-distro stacking level, resulting in:
PN-PV-rA.B.C.D.ebuild
       | | | |
       | | | `- Haubison/distro (hypothetically)
       | |  `- Haubi/Distro
       |  `- Gentoo/Prefix
        `- Gentoo/Linux

The only reason I can live without C-level is because of full commit rights to B-level, while the reason to need B-level (in Prefix) is because of restricted commit rights to A-level.

Just another use-case to keep in mind, please!
Comment 38 Zac Medico gentoo-dev 2014-09-22 17:17:39 UTC
Here is the council's decision on this topic, from http://www.gentoo.org/proj/en/council/meeting-logs/20140826-summary.txt:

Vote:
- "The council asks the Portage team to first outline their long-term
  plan regarding removal or replacement of dynamic dependencies,
  before they remove this feature. In particular, tree policies and
  the handling of eclasses and virtuals need to be clarified."
  Accepted unanimously.