Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 192319 (force-rebuild) - [Future EAPI] Force rebuilds for specific packages when emerging another package
Summary: [Future EAPI] Force rebuilds for specific packages when emerging another package
Status: RESOLVED FIXED
Alias: force-rebuild
Product: Gentoo Hosted Projects
Classification: Unclassified
Component: PMS/EAPI (show other bugs)
Hardware: All Linux
: High normal with 5 votes (vote)
Assignee: PMS/EAPI
URL: http://archives.gentoo.org/gentoo-dev...
Whiteboard: in-eapi-5
Keywords:
: 193036 194340 199023 201712 262628 308597 333125 350265 365899 381069 387727 413619 (view as bug list)
Depends on:
Blocks: future-eapi 279623 327809 1991 253802 300866 312539 345331 350320 365899 370439
  Show dependency tree
 
Reported: 2007-09-12 16:43 UTC by Doug Goldstein
Modified: 2020-10-25 17:00 UTC (History)
29 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Doug Goldstein gentoo-dev 2007-09-12 16:43:33 UTC
Basically the idea comes from the linux-header & glibc situation. Whenever you emerge a new linux-headers, technically you need to rebuild your glibc. A lot of users have been bit by the inotify issue (glibc puts stubs for the inotify methods that spit out the stderr that it wasn't implemented if you linux-headers weren't at least "this tall")

<Cardoe> so crazy idea for EAPI=1
<Cardoe> FORCEREBUILD=
<Cardoe> when you emerge that package..
<Cardoe> it would force a rebuild of the listed pkgs automatically
<DrEeevil> Cardoe: interesting, but wouldn't integrating revdep-rebuild functionality into portage be easier?
<Cardoe> yes.
<DrEeevil> Cardoe: I see a few applications that could use it
<Cardoe> but the package combination in question isn't caught by revdep-rebuild
<solar> then it's done pretty much
<DrEeevil> e.g. Qt / kdelibs / kde-apps
<Cardoe> when you emerge a new linux-headers
<Cardoe> you need to rebuild glibc
<Cardoe> however, revdep-rebuild doesn't catch that
<Cardoe> and never will
<DrEeevil> if you rebuild Qt kde-apps will fail because kdelibs are in a "wrong" state
<Cardoe> well my idea was more for "system" packages
<Cardoe> or co-depends
<DrEeevil> the Qt/KDE scenario would profit from it
* antarus has quit (Read error: 110 (Connection timed out))
<Cardoe> but I believe that's an invalid usage
<DrEeevil> well, suggestion for EAPI=0 ... RESTRICT="interactive"
<Cardoe> since not everyone has KDE on their system when they have Qt
<Cardoe> i.e. stuff like MythTV and MythTV themes
<Cardoe> MythTV PDEPENDS on the themes
<DrEeevil> yes, it would have to be conditional then
<Cardoe> when you build a new MythTV... you need to rebuild your themes
<Cardoe> it essentially could just change the behavior of PDEPEND
<zmedico> we could add syntax extensions to indicate things that need to be rebuilt
<zmedico> in *DEPEND
<Cardoe> zmedico: alright. that'd be useful.
<Cardoe> it just seems time and time again, users are bit by that issue
<zmedico> yeah, we certainly need automation for things like that
* wwall has quit (Remote closed the connection)
<zmedico> Cardoe: if you want you can file a bug and make it block bug 174380
<jeeves> zmedico: https://bugs.gentoo.org/174380 enh, P2, All, vapier@gentoo.org->pms-bugs@gentoo.org, NEW, pending, EAPI-1 tracker
<Cardoe> sure
Comment 1 SpanKY gentoo-dev 2007-09-13 05:10:53 UTC
there's no way i'd add that to glibc or linux-headers
Comment 2 Zac Medico gentoo-dev 2007-09-14 03:36:10 UTC
I can see how a feature like this could be annoying if it was mandatory. It doesn't necessarily have to be though. It could be triggered by a certain config setting or command line option, sort of like --newuse. It might be nice to have an automated way to identify packages that are "supposed to be rebuilt", even if the user decides not to rebuild them or to only rebuild a subset of them.
Comment 3 Jakub Moc (RETIRED) gentoo-dev 2007-09-20 12:02:56 UTC
*** Bug 193036 has been marked as a duplicate of this bug. ***
Comment 4 Jakub Moc (RETIRED) gentoo-dev 2007-10-01 07:09:26 UTC
*** Bug 194340 has been marked as a duplicate of this bug. ***
Comment 5 Marius Mauch (RETIRED) gentoo-dev 2007-10-02 09:32:20 UTC
As for portage we could simply add new sets 'rebuild-needed' and/or 'rebuild-recommended' if the necessary information is available.
Comment 6 Jakub Moc (RETIRED) gentoo-dev 2007-11-13 09:49:34 UTC
*** Bug 199023 has been marked as a duplicate of this bug. ***
Comment 7 Jakub Moc (RETIRED) gentoo-dev 2007-12-08 20:01:06 UTC
*** Bug 201712 has been marked as a duplicate of this bug. ***
Comment 8 Santiago M. Mola (RETIRED) gentoo-dev 2007-12-12 14:19:18 UTC
This could be a label in the spec discussed in Bug 201499
Comment 9 Lars Wendler (Polynomial-C) gentoo-dev 2009-03-17 18:41:33 UTC
*** Bug 262628 has been marked as a duplicate of this bug. ***
Comment 10 Zac Medico gentoo-dev 2009-11-23 08:31:50 UTC
The probably cleanest to implement this type of thing as a new type of dependency (similar to EAPI 3 lot operator dependencies: bug 273625). For example, the package manager will generate a special atom in *DEPEND of the installed package, which will specify which version of mythtv the theme package was built against. Then when mythtv is updated to a new version, the dependency will become unsatisfied and therefore it will naturally need to be rebuilt. This approach should work for many types of packages, including xorg driver modules.
Comment 11 David Leverton 2009-12-06 12:46:39 UTC
(In reply to comment #10)
> For example, the package manager will generate a special atom in *DEPEND of the
> installed package, which will specify which version of mythtv the theme package
> was built against.

Does "version" mean ${PVR}, ${PV}, major.minor, etc?  Obviously it's better to force unnecessary rebuilds than to leave things broken, so erring towards the more specific end is better, but if there's a clean way to let the package specify what it needs, it would be nice.
Comment 12 Zac Medico gentoo-dev 2009-12-06 13:25:30 UTC
(In reply to comment #11)
> Does "version" mean ${PVR}, ${PV}, major.minor, etc?

It seems like the package would have to export some kind of metadata that basically corresponds to something like an ABI version. Then the package which depends on it would have the dependency locked to the ABI at build time. This approach would be generalizable to cover the use case that EAPI 3 slot operator dependencies are intended to fill.
Comment 13 Ciaran McCreesh 2009-12-06 19:21:59 UTC
What's wrong with just always slotting on ABI?
Comment 14 Ulrich Müller gentoo-dev 2009-12-06 19:41:43 UTC
(In reply to comment #13)
> What's wrong with just always slotting on ABI?

I doubt that it would be possible for all packages. (Openmotif comes to my mind, where the attempt miserably failed, because the package doesn't support it.)
Comment 15 Ciaran McCreesh 2009-12-06 19:47:36 UTC
You can't even slot and block every other slot?
Comment 16 Ulrich Müller gentoo-dev 2009-12-06 20:14:23 UTC
(In reply to comment #15)
> You can't even slot and block every other slot?

Yes, that's always possible of course. But it sort of defeats the purpose of slots, which is to allow simultaneous installation of different versions.

That slot dependencies are available shouldn't imply that we use (abuse?) slots for the sole purpose of dependency resolution.

Comment 17 Ciaran McCreesh 2009-12-06 20:24:46 UTC
There are already slots of things that block each other in use, so a slot isn't a guarantee that things can be installed in parallel as such.

I'm not really sure that using slots this way is abuse. Perhaps we should be looking at slots as a mechanism for handling incompatible groups of versions of a particular package -- that's a higher level view, and I don't think it's incompatible conceptually with how things are now. If we do take this view, it has the added advantage that existing mechanisms work cleanly with it. We don't need to introduce a new concept that does exactly the same thing in a much less flexible manner.
Comment 18 Zac Medico gentoo-dev 2009-12-07 01:13:40 UTC
Or, rather than shoehorn it into SLOT, we could add a new ABI_SLOT metadata variable to serve this purpose.
Comment 19 Brian Harring gentoo-dev 2009-12-07 01:48:05 UTC
(In reply to comment #18)
> Or, rather than shoehorn it into SLOT, we could add a new ABI_SLOT metadata
> variable to serve this purpose.

Seperate ABI_SLOT'ing is needed last time I looked at this, although when I originally pushed for it I got significant push back from devs (spanky/vapier in particular) due to the extra tracking they have to do of ABI breaks.

It's arguable that they should be doing this already mind you, but I'm just reporting what occurred last time this path was pursued.
Comment 20 Zac Medico gentoo-dev 2009-12-07 08:01:15 UTC
(In reply to comment #19)
> Seperate ABI_SLOT'ing is needed last time I looked at this, although when I
> originally pushed for it I got significant push back from devs (spanky/vapier
> in particular) due to the extra tracking they have to do of ABI breaks.

We can add a pre_pkg_preinst_abi_check ebuild phase that delegates ABI breakage checking to the ebuilds, and implement some automatic checks in the package manager (comparing the new version to the old version just before pkg_preinst).

If an ebuild dev fails to bump the ABI_SLOT when appropriate (for any reason), then it's not a big problem. It will only cause a slight annoyance for a few unstable users.

> It's arguable that they should be doing this already mind you, but I'm just
> reporting what occurred last time this path was pursued.

It will make it possible to plan @preserved-rebuild tasks in advance, which is much nicer than having to detect it after-the-fact as preserve-libs currently does. For example note problems with the current implementation:
  http://blog.flameeyes.eu/2008/06/30/a-few-risks-i-see-related-to-the-new-portage-2-2-preserve-libs-behaviour
Comment 21 Ulrich Müller gentoo-dev 2009-12-07 08:07:18 UTC
(In reply to comment #18)
> Or, rather than shoehorn it into SLOT, we could add a new ABI_SLOT metadata
> variable to serve this purpose.

ABI_SLOT would default to ${SLOT}, unless explicitely specified?

(In reply to comment #19)
> Seperate ABI_SLOT'ing is needed last time I looked at this, although
> when I originally pushed for it I got significant push back from devs
> (spanky/vapier in particular) due to the extra tracking they have to do
> of ABI breaks.

But this argument would also apply if SLOT were used for this purpose.

Comment 22 Brian Harring gentoo-dev 2009-12-07 08:08:08 UTC
(In reply to comment #20)
> We can add a pre_pkg_preinst_abi_check ebuild phase that delegates ABI breakage
> checking to the ebuilds, and implement some automatic checks in the package
> manager (comparing the new version to the old version just before pkg_preinst).

Hell no.  Breakage should be able to be calculated up front- no one wants to come back to what was supposed to be just 2 builds, being 200.

It needs to be a metadata variable so the resolver can act on it- doing ebuild hacks w/ at the time of preinst is going to result in a very bad user experience (especially if somehow a cycle is created).

This is why pkg_pretend is generally a bad idea- it'll be abused in the same fashion.
Comment 23 Zac Medico gentoo-dev 2009-12-07 08:11:28 UTC
(In reply to comment #22)
> (In reply to comment #20)
> > We can add a pre_pkg_preinst_abi_check ebuild phase that delegates ABI breakage
> > checking to the ebuilds, and implement some automatic checks in the package
> > manager (comparing the new version to the old version just before pkg_preinst).
> 
> Hell no.  Breakage should be able to be calculated up front- no one wants to
> come back to what was supposed to be just 2 builds, being 200.

We're saying the same thing. The pre_pkg_preinst_abi_check is intended only as an aid, allowing the ebuild to add some additional checks in addition to whichever ones the package manager uses to warn the ebuild dev that ABI_SLOT needs to be bumped.
Comment 24 Zac Medico gentoo-dev 2009-12-07 08:13:32 UTC
(In reply to comment #21)
> (In reply to comment #18)
> > Or, rather than shoehorn it into SLOT, we could add a new ABI_SLOT metadata
> > variable to serve this purpose.
> 
> ABI_SLOT would default to ${SLOT}, unless explicitely specified?

Maybe. We might also find that it's more useful to leave it unset when it's not specified. I don't know yet.
Comment 25 Brian Harring gentoo-dev 2009-12-07 08:24:21 UTC
(In reply to comment #23)
> We're saying the same thing. The pre_pkg_preinst_abi_check is intended only as
> an aid, allowing the ebuild to add some additional checks in addition to
> whichever ones the package manager uses to warn the ebuild dev that ABI_SLOT
> needs to be bumped.

name it QA then (since it's QA), and since users probably aren't going to want nasty symbol analysis being ran every upgrade checking...
Comment 26 Ciaran McCreesh 2009-12-07 16:09:25 UTC
(In reply to comment #22)
> This is why pkg_pretend is generally a bad idea- it'll be abused in the same
> fashion.

Huh? Kindly explain.
Comment 27 Ciaran McCreesh 2009-12-07 16:10:33 UTC
(In reply to comment #18)
> Or, rather than shoehorn it into SLOT, we could add a new ABI_SLOT metadata
> variable to serve this purpose.

So, what, SLOT's used for what installs in parallel, and ABI_SLOT is used for := dependencies?

I still don't see what's wrong with just making better use of SLOT...
Comment 28 Zac Medico gentoo-dev 2009-12-07 20:34:19 UTC
(In reply to comment #27)
> I still don't see what's wrong with just making better use of SLOT...

"better" is open to interpretation. Somehow I doubt that the majority of the dev community is going to accept the idea of bumping SLOT for every ABI change, but I haven't taken a poll.
Comment 29 Ciaran McCreesh 2009-12-07 20:39:37 UTC
No, not every ABI change. Every non-backwards-compatible ABI change for which you would otherwise be going through and making a huge list of packages (some of which are in overlays) and shoving it in an ebuild.
Comment 30 Zac Medico gentoo-dev 2009-12-07 20:42:24 UTC
(In reply to comment #29)
> No, not every ABI change. Every non-backwards-compatible ABI change for which
> you would otherwise be going through and making a huge list of packages (some
> of which are in overlays) and shoving it in an ebuild.

And also a fairly large number of ABI changes that people currently use revdep-rebuild to handle.
Comment 31 Ciaran McCreesh 2009-12-07 20:49:04 UTC
No, developers can continue forcing people to revdep if they'd rather not solve the problem cleanly.
Comment 32 Zac Medico gentoo-dev 2009-12-07 20:54:14 UTC
That's fine if people choose not use ABI_SLOT in the short term, but I'd expect it to be quite popular since it allows you to avoid having to do a revdep-rebuild due to failures happening part way through a big series of updates.
Comment 33 Zac Medico gentoo-dev 2009-12-10 06:53:55 UTC
Here are some more things to think about:

* Multi-valued ABI_SLOT, for packages that provide multiple interfaces that can change independently. For this we will probably want to allow any number of key/value pairs. The reverse dependencies will have to specify which key(s) they bind to. Atom syntax might be something like {key1=,key2=} which expands to {key1=value1,key2=value2} when the package is locked to a specific interface at build time.

* Packages like python modules that can be built for multiple SLOTs of python simultaneously. This means that the python module dependencies will have to be locked at build time to multiple ABI_SLOT values that are provided by different versions of python that are in different SLOTs.

* Support USE conditionals in ABI_SLOT for cases when a given interface is only available if a certain USE flag is enabled.

* Possible integration of multilib abis. Maybe this isn't desired or necessary since the USE dependency approach that is employed by the gentoo-multilib-overlay seems like it would neither interfere nor need to be coupled with ABI_SLOT.
Comment 34 Torsten Veller (RETIRED) gentoo-dev 2010-08-18 06:28:21 UTC
*** Bug 308597 has been marked as a duplicate of this bug. ***
Comment 35 David Abbott gentoo-dev 2010-08-18 17:36:46 UTC
*** Bug 333125 has been marked as a duplicate of this bug. ***
Comment 36 Samuli Suominen gentoo-dev 2011-01-01 05:56:51 UTC
*** Bug 350265 has been marked as a duplicate of this bug. ***
Comment 37 Doug Goldstein gentoo-dev 2011-01-28 17:47:34 UTC
This would also be extremely useful for xf86-{input,video}-* when you get a new xorg-server installed since many users find they don't have a working keyboard/mouse or video card driver when their X gets updated and they need to log out and complete the upgrade at a later time.

It could also solve a lot of the python-updater and perl-cleaner use cases.
Comment 38 Zac Medico gentoo-dev 2011-04-21 18:31:49 UTC
(In reply to comment #33)
Another thing to think about:

* Indirect ABI dependencies via virtuals need special handling since there can be multiple providers of a given interface, and these providers may or may not be interchangeable at runtime.
Comment 39 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2011-08-29 16:00:54 UTC
*** Bug 381069 has been marked as a duplicate of this bug. ***
Comment 40 Richard 2011-08-29 16:20:41 UTC
I proposed a similar idea in bug #381069, which Jeremy Olexa very kindly pointed out was a duplicate of this. I think my approach would integrate into portage in a way that is easy for ebuild maintainers to maintain, but powerful enough to eliminate the need for revdep-rebuilds in most cases, so I am reposting it here:

> I upgraded from xorg-server 1.10.4 to xorg-server 1.10.4 and went through the
> ritual recompilation of the xorg-drivers following the realization that ABI
> compatibility broke. I would like portage to be able to identify these
> situations beforehand and force a recompilation of the portage modules.
> 
> In order to do this, I propose that portage be modified to read an array of
> tuples containing a pair of regular expressions from ebuilds. The first member
> would be run against the version of the previous package being upgraded. If
> that matches, the second member would be run against the world set of packages
> in the system. Any packages matched by the second member would then be queued
> for recompilation as if they had been passed to emerge with --oneshot. A
> FEATURE flag could be implemented in portage to control whether or not this
> behavior occurs. Additionally, portage could flag the reason for the
> recompilation in verbose output so end users know why these packages are being
> recompiled.
> 
> I imagine that this would predominantly find use in xorg, but it could be used
> in any situation where the package maintainer knows that a revdep-rebuild will
> be required to make things work. Of course, package maintainers would need to
> do additional work to make this work.
> 
> Some sort of script would be needed for package maintainers so that they can
> minimize the possibility of false positives by testing their regular
> expressions against the tree, but even if their regular expressions do generate
> false positives, the worst that will happen is that the false positives will be
> queued for recompilation.

Note that after thinking about this some more, I think that portage could also require that matched ebuilds actually depend on the package being rebuilt by verifying that they have such dependencies before queuing them. That would significantly reduce the impact of false positives.
Comment 41 Richard 2011-08-29 16:22:54 UTC
Just to be clear, "portage modules" is a typo. I meant "xorg modules".
Comment 42 Victor Mataré 2011-12-02 15:31:55 UTC
*** Bug 387727 has been marked as a duplicate of this bug. ***
Comment 43 Zac Medico gentoo-dev 2012-04-26 15:31:13 UTC
*** Bug 413619 has been marked as a duplicate of this bug. ***
Comment 44 Pacho Ramos gentoo-dev 2012-05-15 11:23:16 UTC
(In reply to comment #43)
> *** Bug 413619 has been marked as a duplicate of this bug. ***

In that bug report, I suggested to unify the way used by ebuilds suggesting commands to rebuild things. Currently, each package suggests different commands to be run (some rely on equery, others in qdepends, others in running some customized script). My idea was to create another "ecommand" (or whatever the name is) to only save the command to run, that would allow admins to run that commands via a script after update without needing to surf inside all logs saved in summary.log.
Comment 45 Zac Medico gentoo-dev 2012-06-23 08:25:22 UTC
In portage-2.1.11 there's experimental support for EAPI "4-slot-abi":

http://dev.gentoo.org/~zmedico/portage/doc/portage.html#package-ebuild-eapi-4-slot-abi
Comment 46 Zac Medico gentoo-dev 2012-08-19 21:19:27 UTC
(In reply to comment #45)
> In portage-2.1.11 there's experimental support for EAPI "4-slot-abi":
> 
> http://dev.gentoo.org/~zmedico/portage/doc/portage.html#package-ebuild-eapi-
> 4-slot-abi

This is now included in the eapi-5 branch of PMS:

http://git.overlays.gentoo.org/gitweb/?p=proj/pms.git;a=commit;h=5d6749ac9e5ddc5b1daaad7737b65fa81c6ece47
Comment 47 Ulrich Müller gentoo-dev 2012-09-23 06:32:49 UTC
(In reply to comment #46)
> This is now included in the eapi-5 branch of PMS:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/pms.git;a=commit;
> h=5d6749ac9e5ddc5b1daaad7737b65fa81c6ece47

Commit in master:
http://git.overlays.gentoo.org/gitweb/?p=proj/pms.git;a=commit;h=d4b2fa245918e4704e6cf660575cf2f45c922805
Comment 48 Ulrich Müller gentoo-dev 2020-10-25 17:00:29 UTC
*** Bug 365899 has been marked as a duplicate of this bug. ***