Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 458866

Summary: Add the contents of /var/db/pkg specified to the PMS specs.
Product: Gentoo Hosted Projects Reporter: Anthony Basile <blueness>
Component: PMS/EAPIAssignee: PMS/EAPI <pms>
Status: RESOLVED DUPLICATE    
Severity: enhancement CC: dwfreed, zmedico
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: https://wiki.gentoo.org/wiki/GLEP:64
See Also: https://bugs.gentoo.org/show_bug.cgi?id=290428
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 458030    

Description Anthony Basile gentoo-dev 2013-02-23 15:44:09 UTC
NEEDED.ELF.2 (found under /var/db/pkg/<cat>/<pkg>) contains all the linking information between the package executables and shared objects on a system.  This is important information if one wants to construct a complete linkage graph as does the migrate-pax script in sys-apps/elfix, or revdep-rebuild.  However, NEEDED.ELF.2 is not required in the PMS specs and is not made available by paludis, see bug #458030.

While we can always fall back on readelf -d this is very slow and repeats work already done during emerging.  Eg. consider how slow revdep-rebuild is.

We should make NEEDED.ELF.2 required by the PMS specs.
Comment 1 Ciaran McCreesh 2013-02-23 16:08:39 UTC
VDB isn't specified by PMS, and isn't accessible to ebuilds. This is outside our scope.
Comment 2 Anthony Basile gentoo-dev 2013-02-23 17:31:23 UTC
The factual question of whether or not VDB is currently specified in the PMS specs does not answer the design question of whether or not it *ought* to be specified in the PMS specs.

For the above reason, and others, I assert that it *ought* to be specified.

Discuss in 10,000 or more words.  I will be grading on grammer and punctution!
Comment 3 Ciaran McCreesh 2013-02-23 17:35:21 UTC
It ought not to be, because it's not relevant for ebuilds. If you want the scope of PMS expanded, you'll need to a) get Council approval, b) write a spec, and c) check that spec against current and historical versions of package manglers. But I'd be opposed to a), since VDB is a bad format and we want to move away from anything relying upon it. In other words, if you care what's in VDB and you're not writing a package mangler, you're doing something terribly wrong.
Comment 4 Anthony Basile gentoo-dev 2013-02-23 20:54:20 UTC
(In reply to comment #3)
> It ought not to be, because it's not relevant for ebuilds. If you want the
> scope of PMS expanded, you'll need to a) get Council approval, b) write a
> spec, and c) check that spec against current and historical versions of
> package manglers. But I'd be opposed to a), since VDB is a bad format and we
> want to move away from anything relying upon it. In other words, if you care
> what's in VDB and you're not writing a package mangler, you're doing
> something terribly wrong.

Okay I'm open about how to store this info, but linkage info *is* part of package management, especially on a rolling release like gentoo because you need to keep track of what elf exes are linking against libraries that get upgraded, and, in the case of PaX, migrate flags along the linkage graph.

If you suggest can suggest a way of doing this with the least impact to current PMS's then the above stated needs are satisfied.
Comment 5 Ciaran McCreesh 2013-02-23 21:02:38 UTC
I don't get what you're trying to solve. What I do get is a) having external scripts try to do anything package mangler related manually is a bad idea, since there's a lot to implement to be able to handle packages *correctly*, and b) anything inherently ELF-centric is probably a bad design.
Comment 6 Anthony Basile gentoo-dev 2013-06-02 01:35:19 UTC
Yet another reason why we need this information is for FEATURES=preserve-libs.  In a rolling release like Gentoo, we must be able to trace dependencies through SONAME's so we can follow bumps in library versions and not break exes.  Recalculating this information every time (as revdep-rebuild and revdep-pax do) rather than calculating it only once at emerge time is wasteful and unnecessary.
Comment 7 Ciaran McCreesh 2013-06-02 06:29:17 UTC
preserve-libs, revdep-rebuild and revdep-pax have nothing to do with PMS. This is still outside our scope.
Comment 8 Anthony Basile gentoo-dev 2013-06-02 09:30:15 UTC
Let this debate happen in a wider audience.
Comment 9 Ciaran McCreesh 2013-06-02 09:35:26 UTC
No, see comment #3 for what you need to do if you want to take this further. There is nothing the PMS team can do about this unless you get the Council to extend the scope of PMS.
Comment 10 Ulrich Müller gentoo-dev 2013-06-02 09:45:03 UTC
What is the goal of this request? Do you want to access the VDB from inside ebuilds, or with other tools?

I tend to agree with Ciaran here, we don't want to cement the VDB format. Otherwise, it might have the same fate as the PMS specification for the metadata cache, which is no longer used by Portage.
Comment 11 Anthony Basile gentoo-dev 2013-06-02 11:05:41 UTC
(In reply to Ulrich Müller from comment #10)
> What is the goal of this request? Do you want to access the VDB from inside
> ebuilds, or with other tools?

The goal is to cache and not have to recalculate linking information that is generated at build time.  Any package management system must keep track of dependencies and one set of dependencies comes from the question "what links against what".  This comes up repeatedly when having to trace back reverse linking dependencies eg. during a library upgrade or pax markings.  Tools like revdep-rebuild are notoriously slow because they have to recalculate this information that could easily have been cached.  Portage's FEATURES="preserve-libs" already makes use of NEEDED.ELF.2.

As for you other quesiton: 1) portage could make this information available to other utilities as it does other information via "import portage".  2) This will not go the way of metadata because it is not some ad hoc time saving.  Linking info is essential to interdependency on any *nix system, so its not going to go away.  Currently it resides only in the dynamic section of ELFs where it is slowly retrieved when you must fix reverse dependencies with revdep-rebuild.  Since PMS has access to that info at build time, it should cache it to speed things up.

The statement that revdep-rebuild has nothing to do with PMS only begs the question: why not?  Why should PMS keep track of linking dependencies?  Evidence that the need is there is the fact that revdep-rebuild was written as was FEATURES=preserve-libs.
Comment 12 Ciaran McCreesh 2013-06-02 12:19:37 UTC
PMS describes ebuilds, and things ebuilds may use. It does not describe tools. The scope of PMS is not to specify the entire package mangler. If you want to change this, talk to the Council.
Comment 13 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-06-02 14:04:50 UTC
If we're to really consider specifying something like that in the PMS, I'd just focus on adding something vague like 'PM is supposed to store this and that' and not getting into implementations details.
Comment 14 Ciaran McCreesh 2013-06-02 14:16:39 UTC
That's of no use, since if we're not specifying how it's stored and how it can be accessed, then it doesn't matter whether it's stored or not. If this is really something needing solving, which it almost certainly isn't, then either we go all out and fully specify VDB (which means we suddenly can't change it after all once we get rid of the last few things using built_with_use), or we have specify some new API for accessing it (lots of work, and previous attempts at this have all been awful).
Comment 15 Anthony Basile gentoo-dev 2013-06-02 18:25:14 UTC
(In reply to Michał Górny from comment #13)
> If we're to really consider specifying something like that in the PMS, I'd
> just focus on adding something vague like 'PM is supposed to store this and
> that' and not getting into implementations details.

In a discussion with ulm, the idea came up that the PMS create an abstraction layer where the internals of how the linking information is cached are hidden and all that is presented is some API.  I think this is a great idea.
Comment 16 Ciaran McCreesh 2013-06-02 18:30:17 UTC
If you're proposing the creation of a full abstract package management API, that's best done as a different project. And be aware that it's an awful lot of work to do a decent job of it (for example, "PackageKit" is not an answer).
Comment 17 Ulrich Müller gentoo-dev 2013-06-02 21:13:21 UTC
(In reply to Anthony Basile from comment #15)
> In a discussion with ulm, the idea came up that the PMS create an
> abstraction layer where the internals of how the linking information is
> cached are hidden and all that is presented is some API.

What I said was this:
<ulm> how about some interface in all PMs to query them for the needed          
      information?

PMs = package managers. This not necessarily means that PMS is the right place to specify it.
Comment 18 Zac Medico gentoo-dev 2013-06-02 21:55:59 UTC
(In reply to Ulrich Müller from comment #10)
> What is the goal of this request? Do you want to access the VDB from inside
> ebuilds, or with other tools?

I think the goal should be compatibility between tools, including package managers and related tools such as revdep-rebuild.

> I tend to agree with Ciaran here, we don't want to cement the VDB format.

Documenting the existing format cements it for the purposes of compatibility between tools as long the specified format is used. However, it does not preclude the introduction of new formats that are entirely incompatible.

> Otherwise, it might have the same fate as the PMS specification for the
> metadata cache, which is no longer used by Portage.

It got replaced by a new incompatible format. Didn't we all expect that this would happen eventually? We certainly expect the VDB format to be replaced with a new incompatible format eventually, don't we? That doesn't stop us from documenting the existing format, for compatibility between tools, as long as the this format is used.
Comment 19 Anthony Basile gentoo-dev 2014-04-07 13:00:09 UTC
This has not fallen off the radar.  Please see

http://www.gentoo.org/proj/en/council/meeting-logs/20130910-summary.txt
Comment 20 Ciaran McCreesh 2014-04-07 14:47:33 UTC
(In reply to Anthony Basile from comment #19)
> This has not fallen off the radar.  Please see
> 
> http://www.gentoo.org/proj/en/council/meeting-logs/20130910-summary.txt

Do you have a proposal for the next EAPI?
Comment 21 Anthony Basile gentoo-dev 2014-04-07 20:13:08 UTC
(In reply to Ciaran McCreesh from comment #20)
> (In reply to Anthony Basile from comment #19)
> > This has not fallen off the radar.  Please see
> > 
> > http://www.gentoo.org/proj/en/council/meeting-logs/20130910-summary.txt
> 
> Do you have a proposal for the next EAPI?

I will.  BTW, the issue came up again because of bug #506276.
Comment 22 Anthony Basile gentoo-dev 2014-11-02 12:28:48 UTC
(In reply to Anthony Basile from comment #21)
> (In reply to Ciaran McCreesh from comment #20)
> > (In reply to Anthony Basile from comment #19)
> > > This has not fallen off the radar.  Please see
> > > 
> > > http://www.gentoo.org/proj/en/council/meeting-logs/20130910-summary.txt
> > 
> > Do you have a proposal for the next EAPI?
> 
> I will.  BTW, the issue came up again because of bug #506276.

This is now GLEP 64
Comment 23 Ulrich Müller gentoo-dev 2017-10-12 14:16:01 UTC

*** This bug has been marked as a duplicate of bug 518630 ***