Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 194987 - Yet another revdep-rebuild rewrite
Summary: Yet another revdep-rebuild rewrite
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Third-Party Tools (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Portage Tools Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-10-07 12:53 UTC by TGL
Modified: 2017-01-29 13:51 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
rdrb (rdrb,16.09 KB, text/plain)
2007-10-07 12:55 UTC, TGL
Details
rdrb (rdrb,16.19 KB, text/plain)
2007-10-07 13:25 UTC, TGL
Details
rdrb (rdrb,23.38 KB, text/plain)
2007-10-14 17:23 UTC, TGL
Details

Note You need to log in before you can comment on or make changes to this bug.
Description TGL 2007-10-07 12:53:57 UTC
Hi,

I'm using a custom script for my revdep-rebuild needs, which i thought you might want to have a look at, since it has several important differences compared to the original one.  So i've added comments and debug mode, done a few cleanups, and here it is.  I'm not submitting it as a candidate for replacing revdep-rebuild, but more as a proof of concept for some possible enhancements.

The main difference beetween revdep-rebuild and my script (rdrb) is that revdep-rebuild scans the system and then assign broken files to packages, whereas my script checks packages one by one, taking advantage of the "NEEDED" files which are in the VDB entries.  Doing that makes it faster (~3 minutes vs. ~8 minutes on my laptop, with a cold I/O cache), because:

 - there's no need to run "find" on half your system, since all files paths are already known,

 - for the same reason, there can't be duplicate files checks because of directory symlinks, and there's no scanning of statically linked files or things like that,

 - there's no need to run objdump or scanelf to list NEEDED libs of broken binaries, since the info is already read from the VDB,

 - there's no need to run qfile (or to grep CONTENTS), since we always know what package we are currently looking at.


Sure, this method also has drawbacks:

 - some old VDB entries may have been created before Portage started to use pax-utils, and thus may not have the required NEEDED file.  I don't think it's a big issue though, it would be easy to write a script which creates the missing NEEDED files by running scanelf on the CONTENTS objs of affected packages.

 - it won't detect broken binaries which have been installed without Portage.  Personnaly, i don't mind (it's out of the scope imho), but some people might see that as a feature loss.


On the other hand, this method also allows have some functional advantages:

 - there is no more need for SEARCH_DIRS (not a killer-feature, but one less config var to take care of is always welcome),

 - it becomes easy to exclude some usual false positives of binary packages ("/opt/foo/bin/foo" missing "libfoo.so", when "libfoo.so" actually has been installed in "/opt/foo/lib/" by the package itself),

 - it becomes easy to exclude some packages by there names (bug #139455): as examples, i've set defaults which exclude "*/*-bin" and "dev-java/sun-*" from the needed libs checking, and "sys-devel/gcc" from the .la scanning.


Also implemented in this script is a "turbo" mode, which is a less reliable but very fast needed-libs checking.  It simply extracts the list of names of all the  required libs from the NEEDED files, and run "qfile -o" on it (keeping only those that are not provided by any package).  
Then, it only ldd-scans the packages which depend on some of this libs (running ldd is still required, because the libs may have been installed by hand, or in a few rare cases be some libfoo.so symlinks to libfoo.so.X which have been created by ldconfig, post-installation, out of the control of the package manager).  
This mode is really intended to be ran after any world update (something many users wouldn't do with revdep-rebuild because of its slowness), since it's often enough to detect just-introduced breakages.


Finally, not implemented in this script, but would be a no-brainer, is a faster "--library" checking: the NEEDED files directly tells you what are the packages you're looking for.  Actually, it's missing because i've been too lazy so far to write some command-line parsing code.


And well, that's about it.  If you wan't to pick some code snippets from this script for integration into revdep-rebuild, you're very welcome.  If you are interested in enhancing it to the point of replacing revdep-rebuild, then i can help, just ask.
Comment 1 TGL 2007-10-07 12:55:23 UTC
Created attachment 132818 [details]
rdrb
Comment 2 TGL 2007-10-07 13:25:46 UTC
Created attachment 132825 [details]
rdrb

Same with a tiny bug fixed (packages depending on exactly one dynamic library where seen as depending on none).


And while i'm at it, one thing i've forgot to mention about why this version is faster is that it calls `ldd` only once per package, with multiple arguments, instead of once per checked file in revdep-rebuild.
Comment 3 Marius Mauch (RETIRED) gentoo-dev 2007-10-07 15:39:20 UTC
Heh, sounds pretty similar to FEATURES=preserve-libs in portage trunk ;)
Comment 4 TGL 2007-10-07 16:36:02 UTC
(In reply to comment #3)
> Heh, sounds pretty similar to FEATURES=preserve-libs in portage trunk ;)
> 

Great :)
It was a shame that Portage itself was not using this valuable NEEDED files it was producing. Hopefully one day we can finally forget about revdep-rebuild and all its variants.
Comment 5 TGL 2007-10-14 17:23:07 UTC
Created attachment 133457 [details]
rdrb

New version with the following improvements:
 - read options from command line
 - added a "--library"-like mode
 - added limited $ROOT support (.la checking is OK, but dynamic libs checking will only look at the NEEDED files in the VDB, without actually running "ldd")
 - added a quiet mode
 - a few bugfixes