Will attach refactored revdep-rebuild Reproducible: Always Steps to Reproduce: I did some testing and it seems to work, but I need more testing done, please help :) Here are some targets I tried to meet: 1) No longer deletes or moves /etc/portage/package.mask 2) Should be compatible with freebsd/openbsd find as well as gnu find 3) Takes advantage of portage-utils, pquery, and equery features when possible 4) Implemented progress meter 5) Fewer pipes and external calls, better modern bash/find/portage syntax 6) Protected ~/ from accidental rm -f .*
Created attachment 123727 [details] revdep-rebuild A diff/patch turns out to be larger than the whole thing.
I like it! However, it definitely needs testing and debugging. The following is a run on a clean system. Checking reverse dependencies... Packages containing binaries and libraries broken by a package update will be emerged. Collecting system binaries and libraries... done. (/root/.revdep-rebuild.1_files) Collecting complete LD_LIBRARY_PATH... done. (/root/.revdep-rebuild.2_ldpath) Checking dynamic linking consistency... [ 100% ] done. (/root/.revdep-rebuild.3_rebuild) Assigning files to ebuilds... Nothing to rebuild rm: missing operand Try `rm --help' for more information. Evaluating package order... done. (/root/.revdep-rebuild.4_ebuilds) All prepared. Starting rebuild... emerge --oneshot --pretend = These are the packages that would be merged, in order: Calculating dependencies | emerge: there are no ebuilds to satisfy "=".
Created attachment 123795 [details, diff] revdep-rebuild.patch This patch addresses the bugs in comment #2. Additionally I made the following changes: 1) emerge by package names instead of exact versions is now the *default* behavior. You can still build via explicit versions via the -e option. 2) users can disable the use of various utilities such as portage-utils, pkgcore, and equery via the -u option 3) users can skip the ordering packages step via the -o option
I'm working on a patch, but of you beat me to it. --ignore no longer works and --verbose doesn't set --verbose as an emerge option. I'm also getting the following: Checking reverse dependencies... Packages containing binaries and libraries broken by a package update will be emerged. Collecting system binaries and libraries... done. (/home/pvarner/.revdep-rebuild.1_files) Collecting complete LD_LIBRARY_PATH... done. (/home/pvarner/.revdep-rebuild.2_ldpath) Checking dynamic linking consistency... [ 100% ] done. (/home/pvarner/.revdep-rebuild.3_rebuild) Assigning files to packages..../revdep-rebuild: line 628: /home/pvarner/.revdep-rebuild.3_rebuild: No such file or directory done. (/home/pvarner/.revdep-rebuild.4_packages_raw, /home/pvarner/.revdep-rebuild.4_package_owners) Cleaning list of packages to rebuild...sort: open failed: /home/pvarner/.revdep-rebuild.4_packages_raw: No such file or directory done. (/home/pvarner/.revdep-rebuild.4_packages) Assigning packages to ebuilds... Nothing to rebuild Evaluating package order..../revdep-rebuild: line 637: /home/pvarner/.revdep-rebuild.4_ebuilds: No such file or directory done. (/home/pvarner/.revdep-rebuild.4_ebuilds) ./revdep-rebuild: line 677: /home/pvarner/.revdep-rebuild.4_ebuilds: No such file or directory Dynamic linking on your system is consistent... All done.
Created attachment 123821 [details, diff] revdep-rebuild.patch 1) unbroke --ignore 2) --verbose gets passed to portage 3) Shortcut exits if dynamic linking is consistent Hopefully this addresses all the bugs in comment #4
I will probably dual release this in gentoolkit-0.2.4_pre6 with goal being to replace the current version when gentoolkit-0.2.4 is released.
$ svn commit -m "Added refactored revdep-rebuild from Michael A. Smith (Bug #184042)" Sending ChangeLog Sending src/revdep-rebuild/Makefile Adding src/revdep-rebuild/revdep-rebuild-rewrite Transmitting file data ... Committed revision 411.
Created attachment 124074 [details, diff] revdep-rebuild.patch 1) Fixed: [[ avoid_utils ... ]] should have been [[ $avoid_utils ... ]] 2) Fixed: Old environment is remembered in $LIST.0_env and compared with current environment. If they don't match, tempfiles are deleted. 3) Removed references to bugs 23018 and 4698 as they no longer apply -- you may want to close bug 41495 as it seems to be resolved by using the --deep option.
Checking reverse dependencies... Packages containing binaries and libraries broken by a package update will be emerged. Collecting system binaries and libraries... done. (/root/.revdep-rebuild.1_files) Collecting complete LD_LIBRARY_PATH... done. (/root/.revdep-rebuild.2_ldpath) Checking dynamic linking consistency... awk: cmd. line:4: fatal: cannot open file `/root/.revdep-rebuild.3_ldd_errors' for reading (No such file or directory) Dynamic linking on your system is consistent... All done. This was caused by an empty .revdep-rebuild.1_files Collecting system binaries and libraries...+ [[ -r /root/.revdep-rebuild.1_files ]] + rm -f '/root/.revdep-rebuild.[1-9]_*' + [[ -n /root/.revdep-rebuild ]] + [[ -n revdep-rebuild ]] + for i in '$@' + [[ -f = -* ]] + for i in '$@' + [[ /root/.revdep-rebuild.[1-9]_* = -* ]] + [[ /root/.revdep-rebuild.[1-9]_* = *.revdep-rebuild* ]] + command rm -f '/root/.revdep-rebuild.[1-9]_*' + rm -f '/root/.revdep-rebuild.[1-9]_*' + set_trap '/root/.revdep-rebuild.1_*' + trap 'rm_temp /root/.revdep-rebuild.1_*' SIGHUP SIGINT SIGQUIT SIGABRT SIGTERM + findMask=($SEARCH_DIRS_MASK) + findMask= + findMask= + find /bin /lib /opt/bin /sbin /usr/bin /usr/i686-pc-linux-gnu/gcc-bin/4.1.2 /usr/i686-pc-linux-gnu/lib /usr/lib /usr/lib/gcc/i686-pc-linux-gnu/4.1.1 /usr/lib/gcc/i686-pc-linux-gnu/4.1.2 /usr/lib/nspr /usr/lib/nss /usr/lib/opengl/xorg-x11/lib /usr/lib/qt4 /usr/libexec /usr/local/lib /usr/qt/3 /usr/qt/3/bin /usr/qt/3/lib /usr/sbin '(' ')' -prune -o -type f '(' -executable -o -name '*.so' -o -name '*.so.*' -o -name '*.la' ')' -print + sort -u + echo -e ' done.\n (/root/.revdep-rebuild.1_files)' done. (/root/.revdep-rebuild.1_files) + set +x Collecting complete LD_LIBRARY_PATH... done. (/root/.revdep-rebuild.2_ldpath) Checking dynamic linking consistency... awk: cmd. line:4: fatal: cannot open file `/root/.revdep-rebuild.3_ldd_errors' for reading (No such file or directory) Dynamic linking on your system is consistent... All done. vmware ~ # find /bin /lib /opt/bin /sbin /usr/bin /usr/i686-pc-linux-gnu/gcc-bin/4.1.2 /usr/i686-pc-linux-gnu/lib /usr/lib /usr/lib/gcc/i686-pc-linux-gnu/4.1.1 /usr/lib/gcc/i686-pc-linux-gnu/4.1.2 /usr/lib/nspr /usr/lib/nss /usr/lib/opengl/xorg-x11/lib /usr/lib/qt4 /usr/libexec /usr/local/lib /usr/qt/3 /usr/qt/3/bin /usr/qt/3/lib /usr/sbin '(' ')' -prune -o -type f '(' -executable -o -name '*.so' -o -name '*.so.*' -o -name '*.la' ')' -print find: invalid expression; empty parentheses are not allowed. vmware ~ # emerge -pv findutils These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] sys-apps/findutils-4.3.7 USE="nls (-selinux) -static" 0 kB Total: 1 package (1 reinstall), Size of downloads: 0 kB
Created attachment 124178 [details, diff] revdep-rebuild.patch 1) Fixed the bug in comment #9 2) Implemented /etc/init.d/functions.sh stuff to improve consistency with other Gentoo apps 3) If UID!=0, revdep-rebuild will automatically append --pretend to EMERGE_OPTIONS. 4) Wrapped blocks of coherent functionality into functions for easier debugging.
--pretend is no longer working correctly revdep-rebuild --verbose --pretend --keep-temp --ignore * Configuring search environment for revdep-rebuild ... EMERGE_OPTIONS="--verbose --pretend" ... * All prepared. Starting rebuild emerge --oneshot --verbose =app-crypt/gnupg-1.9.21 =gnome-base/gnome-applets-2.16.2 =gnome-base/nautilus-2.16.3 =gnome-extra/gnome-screensaver-2.16.2 =net-libs/gnutls-1.4.4-r1 =net-misc/vino-2.16.0-r1
Created attachment 124343 [details] revdep-rebuild The patch is longer than the file again. :) 1) Whoops, I forgot to markup the last two instances of EMERGE_OPTIONS, which is now an array 2) The last time progress() runs successfully it will output a newline, which I think makes the output with functions.sh functions a little cleaner. NOTE: Because of functions.sh revdep-rebuild is dumping a lot of noise to syslog -- I will have to figure out how best to handle this. Are there any Gentoo conventions or tips I should know about?
Created attachment 124352 [details, diff] revdep-rebuild.noequery.patch Output with equery is potentially broken with portage 2.1.2, this patch disables utilizing equery
Created attachment 124370 [details, diff] revdep-rebuild.noequery.patch I should actually read the logic of the script before uploading patches
I think we are getting a lot closer to a release. Here is a minor glitch in output from the conversion to functions.sh * !!! /usr/lib/libgnutls-extra.so.13 not owned by any package is broken !!! * -n -e /usr/lib/libgnutls-extra.so.13 -> (none)
Another couple of minor issues: * Generated new /root/.revdep-rebuild.4_packages_raw and /root/.revdep-rebuild.4_package_owners * Cleaning list of packages to rebuild sort: open failed: /root/.revdep-rebuild.4_packages_raw: No such file or directory * Generated new /root/.revdep-rebuild.4_packages * Assigning packages to ebuilds * Generated new /root/.revdep-rebuild.4_ebuilds * Evaluating package order * Nothing to rebuild. * (The program should have already quit, so this is a minor bug.)
Created attachment 124464 [details, diff] revdep-rebuild.noequery.patch (In reply to comment #13) > Output with equery is potentially broken with portage 2.1.2, this patch > disables utilizing equery I implemented the options to disable utilities in case the users care which ones they want to use, or have a specific problem with one. It does seem a little odd to me that equery and revdep-rebuild, which are packaged together in gentoolkit, should not interact together, but I don't really care enough to dig deep right now. In this patch equery is disabled and isn't listed as an option in --help, and get_file_owner() encapsulates the various different utilities, so I don't have to keep pasting that 'if' block everywhere. ;) (In reply to comment #16) Paul, can you run that again and tell me if ~/$LIST.3_rebuild has any contents, and if so, what? The program should shortcut out if that file is empty, as you can see at the end of main_checks().
Created attachment 124466 [details] revdep-rebuild files Here are the .revdep-rebuild files for the latest issues.
Created attachment 124474 [details, diff] revdep-rebuild.patch OK, I see what happened now. I don't think revdep can be expected to automatically fix breakages if it can't figure out which packages are broken, so when every broken package is unassociated with a package, revdep now quits with a warning. But should it exit 0 or exit 1?
Crap, sorry for the bugspam -- the patch in comment #19 also fixes a bug where revdep would delete all tempfiles if you removed -p|--pretend|-f|--fetchonly.
> (In reply to comment #13) > > Output with equery is potentially broken with portage 2.1.2, this patch > > disables utilizing equery > > I implemented the options to disable utilities in case the users care which > ones they want to use, or have a specific problem with one. It does seem a > little odd to me that equery and revdep-rebuild, which are packaged together > in gentoolkit, should not interact together, but I don't really care enough to > dig deep right now. The problem is that portage-2.1.2/equery and revdep-rebuild are not playing nicely together. Once portage-2.1.3 is stable, it won't be an issue. What is happening is that not all error messages in portage-2.1.2 are going to stderr so the error message is getting into the input going to revdep-rebuild. That is fixed with portage-2.1.3, so it will be safe to enable equery at that point.
Created attachment 124489 [details, diff] revdep-rebuild.patch OK, summarizes orphaned files at the end, and *should* delete tempfiles if the emerge is real and --keep-temp option is not used. I didn't test that last bit, so prove me wrong :)
Created attachment 124704 [details, diff] revdep-rebuild.broken-files.patch Small fix to have the broken files report to only report the files without a corresponding package
Created attachment 124760 [details, diff] revdep-rebuild.patch Includes the fix in comment #23 and attempts to fix bug #183657 by dropping md5* altogether.
Created attachment 125088 [details] revdep-rebuild Latest version from subversion that incorporates all fixes to date
Just documenting some remaining issues * Need to put the staleness check back in for the cache files (bug #34052) * Hangs when running --library expr="/$SONAME_SEARCH/s/^\([^[:space:]]*\).*$/\1/p" doesn't appear to be seeing the tab in the output of ldd * Doesn't print multiple libraries on one line in verbose mode: [ 58% ] * broken /usr/lib/kde3/libk3bflacdecoder.so (requires libFLAC++.so.5 libFLAC.so.7) * Prints the following when package ownership cannot be determined: * !!! /usr/lib/libgnutls-extra.so.13.0.9 not owned by any package is broken !!! * -n -e /usr/lib/libgnutls-extra.so.13.0.9 -> (none)
Paul, I just ran it on my system, which should not need any rebuilding, and it behaved as I'd expect. I'll continue to try it, especially after installs that might need it. Output: WARNING WARNING *** This is a rewritten version of revdep-rebuild *** WARNING WARNING WARNING Please report any bugs to http://bugs.gentoo.org WARNING WARNING In the bug report please include the following information: WARNING emerge --info WARNING A copy of the output from the revdep-rebuild command WARNING A copy of the .revdep-rebuild* files as an attachment WARNING WARNING If the bug is severe, the previous version of revdep-rebuild is located WARNING at: /usr/lib/gentoolkit/bin/revdep-rebuild WARNING WARNING WARNING *** This is a rewritten version of revdep-rebuild *** WARNING * Configuring search environment for revdep-rebuild-rewrite * Checking reverse dependencies * Packages containing binaries and libraries broken by a package update * will be emerged. * Collecting system binaries and libraries * Generated new /root/.revdep-rebuild-rewrite.1_files * Collecting complete LD_LIBRARY_PATH * Generated new /root/.revdep-rebuild-rewrite.2_ldpath * Checking dynamic linking consistency [ 100% ] * Dynamic linking on your system is consistent... All done.
Created attachment 125136 [details, diff] revdep-rebuild.patch The current ordering code is not correct, this patch fixes that.
(In reply to comment #26) > Just documenting some remaining issues > > * Need to put the staleness check back in for the cache files (bug #34052) I'm pretty sure I didn't remove that... Is it not working? # If any of our temporary files are older than 1 day, remove them all [[ ! $keep_tempfiles && -r $LIST && $( find -L "$LIST" -type f -mmin +1440 -print | while read; do echo 1; break; done ) ]] && rm -f $LIST.* Or are you talking about removing the files that don't match previous runs? (That I did remove, but I added it back later.) if [[ $old_env != $new_env ]]; then ewarn 'Environment mismatch from previous run, deleting temporary files...' rm -f "$LIST"* fi
(In reply to comment #29) > > * Need to put the staleness check back in for the cache files (bug #34052) > I'm pretty sure I didn't remove that... Is it not working? > > # If any of our temporary files are older than 1 day, remove them all > [[ ! $keep_tempfiles && -r $LIST && > $( > find -L "$LIST" -type f -mmin +1440 -print | > while read; do echo 1; break; done > ) ]] && rm -f $LIST.* It is not working. # ls -al .revdep-rebuild.* -rw-r--r-- 1 root root 986 Jun 17 11:04 .revdep-rebuild.0_env -rw-r--r-- 1 root root 259786 Jun 17 11:04 .revdep-rebuild.1_files -rw-r--r-- 1 root root 13185 Jun 17 11:04 .revdep-rebuild.2_ldpath -rw-r--r-- 1 root root 1350 Jun 17 11:04 .revdep-rebuild.3_ldd_errors -rw-r--r-- 1 root root 317 Jun 17 11:04 .revdep-rebuild.3_rebuild -rw-r--r-- 1 root root 130 Jun 17 11:04 .revdep-rebuild.4_ebuilds -rw-r--r-- 1 root root 609 Jun 17 11:04 .revdep-rebuild.4_package_owners -rw-r--r-- 1 root root 130 Jun 17 11:04 .revdep-rebuild.4_packages -rw-r--r-- 1 root root 303 Jun 17 11:04 .revdep-rebuild.4_packages_raw -rw-r--r-- 1 root root 145 Jun 17 11:04 .revdep-rebuild.5_order -rw-r--r-- 1 root root 2 Jun 17 11:04 .revdep-rebuild.6_status garath ~ # revdep-rebuild --pretend --verbose * Configuring search environment for revdep-rebuild * Checking reverse dependencies * Packages containing binaries and libraries broken by a package update * will be emerged. * Collecting system binaries and libraries * Found existing /root/.revdep-rebuild.1_files * Collecting complete LD_LIBRARY_PATH * Found existing /root/.revdep-rebuild.2_ldpath. * Checking dynamic linking consistency * Found existing /root/.revdep-rebuild.3_rebuild. * Assigning files to packages * Found existing /root/.revdep-rebuild.4_packages_raw * Cleaning list of packages to rebuild * Found existing /root/.revdep-rebuild.4_packages * Assigning packages to ebuilds * Found existing /root/.revdep-rebuild.4_ebuilds * Evaluating package order * Found existing /root/.revdep-rebuild.5_order * Generated new /root/.revdep-rebuild.5_order * All prepared. Starting rebuild
Created attachment 125180 [details, diff] revdep-rebuild.library.patch Patch to fix the issue with --library hanging
Created attachment 125187 [details, diff] revdep-rebuild.patch Found another issue with --library, this patch fixes it and includes the previous two patches
Created attachment 125206 [details, diff] revdep-rebuild-rmOldTempfiles.patch This is a saner approach to handling old tempfiles that hopefully resolves issues such as in comment #26
Another small issue SEARCH_DIRS_MASK is no longer expanded so the following will no longer work: env SEARCH_DIRS_MASK="/usr/lib*/libswt-*.so" revdep-rebuild ... This semi-undocumented feature is used to mask out specific files from being checked. (I know the variable says directories, but it is also intended to support files as well)
Created attachment 125347 [details, diff] revdep-rebuild.patch OK, I think this fixes the bug in comment #34: Here are some results, please let me know if they're not the intended behavior. # export -n SEARCH_DIRS_MASK # ./revdep-rebuild -v -i -p -k <SNIP> # cp ~/.revdep-rebuild.1_files revdep-rebuild.1_files-noExtSDMASK # export SEARCH_DIRS_MASK='/usr/*bin/gss-*' # ./revdep-rebuild -v -i -p -k <SNIP> # diff revdep-rebuild.1_files-noenvSDMASK ~/.revdep-rebuild.1_files 911d910 < /usr/bin/gss-client 8033d8031 < /usr/sbin/gss-server And just for completeness' sake: # env SEARCH_DIRS_MASK="/usr/*bin/gss-*" ./revdep-rebuild -v -i -p -k <SNIP> # diff revdep-rebuild.1_files-noenvSDMASK ~/.revdep-rebuild.1_files 911d910 < /usr/bin/gss-client 8033d8031 < /usr/sbin/gss-server
Created attachment 125349 [details, diff] revdep-rebuild_ignoreVerbose_exportNOCOLOR.patch Oops, forgot to do this trivial bit. Now changes in --verbose will not cause revdep-rebuild to remove tempfiles. Also exports NOCOLOR properly; however I noticed that functions.sh does not respect the NOCOLOR variable -- I'll look into functions.sh independently and report a bug.
Created attachment 125357 [details, diff] revdep-rebuild.patch OK, hacked around /etc/init.d/functions.sh's annoying NOCOLOR handling. Now revdep-rebuild respects the -C type flags and the NOCOLOR environment variable. Hope I didn't break anything along the way. I'm kinda peeved at how hard this particular step was. If we used tput instead of terminal codes for color management we wouldn't even need a NOCOLOR variable. tput should be termsafe no matter what.
Created attachment 125494 [details, diff] revdep-rebuild.patch I didn't change much logic here. I removed or localized some global variables and documented all the global variables I couldn't easily localize.
Released in gentoolkit-0.2.4_pre6
Oops! [[ -rs file ]] is not a valid test -- while test -rs and [ -rs "file" ] throw errors, [[ -rs file ]] happily returns the same results as [[ -r file ]] without regard to the value of [[ -s file ]]. I reported this as a bug to the GNU bash team, but in the meantime this sed should fix revdep-rebuild-rewrite. sed -r 's/\[\[\ -rs ([^[:space:]]*)\ \]\]/[[ -r \1 \&\& -s \1 ]]/g' revdep-rebuild-rewrite
Patch applied in subversion. I want to fix a couple of other bugs before releasing pre7.
If the emerge run by the new revdep-rebuild fails, revdep-rebuild still exits with a return code of 0. Since the operation failed, the return code should be something other than zero. E.g.: $ sudo revdep-rebuild -X -- cate-gory/pack-age; echo $? [...lots of output...] * All prepared. Starting rebuild emerge --oneshot -X cate-gory/pack-age =media-libs/tunepimp-0.5.3 =media-libs/xine-lib-1.1.7 =net-dns/pdns-2.9.21 .......... usage: emerge [options] emerge: error: no such option: -X * * revdep-rebuild failed to emerge all packages. [...some more output...] 0 $ I was using revdep-rebuild in a number of local scripts and was dismayed when checking the return code (with bash's || and && operators) no longer functioned properly.
can you add a --no-progress (guess pipe detection isn't really feasible in bash) to revdep-rebuild? the new progress thingie really doesn't work well with pastebin scripts and --quiet is too quiet for that purpose imo.
Created attachment 129344 [details, diff] rr_smart-progress.patch This is a patch against revision 431 http://sources.gentoo.org/viewcvs.py/*checkout*/gentoolkit/trunk/src/revdep-rebuild/revdep-rebuild-rewrite?rev=431 It enables stdout tty detection and adds an option to manually disable just the progress meter without sending --quiet to portage.
Sorry I missed the latest patch in my working to get pre7 out the door. Anyhow, the smart-progress patch is now in subversion. $ svn commit -m "Add --no-progress option" Sending revdep-rebuild/revdep-rebuild-rewrite Transmitting file data . Committed revision 438.
