Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 504654 - app-portage/gentoolkit-0.3.0.9 - revdep-rebuild,py UnicodeDecodeError and KeyError tracebacks
Summary: app-portage/gentoolkit-0.3.0.9 - revdep-rebuild,py UnicodeDecodeError and Key...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: ARM Linux
: Normal normal (vote)
Assignee: Portage Tools Team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 346443
  Show dependency tree
 
Reported: 2014-03-15 05:23 UTC by Mike Johnson
Modified: 2022-07-10 01:38 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
strace of revdep-rebuild (revdep-rebuild-problem.bz2,108.75 KB, application/x-bzip)
2014-03-15 05:23 UTC, Mike Johnson
Details
revdep-rebuild (bash) x86 la file errors (x86_la_errors,9.15 KB, text/plain)
2014-03-15 20:57 UTC, Mike Johnson
Details
results of revdep-rebuild -ipkv on x86 (x86-revdep-rebuild-0.3.0.9.tar,500.00 KB, application/x-tar)
2014-03-17 18:05 UTC, Mike Johnson
Details
results of revdep-rebuild -ipkv on ARM (ARM-revdep-rebuild-0.3.0.9.tar,110.00 KB, application/x-tar)
2014-03-17 18:07 UTC, Mike Johnson
Details
x86-revdep-rebuild.py results (temp,12.68 KB, text/plain)
2014-03-18 06:58 UTC, Mike Johnson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Johnson 2014-03-15 05:23:58 UTC
Created attachment 372694 [details]
strace of revdep-rebuild

This error occurs on my armv5tel installation.  Revdep-rebuild repeatedly finds the following:
****************************
[ 58% ]  *   broken /usr/lib/gcc/armv5tel-softfloat-linux-gnueabi/4.8.2/libgomp.la (requires -lrt)
 *   broken /usr/lib/gcc/armv5tel-softfloat-linux-gnueabi/4.8.2/libmudflap.la (requires -ldl)
 *   broken /usr/lib/gcc/armv5tel-softfloat-linux-gnueabi/4.8.2/libmudflapth.la (requires -ldl)
[ 68% ]  *   broken /usr/lib/libdb-4.8.la (requires -lpthread)
[ 69% ]  *   broken /usr/lib/libdb_cxx-4.8.la (requires -lpthread)
 *   broken /usr/lib/libdb_stl-4.8.la (requires -lpthread)
 *   broken /usr/lib/libfl.la (requires -lm)
 *   broken /usr/lib/libfl_pic.la (requires -lm)
[ 71% ]  *   broken /usr/lib/libiksemel.la (requires -lgnutls)
 *   broken /usr/lib/libilbc.la (requires -lm)
[ 72% ]  *   broken /usr/lib/libltdl.la (requires -ldl)
 *   broken /usr/lib/liblua.la (requires -lm)
 *   broken /usr/lib/liblua.la (requires -ldl)
[ 73% ]  *   broken /usr/lib/libsasl2.la (requires -ldl)
 *   broken /usr/lib/libsasl2.la (requires -lresolv)
[ 74% ]  *   broken /usr/lib/libtorrent.la (requires -lcrypto)
 *   broken /usr/lib/libtorrent.la (requires -lsigc-2.0)
 *   broken /usr/lib/libudev.la (requires -lrt)
*****************************
After twice allowing re-emerge of those "problem" files it still does the same.  I then downgraded to gentoolkit-0.3.0.8 and it says the system is fine, finds nothing.  Then I re-emerged (not trusting pkg file) gentoolkit-0.3.0.9 and same results.
gentoolkit-0.3.0.9 appears to work fine on my x86 and amd86 machines.
I have attached an strace of this error in case it helps and of course, will be happy to help any way I can.

# emerge --info
Portage 2.3.6-r5 (funtoo/1.0/linux-gnu/arch/arm-32bit/armv5te, gcc-4.8.2, glibc-2.15-r4, 3.13.5-FUNTOO armv5tel)
=================================================================
System uname: Linux-3.13.5-FUNTOO-armv5tel-with-gentoo-2.2.0
KiB Mem:      121988 total,      9364 free
KiB Swap:    2097148 total,   2077476 free
Timestamp of tree: Unknown
ld GNU ld (GNU Binutils) 2.23.1
distcc 3.1 armv5tel-softfloat-linux-gnueabi [enabled]
ccache version 3.1.9 [disabled]
app-shells/bash:          4.2_p45-r1
dev-lang/python:          2.7.6-r1001, 3.3.4-r1000
dev-util/ccache:          3.1.9-r3
dev-util/cmake:           2.8.11.1
sys-apps/baselayout:      2.2.0-r4
sys-apps/openrc:          0.12.3-r1
sys-apps/sandbox:         2.6-r2
sys-devel/autoconf:       2.69
sys-devel/automake:       1.11.6, 1.13.4
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.8.2-r1
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.2
sys-devel/make:           4.0-r1
sys-kernel/linux-headers: 3.7 (virtual/os-headers)
sys-libs/glibc:           2.15-r4
Repositories: gentoo local
ACCEPT_KEYWORDS="arm ~arm"
ACCEPT_LICENSE="* -@EULA"
CBUILD="armv5tel-softfloat-linux-gnueabi"
CFLAGS="-O2 -pipe -march=armv5te -fomit-frame-pointer"
CHOST="armv5tel-softfloat-linux-gnueabi"
CONFIG_PROTECT="/etc /usr/lib/distcc/bin/c++ /usr/lib/distcc/bin/cc /usr/lib/distcc/bin/g++ /usr/lib/distcc/bin/gcc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/udev/rules.d"
CXXFLAGS="-O2 -pipe -march=armv5te -fomit-frame-pointer"
DISTDIR="/usr/portage_large/distfiles"
FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distcc distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://gentoo.llarian.net/ http://gentoo.mirrors.easynews.com/linux/gentoo/ http://gentoo.gossamerhost.com"
LANG="en_US.utf8"
LC_ALL="en_US.utf-8"
LDFLAGS="-Wl,-O1 -Wl,--sort-common -Wl,--as-needed"
MAKEOPTS="-j16"
PKGDIR="/usr/portage_large/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="git://github.com/funtoo/ports-2012.git"
SYNC_USER="root"
USE="acl alsa arm berkdb bzip2 cracklib crypt curl cxx gdbm gnutls iconv icu modules mudflap ncurses nls nptl openmp openssl pam pcre python readline resolvconf ssl symlink tcpd unicode xml zlib" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias authn_core authz_core socache_shmcb unixd" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_ABIS="2.7 3.3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2 python3_3" RUBY_TARGETS="ruby18 ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev omapfb" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, SYNC_UMASK
Comment 1 Brian Dolbec (RETIRED) gentoo-dev 2014-03-15 16:35:16 UTC
Where does the -r1000 come from?

Could you please try running revdep-rebuild.py from gentoolkit-0.3.0.9.
I've just done a number of updates to it, we are trying to get a little more testing before it becomes the default instead of the old bash version.
Note: it is a lot faster.  It also might come up with an additional pkg or two compared to -0.3.0.8's bash version.  It should be a little more thorough.
It is also a good comparison to the false positives the bash version is reporting.
Comment 2 octoploid 2014-03-15 17:58:09 UTC
I'm also getting tons of false positives in .la files.

Running revdep-rebuild.py doesn't work on my machine:

x4 ~ # revdep-rebuild.py -ipv
 * This is a prerelease version, so it may not work correctly
 * The original revdep-rebuild script is installed as revdep-rebuild.sh
 * Please files bugs at: https://bugs.gentoo.org/
 * Collecting system binaries and libraries
 * Collecting dynamic linking informations
 * Scanning files
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/revdep-rebuild.py", line 40, in <module>
    success = rebuild.main(rebuild.parse_options())
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/rebuild.py", line 243, in main
    assigned, orphaned = analyse(settings, logger, _libs_to_check=_libs_to_check)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/analyse.py", line 325, in analyse
    logger, searchbits)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/analyse.py", line 34, in scan_files
    libs_and_bins, cmd_max_args, logger)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py", line 46, in scan
    ['scanelf'] + params + _files[i:i+max_args]).strip().split('\n')
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py", line 24, in call_program
    stdout = stdout.decode('utf-8')
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 33931: invalid start byte
Comment 3 Brian Dolbec (RETIRED) gentoo-dev 2014-03-15 18:24:27 UTC
(In reply to octoploid from comment #2)
> I'm also getting tons of false positives in .la files.
> 
> Running revdep-rebuild.py doesn't work on my machine:
...
>   File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
>     return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 33931:
> invalid start byte

What are your LANG and LOCALE settings?
Comment 4 octoploid 2014-03-15 19:16:29 UTC
x4 ~ # locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Comment 5 Mike Johnson 2014-03-15 20:01:33 UTC
Only version available in tree, besides 9999:
# eix gentoolkit
[I] app-portage/gentoolkit
     Available versions:  0.3.0.9-r1000{tbz2} **9999

# revdep-rebuild.py
 * This is a prerelease version, so it may not work correctly
 * The original revdep-rebuild script is installed as revdep-rebuild.sh
 * Please files bugs at: https://bugs.gentoo.org/
 * Adding --pretend to portage options
 * Collecting system binaries and libraries
 * Checking dynamic linking consistency
Traceback (most recent call last):
  File "/usr/bin/revdep-rebuild.py", line 40, in <module>
    success = rebuild.main(rebuild.parse_options())
  File "/usr/lib/python3.3/site-packages/gentoolkit/revdep_rebuild/rebuild.py", line 243, in main
    assigned, orphaned = analyse(settings, logger, _libs_to_check=_libs_to_check)
  File "/usr/lib/python3.3/site-packages/gentoolkit/revdep_rebuild/analyse.py", line 335, in analyse
    broken_pathes = libcheck.process_results(libcheck.search())
  File "/usr/lib/python3.3/site-packages/gentoolkit/revdep_rebuild/analyse.py", line 188, in search
    scanned = scanned_files[bits]
KeyError: '64'
Comment 6 Mike Johnson 2014-03-15 20:56:13 UTC
For comparision I ran revdep-rebuild.py on my x86 and amd64 platforms.  The results were surprising.  amd64 seems relatively benign, but the x86 came up with a ton of errors, too many to list.  Now I haven't run revdep-rebuild on the 32 bit for a few days due to unrelated other things I've been working on, but when I run the bash version now, again, a ton of errors like the ARM.  Working on downgrading that version of gentoolkit now to see, I dunno, doesn't seem to want to downgrade anything.  Anyhow, here are the results:
amd64
 * This is a prerelease version, so it may not work correctly
 * The original revdep-rebuild script is installed as revdep-rebuild.sh
 * Please files bugs at: https://bugs.gentoo.org/
 * Adding --pretend to portage options
 * Collecting system binaries and libraries
 * Checking dynamic linking consistency
 * Assign files to packages

 !!! Broken orphaned files: No installed package was found for the following:
	* /usr/lib64/gcc/armv5tel-softfloat-linux-gnueabi/4.8.0/libstdc++.so.6.0.18

There is nothing to emerge. Exiting.

x86
 * This is a prerelease version, so it may not work correctly
 * The original revdep-rebuild script is installed as revdep-rebuild.sh
 * Please files bugs at: https://bugs.gentoo.org/
 * Adding --pretend to portage options
 * Checking dynamic linking consistency
 * Assign files to packages

 !!! Broken orphaned files: No installed package was found for the following:
	* /usr/bin/sp2sp
	* /usr/bin/gwaverepl
	* /usr/bin/smartdimmer
	* /usr/bin/udisks
	* /usr/bin/rzxdump
	* /usr/bin/nvidia-settings
	* /usr/bin/cheese
	* /usr/bin/listbasic
	* /usr/bin/profile2map
	* /usr/bin/chronyc
	* /usr/bin/tzxlist
	* /usr/bin/secret-tool
	* /usr/bin/nvclock
	* /usr/bin/snapconv
	* /usr/bin/nvclock_gtk
	* /usr/bin/blender
	* /usr/bin/rzxtool
	* /usr/bin/createhdf
	* /usr/bin/nm-connection-editor
	* /usr/bin/nm-applet
	* /usr/bin/gwave-exec
	* /usr/bin/wpa_gui
	* /usr/bin/raw2hdf
	* /usr/bin/tapeconv
	* /usr/bin/wine64
	* /usr/bin/snap2tzx
	* /usr/bin/scl2trd
	* /usr/bin/cmake-gui
	* /usr/bin/git-credential-gnome-keyring
	* /usr/bin/reptyr
	* /usr/bin/gwave
Kinda weird, some of those don't belong, for example, it's intel, my amd64 is nvidia.  Just confirmed, there's no nvidia anything installed on the x86.
Oh I see.  Yes indeed, there is a /usr/bin/nvidia-settings, what's that doing there.  So yes it seems the py version is doing the right thing in some ways, but also kicking out many .la files that shouldn't(?) be?  Maybe I'd better give you that list also, I will attach it so as not to gum up this thread.
Comment 7 Mike Johnson 2014-03-15 20:57:13 UTC
Created attachment 372768 [details]
revdep-rebuild (bash) x86 la file errors
Comment 8 Mike Johnson 2014-03-15 22:03:06 UTC
Results of revdep-rebuild (gentoolkit-0.3.0.8-r1001) on x86:
 * Checking dynamic linking consistency
[ 100% ]                 

 * Dynamic linking on your system is consistent... All done.
Comment 9 Paul Varner (RETIRED) gentoo-dev 2014-03-17 16:08:13 UTC
For the broken machines, please run 'revdep-rebuild -ipkv' and attach the full output from revdep-rebuild and the .rr files located in /var/cache/revdep-rebuild
Comment 10 Mike Johnson 2014-03-17 18:05:53 UTC
Created attachment 372868 [details]
results of revdep-rebuild -ipkv on x86
Comment 11 Mike Johnson 2014-03-17 18:07:21 UTC
Created attachment 372870 [details]
results of revdep-rebuild -ipkv on ARM
Comment 12 Paul Varner (RETIRED) gentoo-dev 2014-03-17 19:18:24 UTC
From the perspective of the files, everything looks good.  When looking for la file breakage, revdep-rebuild is looking for the libraries in the paths specified in /etc/ld.so.conf.  So for example from the x86 output:

*   broken /usr/lib/gcc/i686-pc-linux-gnu/4.8.2/libasan.la (requires -lpthread)

It is looking for libpthread.so in any of the directories specified by the ld.so.conf file.  Normally, you find them /lib/libpthread.so  or /usr/lib/libpthread.so.  

Do these exist on your system? If they do, what directory and is that directory in /etc/ld.so.conf?
Comment 13 Mike Johnson 2014-03-17 22:07:33 UTC
Hmm, well that's revealing, I get this on the x86:

# ls /usr/lib/libpthread.so
/usr/lib/libpthread.so
# cat /etc/ld.so.conf|grep /usr/lib
/usr/lib/opengl/xorg-x11/lib
/usr/lib/qca2
/usr/lib/qt4

Actually there isn't much in there at all.  And on my amd64 which has no problem /usr/lib is listed in in /etc/ld.so.conf.
So it appears the problem is local pending further checking here.
I never touch any of that ld stuff unless a specific problem pops up, so I'll have to trace back the mechanism that should be keeping that up to date, it seems perhaps a Funtoo problem.
The x86 was rebuilt 1 month ago, emerge -e @system & @world, unused except for updates, so it should be clean except for the cruft like the /usr/bin/nvidia-settings, etc. that shouldn't be there at all.
Thank you for your attention and knowledge and sorry for the fuss.
Comment 14 Mike Johnson 2014-03-18 06:56:23 UTC
All of the culprits were in /usr/lib on both machines, I've temporarily added LDPATH=/usr/lib to my /etc/env.d/99local until I can get to the bottom of this, the good news is all is well with gentoolkit-0.3.0.9, bash revdep-rebuilds work perfectly.  On my amd64 there is an /etc/env.d/05multilib with LDPATH=/usr/bin which is what spared it from these problems.  An issue for Funtoo...
My purpose in adding to this thread is to give more feedback on the revdep-rebuild.py version for Brian.  It still fails at the same place on the ARM.  It seems to run well on the x86 but finds I need to emerge libreoffice-bin.  It's not installed, I did once have that on this machine but switched over to libreoffice long ago.  Yet it wants to re-emerge both of them.  It also doesn't like my cross-gcc installations and wants to re-emerge those.  It finds the orphans of course, as it should.  I'm attaching the verbose x86 output of revdep-rebuild.py.
Thanks to both Brian and Paul for their assistance.
Comment 15 Mike Johnson 2014-03-18 06:58:37 UTC
Created attachment 372902 [details]
x86-revdep-rebuild.py results
Comment 16 Brian Dolbec (RETIRED) gentoo-dev 2014-03-18 14:21:18 UTC
The arm failure should be fixed in git.  You could try the -9999 version to test.  I believe the only difference with -0.3.0.9 is the arm patch and one typo fix.

Thank you for the testing.  I'll get to your output a little later.
Comment 17 Brian Dolbec (RETIRED) gentoo-dev 2014-03-18 14:42:16 UTC
* Assign files to packages
	* /usr/lib/libreoffice/program/libofficebean.so -> app-office/libreoffice-4.1.4.2
	* /usr/lib/openoffice/program/libofficebean.so -> app-office/openoffice-bin-4.0.1


The python version uses all the filenames it finds it finds in /var/db/pkg/*/*/CONTENTS to match the broken lib up to the pkg it belongs to.

I think you will find that openoffice-bin is not completely uninstalled.  If it was, then those would have been listed with the orphans.

So you will find /var/db/pkg/app-office/openoffice-bin-4.0.1/CONTENTS on your system.  Your x86 box is not as cruft free as you thought.  Also indicated by all the orphans it found.
Comment 18 Mike Johnson 2014-03-18 18:46:33 UTC
-9999 works great on all 3 platforms here.  Of course the x86 has issues.  I never said there was no cruft, but I fear no cruft now armed with the ruthless orphan hunter revdep-rebuild.py.  And yes, it's fast.
Yep, /var/db/pkg/app-office/openoffice-bin/CONTENTS, I got one.  Gonna stick with -9999 for awhile, see how it goes.
Thanks!
Comment 19 Paul Varner (RETIRED) gentoo-dev 2014-03-18 19:13:51 UTC
I'm changing the title to reflect the fixes for revdep-rebuild.py.

The la file false positives in revdep-rebuild.sh appear to be a local configuration issue.
Comment 20 Brian Dolbec (RETIRED) gentoo-dev 2014-03-23 13:49:01 UTC
(In reply to octoploid from comment #2)
> Running revdep-rebuild.py doesn't work on my machine:
> 
> x4 ~ # revdep-rebuild.py -ipv
...

>   File
> "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py",
> line 46, in scan
>     ['scanelf'] + params + _files[i:i+max_args]).strip().split('\n')
>   File
> "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py",
> line 24, in call_program
>     stdout = stdout.decode('utf-8')
>   File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
>     return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 33931:
> invalid start byte

octoploid  Are you still getting this error?

If so, are you up to a slight program edit to debug which file is causing this?
From there we can determine if the problem is in the file or scanelf itself.
Comment 21 octoploid 2014-03-23 15:06:08 UTC
(In reply to Brian Dolbec from comment #20)
> (In reply to octoploid from comment #2)
> > Running revdep-rebuild.py doesn't work on my machine:
> > 
> > x4 ~ # revdep-rebuild.py -ipv
> ...
> 
> >   File
> > "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py",
> > line 46, in scan
> >     ['scanelf'] + params + _files[i:i+max_args]).strip().split('\n')
> >   File
> > "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py",
> > line 24, in call_program
> >     stdout = stdout.decode('utf-8')
> >   File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
> >     return codecs.utf_8_decode(input, errors, True)
> > UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 33931:
> > invalid start byte
> 
> octoploid  Are you still getting this error?
> 
> If so, are you up to a slight program edit to debug which file is causing
> this?
> From there we can determine if the problem is in the file or scanelf itself.

Last time I tried (yesterday gentoolkit-9999) yes.
Sure, just tell me what to edit.
Comment 22 Brian Dolbec (RETIRED) gentoo-dev 2014-03-23 15:27:06 UTC
ok, in gentoolkit/revdep_rebuild/stuff.py line 44:

	for i in range(0, len(_files), max_args):
		output = call_program(
			['scanelf'] + params + _files[i:i+max_args]).strip().split('\n')


add in a print line:

	for i in range(0, len(_files), max_args):
                print('DEBUG: scanning file:', _files[i:i+max_args])
		output = call_program(
			['scanelf'] + params + _files[i:i+max_args]).strip().split('\n')


and in the same directory, settings.py

change: 'CMD_MAX_ARGS': 1000,

to: 'CMD_MAX_ARGS': 1,

that will run scanelf one file at a time, printing the name of the file it is checking. It should error doing the one that fails, so should be the last file printed before the traceback.

From there, save a copy of that file somewhere, then do an equery b $the-file and try re-emerging that pkg.  If it no longer fails, then you had a corrupt file. If it still fails, then there may be an error in the scanelf utility.  That's where the original file will come in handy.  But that will be a new bug for scanelf.
Comment 23 octoploid 2014-03-23 15:35:22 UTC
OK, found the bug:

My filesystem contained broken filenames:

 '/usr/lib/getconf/POSIX_V7
_LP64_OFF64', '/usr/lib/getconf/POSIX_V7_LP64_OFF64?', '/usr/lib/getconf/POSIX_V7_LP64_OFF64R', '/usr/lib/getconf/POSIX_V7_LP64_OFF64x', '/usr/lib/getconf/POSIX_V7_LP64_OFF64
\x84', '/usr/lib/getconf/POSIX_V7_LP64_OFF64\xbe', '/usr/lib/getconf/POSIX_V7_LP64_OFF64\xc0', '/usr/lib/getconf/POSIX_V7_LP64_OFF64\xce', '/usr/lib/getconf/POSIX_V7_LP64_OFF
64\xd1', '/usr/lib/getconf/POSIX_V7_LP64_OFF64\xd9', '/usr/lib/getconf/POSIX_V7_LP64_OFF64\xe1', '/usr/lib/getconf/POSIX_V7_LP64_OFF64\xed', '/usr/lib/getconf/POSIX_V7_LP64_O
FF64\xf0', '/usr/lib/getconf/XBS5_LP64_OFF64', '/usr/lib/gettext/hostname', '/usr/lib/gettext/project-id', '/usr/lib/gettext/urlget',

I really don't know where they are coming from ("equery b" shows nothing).
When I remove them revdep-rebuild run fine:

x4 ~ # revdep-rebuild -ipv
 * This is a prerelease version, so it may not work correctly
 * The original revdep-rebuild script is installed as revdep-rebuild.sh
 * Please files bugs at: https://bugs.gentoo.org/
 * Collecting system binaries and libraries
 * Collecting dynamic linking informations
 * Scanning files
 * Checking dynamic linking consistency
 * Broken files that requires: libperl.so (64 bits)
        * /usr/bin/urxvt
        * /usr/bin/urxvtd
        * /usr/lib64/weechat/plugins/perl.so
 * Broken files that requires: libbfd-2.24.51.20140218.so (64 bits)
        * /usr/lib64/cairo/libcairo-trace.so.0.0.0
 * Broken files that requires: libfbembed.so.2.5 (64 bits)
        * /usr/local/lib64/libreoffice/program/libfirebird_sdbclo.so
         * /usr/lib64/libunwind-x86_64.la is broken (requires: /usr//lib/libunwind.la)
         * /usr/lib/libunwind-setjmp.la is broken (requires: /usr//lib/libunwind-x86_64.la)
         * /usr/lib/libunwind-setjmp.la is broken (requires: /usr//lib/libunwind.la)
         * /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.la is broken (requires: -l/usr/lib/libtcmalloc.so.4)
         * /usr/lib64/libunwind-setjmp.la is broken (requires: /usr//lib/libunwind-x86_64.la)
         * /usr/lib64/libunwind-setjmp.la is broken (requires: /usr//lib/libunwind.la)
         * /usr/lib/libunwind-x86_64.la is broken (requires: /usr//lib/libunwind.la)
 * Assign files to packages
        * /usr/lib64/weechat/plugins/perl.so -> net-irc/weechat-0.4.3-r1
        * /usr/bin/urxvtd -> x11-terms/rxvt-unicode-9.19
        * /usr/bin/urxvt -> x11-terms/rxvt-unicode-9.19
        * /usr/lib64/cairo/libcairo-trace.so.0.0.0 -> x11-libs/cairo-1.12.16

 !!! Broken orphaned files: No installed package was found for the following:
        * /usr/lib64/libunwind-setjmp.la
        * /usr/lib/libunwind-x86_64.la
        * /usr/local/lib64/libreoffice/program/libfirebird_sdbclo.so
        * /usr/lib/libunwind-setjmp.la
        * /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.la
        * /usr/lib64/libunwind-x86_64.la

The libperl.so issue looks bogus. Rest of the issue are from hand compiled packages (outside portage).
Comment 24 Brian Dolbec (RETIRED) gentoo-dev 2014-03-23 15:38:48 UTC
Sorry, this text entry has different settings for tabs.  make sure the print() statement is at the same indent as the output = line  using 2 tabs, not spaces.
Comment 25 octoploid 2014-03-23 15:44:45 UTC
(In reply to Brian Dolbec from comment #24)
> Sorry, this text entry has different settings for tabs.  make sure the
> print() statement is at the same indent as the output = line  using 2 tabs,
> not spaces.

With your debugging patch:
...
DEBUG: scanning file: ['/usr/lib/gegl-0.2/xor.so']
DEBUG: scanning file: ['/usr/lib/getconf/POSIX_V6_LP64_OFF64']
DEBUG: scanning file: ['/usr/lib/getconf/POSIX_V7_LP64_OFF64']
DEBUG: scanning file: ['/usr/lib/getconf/POSIX_V7_LP64_OFF64?']
DEBUG: scanning file: ['/usr/lib/getconf/POSIX_V7_LP64_OFF64R']
DEBUG: scanning file: ['/usr/lib/getconf/POSIX_V7_LP64_OFF64x']
DEBUG: scanning file: ['/usr/lib/getconf/POSIX_V7_LP64_OFF64\x84']
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/revdep-rebuild", line 40, in <module>
    success = rebuild.main(rebuild.parse_options())
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/rebuild.py", line 243, in main
    assigned, orphaned = analyse(settings, logger, _libs_to_check=_libs_to_check)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/analyse.py", line 329, in analyse
    logger, searchbits)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/analyse.py", line 34, in scan_files
    libs_and_bins, cmd_max_args, logger)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py", line 47, in scan
    ['scanelf'] + params + _files[i:i+max_args]).strip().split('\n')
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/stuff.py", line 24, in call_program
    stdout = stdout.decode('utf-8')
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 36: invalid start byte
Comment 26 Brian Dolbec (RETIRED) gentoo-dev 2014-03-23 15:58:23 UTC
Well, I'm glad you found the source of the trouble.  Those files may be a result of some of those manually compiled programs doing some strange things in error.  

As for the libperl issue, there was a perl upgrade recently.  So, they may very well be linked to the old perl version.  Try re-emerging them, if it does not clear them it may be fixed by a perl-cleaner --reallyall run.
It certainly doesn't hurt to run it, especially after a perl upgrade.  It usually fixes perl issues for me.

Now, what to do about that file...

/me cc's blueness
Comment 27 Brian Dolbec (RETIRED) gentoo-dev 2014-03-23 16:04:02 UTC
Anthony, is there something scanelf could/should be trapping to prevent the error octoploid is getting from scanelf on that file? So, a bug in scanelf?  Or is it simply a corrupt file, garbage in garbage out issue?
Comment 28 octoploid 2014-03-23 16:10:48 UTC
(In reply to Brian Dolbec from comment #26)
> Well, I'm glad you found the source of the trouble.  Those files may be a
> result of some of those manually compiled programs doing some strange things
> in error.  
> 
> As for the libperl issue, there was a perl upgrade recently.  So, they may
> very well be linked to the old perl version.  Try re-emerging them, if it
> does not clear them it may be fixed by a perl-cleaner --reallyall run.
> It certainly doesn't hurt to run it, especially after a perl upgrade.  It
> usually fixes perl issues for me.

Thanks. Yes, re-emerging fixed the issue.

One last question before I'm out of here:

 # env SEARCH_DIRS="/ -*" SEARCH_DIRS_MASK="/var" revdep-rebuild -ipv

doesn't seem to work with the new python version. How can
I achieve the same with the python version?
Comment 29 Sławek Lis (RETIRED) gentoo-dev 2014-03-26 21:08:26 UTC
(In reply to octoploid from comment #28)
> One last question before I'm out of here:
> 
>  # env SEARCH_DIRS="/ -*" SEARCH_DIRS_MASK="/var" revdep-rebuild -ipv
> 
> doesn't seem to work with the new python version. How can
> I achieve the same with the python version?

Can you please check the latest version and tell if it's working as supposed to?

Thanks!
Comment 30 octoploid 2014-03-27 06:36:35 UTC
(In reply to Sławek Lis from comment #29)
> (In reply to octoploid from comment #28)
> > One last question before I'm out of here:
> > 
> >  # env SEARCH_DIRS="/ -*" SEARCH_DIRS_MASK="/var" revdep-rebuild -ipv
> > 
> > doesn't seem to work with the new python version. How can
> > I achieve the same with the python version?
> 
> Can you please check the latest version and tell if it's working as supposed
> to?
> 
> Thanks!

x4 ~ # revdep-rebuild -ipv 
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/revdep-rebuild", line 40, in <module>
    success = rebuild.main(rebuild.parse_options())
  File "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/settings.py", line 117, in parse_options
    settings['library'].update(set(args.library))
TypeError: 'NoneType' object is not iterable
Comment 31 octoploid 2014-03-27 06:54:38 UTC
(In reply to octoploid from comment #30)
> (In reply to Sławek Lis from comment #29)
> > (In reply to octoploid from comment #28)
> > > One last question before I'm out of here:
> > > 
> > >  # env SEARCH_DIRS="/ -*" SEARCH_DIRS_MASK="/var" revdep-rebuild -ipv
> > > 
> > > doesn't seem to work with the new python version. How can
> > > I achieve the same with the python version?
> > 
> > Can you please check the latest version and tell if it's working as supposed
> > to?
> > 
> > Thanks!
> 
> x4 ~ # revdep-rebuild -ipv 
> Traceback (most recent call last):
>   File "/usr/lib/python-exec/python2.7/revdep-rebuild", line 40, in <module>
>     success = rebuild.main(rebuild.parse_options())
>   File
> "/usr/lib64/python2.7/site-packages/gentoolkit/revdep_rebuild/settings.py",
> line 117, in parse_options
>     settings['library'].update(set(args.library))
> TypeError: 'NoneType' object is not iterable

If I comment out that line, it is working fine:

x4 ~ # env SEARCH_DIRS="/ -*" SEARCH_DIRS_MASK="/var" revdep-rebuild -ipv
 * This is a prerelease version, so it may not work correctly
 * The original revdep-rebuild script is installed as revdep-rebuild.sh
 * Please files bugs at: https://bugs.gentoo.org/
 * Collecting system binaries and libraries
 * Collecting dynamic linking informations
 * Scanning files
 * Checking dynamic linking consistency
 * Broken files that requires: libssl.so.6 (64 bits)
        * /opt/modo/extra/PythonModules/DLL/_hashlib.so
        * /opt/modo/extra/PythonModules/DLL/_ssl.so
 * Broken files that requires: libcrypto.so.6 (64 bits)
        * /opt/modo/extra/PythonModules/DLL/_hashlib.so
        * /opt/modo/extra/PythonModules/DLL/_ssl.so
 * Broken files that requires: libperl.so (64 bits)
        * /usr/src/linux/tools/perf/config/feature-checks/test-all.bin
 * Broken files that requires: libfbembed.so.2.5 (64 bits)
        * /usr/local/lib64/libreoffice/program/libfirebird_sdbclo.so
 * Broken files that requires: libclang.so (64 bits)
        * /usr/local/bin/c-index-test
         * /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.la is broken (requires: -l/usr/lib/libtcmalloc.so.4)
 * Assign files to packages

 !!! Broken orphaned files: No installed package was found for the following:
        * /usr/local/bin/c-index-test
        * /usr/local/lib64/libreoffice/program/libfirebird_sdbclo.so
        * /opt/modo/extra/PythonModules/DLL/_ssl.so
        * /usr/src/linux/tools/perf/config/feature-checks/test-all.bin
        * /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.la
        * /opt/modo/extra/PythonModules/DLL/_hashlib.so

There is nothing to emerge. Exiting.

Thank you.
Comment 32 Sławek Lis (RETIRED) gentoo-dev 2014-03-27 07:00:58 UTC
I've fixed that. Thank you.
Comment 33 Oleh 2014-12-09 00:15:05 UTC
(In reply to Sławek Lis from comment #32)
> I've fixed that. Thank you.

and were is the fix in 0.3.0.9-r2 or 9999?
Comment 34 Oleh 2015-01-07 15:48:43 UTC
I'm sorry, a friendly reminder.
Comment 35 Sławek Lis (RETIRED) gentoo-dev 2015-01-07 20:16:53 UTC
(In reply to Oleg from comment #33)
> (In reply to Sławek Lis from comment #32)
> > I've fixed that. Thank you.
> 
> and were is the fix in 0.3.0.9-r2 or 9999?

You can always check lastest (git) version by emerging -9999 (that's live ebuild, that means - its fetched directly from git repo)
Comment 36 Oleh 2015-01-26 13:07:34 UTC
sorry, i understand what 9999 ebuild is and what is git. From comment #32 it is not clear was the fix in regular release or only git. Life is too short to make diffs when it's easily to get yes/no answer.
Comment 37 Brian Dolbec (RETIRED) gentoo-dev 2015-01-30 18:16:45 UTC
there are 3 revdep-rebuild patches in -0.3.0.9-r2 and 2 equery pathces.

PATCHES=(
	"${FILESDIR}"/${PV}-revdep-rebuild-py-504654-1.patch
	"${FILESDIR}"/${PV}-revdep-rebuild-py-504654-2.patch
	"${FILESDIR}"/${PV}-equery-508114.patch
	"${FILESDIR}"/${PV}-equery-strip-XXXFLAGS.patch
	"${FILESDIR}"/${PV}-revdep-rebuild-526400.patch


There are even more patches in 9999 for revdep-rebuild, that have not been released yet.  And it looks like these bug fixes are in the -0.3.0.9-r2 ebuild.
Comment 38 Brian Dolbec (RETIRED) gentoo-dev 2022-07-10 01:38:31 UTC
Marking as resolved, Fixed since this is so old.

Re-open if you find issues still.  New release again in the next few days.