I believe that this might be an issue with libunwind failing to properly be detected at build time but can't say for sure. Building perf from the linux kernel source tree on my system works just fine, but when trying to annotate a function from the portage installed perf, I get the following error: Couldn't annotate tcmalloc::PageHeap::SearchFreeAndLargeLists: Internal error: Invalid -1 error code Symbols either in the kernel or exported directly from binaries and not shared objects seem to annotate just fine, however. Reproducible: Always Steps to Reproduce: 1. perf top 2. pick a function exported from a DSO 3. get the failure message Actual Results: Get the error. Expected Results: Get the function in annotated assembly. Portage 3.0.66.1 (python 3.13.0-final-0, default/linux/amd64/23.0/split-usr/desktop, gcc-14, glibc-2.40-r5, 6.11.6-gentoo-x86_64 x86_64) ================================================================= System uname: Linux-6.11.6-gentoo-x86_64-x86_64-Intel-R-_Core-TM-_i9-10940X_CPU_@_3.30GHz-with-glibc2.40 KiB Mem: 65527448 total, 55211948 free KiB Swap: 33554428 total, 33554428 free Timestamp of repository gentoo: Sun, 03 Nov 2024 22:00:00 +0000 Head commit of repository gentoo: 516bfe3aa30a5fe5afa2faee7bf2997a3b309d77 Timestamp of repository guru: Sun, 10 Nov 2024 15:48:25 +0000 Head commit of repository guru: 763a603d515c411793a9ed0ed533af145b0aa077 Timestamp of repository haskell: Sun, 10 Nov 2024 21:48:25 +0000 Head commit of repository haskell: 93905262b83313bb37ee4dfd33f5d6cdb5b3a331 Timestamp of repository mysql: Fri, 12 Feb 2021 00:22:38 +0000 Head commit of repository mysql: c89884060834e87f994587c98d7c8f2bfdaf4da2 sh bash 5.2_p37 ld GNU ld (Gentoo 2.43 p2) 2.43.1 distcc 3.4 x86_64-pc-linux-gnu [disabled] ccache version 4.10.2 [enabled] app-misc/pax-utils: 1.3.8::gentoo app-shells/bash: 5.2_p37::gentoo dev-build/autoconf: 2.72-r1::gentoo dev-build/automake: 1.16.5-r2::gentoo, 1.17-r1::gentoo dev-build/cmake: 3.30.5::gentoo dev-build/libtool: 2.5.3::gentoo dev-build/make: 4.4.1-r100::gentoo dev-build/meson: 1.6.0::gentoo dev-java/java-config: 2.3.4::gentoo dev-lang/perl: 5.40.0::gentoo dev-lang/python: 3.11.10_p1::gentoo, 3.12.7_p1::gentoo, 3.13.0::gentoo dev-lang/rust: 1.82.0::gentoo dev-util/ccache: 4.10.2-r1::gentoo sys-apps/baselayout: 2.15::gentoo sys-apps/openrc: 0.55.1::gentoo sys-apps/sandbox: 2.39::gentoo sys-devel/binutils: 2.42-r2::gentoo, 2.43-r1::gentoo sys-devel/binutils-config: 5.5.2::gentoo sys-devel/clang: 17.0.6::gentoo, 18.1.8-r6::gentoo, 19.1.3::gentoo sys-devel/gcc: 12.3.1_p20240209::gentoo, 13.3.1_p20241025::gentoo, 14.2.1_p20241026::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/lld: 17.0.6::gentoo, 18.1.8::gentoo, 19.1.3::gentoo sys-devel/llvm: 17.0.6-r2::gentoo, 18.1.8-r6::gentoo, 19.1.3::gentoo sys-kernel/linux-headers: 6.11::gentoo (virtual/os-headers) sys-libs/glibc: 2.40-r5::gentoo Repositories: gentoo location: /scratchdir/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 volatile: True sync-rsync-extra-opts: sync-rsync-verify-max-age: 3 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 LocalOverlay location: /usr/local/portage/overlay masters: gentoo volatile: True guru location: /var/db/repos/guru sync-type: git sync-uri: https://github.com/gentoo-mirror/guru.git masters: gentoo volatile: False haskell location: /var/db/repos/haskell sync-type: git sync-uri: https://github.com/gentoo-mirror/haskell.git masters: gentoo volatile: False mysql location: /var/db/repos/mysql sync-type: git sync-uri: https://github.com/gentoo-mirror/mysql.git masters: gentoo volatile: False crossdev location: /var/db/repos/crossdev masters: gentoo priority: 10 volatile: False ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O3 -pipe -march=cascadelake -mtune=cascadelake -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.9/conf /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.3/ext-active/ /etc/php/cgi-php8.3/ext-active/ /etc/php/cli-php8.3/ext-active/ /etc/php/fpm-php8.3/ext-active/ /etc/php/phpdbg-php8.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O3 -pipe -march=cascadelake -mtune=cascadelake -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --complete-graph" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live ccache config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://mirror.leaseweb.com/gentoo/ http://gentoo.osuosl.org/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs" LEX="flex" LINGUAS="en_US" MAKEOPTS="-j14" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" RUSTFLAGS="-C target-cpu=native" SHELL="/bin/bash" USE="X a52 a52codec aac aalib acl acpi alsa amd64 amd64codecs apache2 asm avi avx bash-completion bluetooth branding bzip2 cairo cd-r cdda cddb cdparanoia cdr cpudetection crypt cscope cups curl custom-cflags custom-cpuopts custom-cxxflags dav1d dbus dirac djbfft djvu dri dts dv dvb dvd dvd-r dvdnav dvdr dvdread elogind encode exif f77 ffmpeg flac fortran gdbm gif gpm gs gtk gui hal iconv icu ieee1394 imagemagick imlib io-uring ipv6 jack jbig jemalloc joystick jpeg jpeg2k kf6compat latin1 lcms libcaca libdvdread libg++ libnotify libtirpc libwww lzma lzo mad matroska mikmod mjpeg mmx mmxext mng mp2 mp3 mp4 mpeg mpeg2 mplayer multilib musepack musicbrainz mysql ncurses nls nptl nvidia nvidia% offensive oft ogg openal opencl opengl openmp openntpd openrc oss pam pango pcre pdf pdflib perl php php5 png pnm policykit posix ppds python qml qt qt3support qt5 qt6 quicktime rar readline real ruby samba sasl sdl seccomp smp sound speex spell split-usr srt sse sse2 sse3 sse4 sse4.1 sse4_2 ssl ssse3 startup-notification suexec svg tcpd test-rust theora threads tiff transcode truetype ttf udev udisks unicode unicode3 upower usb v4l v4l2 vcd vdpau vim-syntax vorbis vulkan wayland win32codecs wxwidgets x264 xattr xcb xft xinerama xml xpm xv xvid zip zlib" ABI_X86="64 32" ADA_TARGET="gcc_12" ALSA_CARDS="ca0132" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 avx512f avx512dq avx512cd avx512bw avx512vl f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3" CURL_SSL="openssl" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GUILE_SINGLE_TARGET="3-0" GUILE_TARGETS="3-0" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres16" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11 python3_12 python3_13" RUBY_TARGETS="ruby31 ruby32 ruby33" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
Err oops, no this happens with the perf built from the kernel source tree, too (though, it does properly detect and build with libunwind support). Still, I can't annotate DSOs anymore from a live perf top session.
Another detail, it's annotate in general (for only DSOs). Last time I ran into an issue with perf annotate, it was related to a popen call to objdump. I suspect something similar is happening and this very well could be an upstream bug. It also could be an incompatibility with the objdump in binutils.
strace output at around the time of the perf annotate failure: write(1, "\33[110;1HTip: For tracepoint even"..., 78) = 78 write(1, "\33[1;1H\33[38;5;15m\33[48;5;4mSamples"..., 102) = 102 write(1, "\33[2;1HOverhead Command Shared "..., 504) = 504 rt_sigprocmask(SIG_UNBLOCK, [WINCH], NULL, 8) = 0 rt_sigaction(SIGWINCH, {sa_handler=0x560f96aebb20, sa_mask=[WINCH], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fdcbec409b0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999998651}) read(6, "\33", 1) = 1 pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=20000}, NULL) = 1 (in [0], left {tv_sec=0, tv_nsec=19706}) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999663}) read(6, "[", 1) = 1 pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999705}) read(6, "B", 1) = 1 write(1, "\33[3;1H\33[38;5;0m\33[48;5;11m 35.80"..., 2224) = 2224 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999998724}) read(6, "\33", 1) = 1 pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=20000}, NULL) = 1 (in [0], left {tv_sec=0, tv_nsec=19467}) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999663}) read(6, "[", 1) = 1 pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999715}) read(6, "B", 1) = 1 write(1, "\33[3;1H\33[38;5;1m 35.80%\33[39m ls"..., 811) = 811 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999998401}) read(6, "\33", 1) = 1 pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=20000}, NULL) = 1 (in [0], left {tv_sec=0, tv_nsec=19677}) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999661}) read(6, "[", 1) = 1 pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999701}) read(6, "A", 1) = 1 write(1, "\33[3;1H\33[38;5;0m\33[48;5;11m 35.80"..., 811) = 811 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999999053}) read(6, "\r", 1) = 1 write(1, "\33[1;1H\33[38;5;15m\33[48;5;4m\33[K\33[11"..., 93) = 93 write(1, "\33[2;1H\33[38;5;0m\33[48;5;11mAnnotat"..., 1173) = 1173 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999998896}) read(6, "\r", 1) = 1 uname({sysname="Linux", nodename="eggsbenedict", ...}) = 0 access("/lib64/libc.so.6", R_OK) = 0 write(1, "\33[53;163H\33[1K \342\224\214\342\224\200Error:\342\224\200\342\224\200"..., 503) = 503 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999998944}) read(6, "\r", 1) = 1 write(1, "\33[110;365H", 10) = 10 write(1, "\33[1;1H\33[38;5;15m\33[48;5;4mSamples"..., 1147) = 1147 write(1, "\33[110;364H\342\226\222\10\33[4h \33[4l\33[9;365H", 31) = 31 pselect6(1, [0], NULL, NULL, NULL, NULL) = 1 (in [0]) pselect6(7, [6], NULL, NULL, {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [6], left {tv_sec=9, tv_nsec=999998888})
So interestingly, it's not even getting to the objdump invocation + pipe. It's not getting much further than the uname call to grab the architecture.
Chasing things down in gdb, I see the objdump command argument attempt to be formulated but the objdump binary doesn't get determined because it seems to struggle to figure out what the `dso__binary_type` is: 1503 } else if (dso__binary_type(dso) == DSO_BINARY_TYPE__NOT_FOUND) { (gdb) 1504 return -1;
Wow...it really does seem to be an erroneous -1 return on that line. Simply commenting it out fixes everything.
https://lore.kernel.org/all/ZvOl7NOloeSn50ZZ@google.com/
(In reply to Sam James from comment #7) > https://lore.kernel.org/all/ZvOl7NOloeSn50ZZ@google.com/ Yes, I found that thread. I've emailed the proper ML, hopefully a fix will be inbound soonish (perhaps an extra condition being checked before that return?).
(In reply to Adam Stylinski from comment #8) > (In reply to Sam James from comment #7) > > https://lore.kernel.org/all/ZvOl7NOloeSn50ZZ@google.com/ > > Yes, I found that thread. I've emailed the proper ML, hopefully a fix will > be inbound soonish (perhaps an extra condition being checked before that > return?). Btw only the first patch is still needed (and applies fine), the second is already in 6.11 via https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/tools/perf?id=e988a5b53ebd40c2fafc86250e95d69929796fbd
The first patch will be in 6.12, but without Fixes: and Cc: stable tags :( https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5363c306787c88d41a41493f81b4308643696f6e The title in the bug is too specific, this is a regression for all perf-annotate uses, not just under 'perf top', so it is a bit mind-boggling it was not caught in time.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e7fd7b802eede73629109b18642583e3400d7bea commit e7fd7b802eede73629109b18642583e3400d7bea Author: Guilherme Amadio <amadio@gentoo.org> AuthorDate: 2024-11-11 12:52:08 +0000 Commit: Guilherme Amadio <amadio@gentoo.org> CommitDate: 2024-11-11 16:15:09 +0000 dev-util/perf: add 6.11.7 Closes: https://bugs.gentoo.org/943203 Signed-off-by: Guilherme Amadio <amadio@gentoo.org> dev-util/perf/Manifest | 1 + dev-util/perf/files/perf-6.11.7-annotate.patch | 19 ++ dev-util/perf/perf-6.11.7.ebuild | 346 +++++++++++++++++++++++++ 3 files changed, 366 insertions(+)
I added perf 6.11.7 to the tree, and applied the missing patch to it to fix this problem. Thank you for filing this. Cheers,
