Recently, all emerge merges and depcleans produce the typical output, but near the removal/installation step, a series of lines saying "$: bad substitution" pop up. The packages in question work as expected, but this output concerns me. Pouring over `emerge --debug` output was fruitless. /etc/profile is untouched, make.conf seems sane (to me and others in IRC), and root's bashrc seems sane as well. I'm not sure what else could be affecting this. Web searches point to bash, but I experience this problem only when using emerge. Reproducible: Always Steps to Reproduce: 1. emerge any package 2. Watch the output Actual Results: The package installs correctly, but a series of "$: bad substitution" lines follow. This applies for every single package I've merged or depcleaned for the past 4 or more days. Expected Results: The "bad substitution" lines shouldn't have shown up. Portage 2.2.18 (python 2.7.9-final-0, default/linux/amd64/13.0/no-emul-linux-x86, gcc-4.9.2, glibc-2.20-r2, 3.19.1-gentoo x86_64) ================================================================= System uname: Linux-3.19.1-gentoo-x86_64-AMD_Phenom-tm-_II_X6_1090T_Processor-with-gentoo-2.2 KiB Mem: 16432928 total, 12259496 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Tue, 10 Mar 2015 07:00:01 +0000 sh bash 4.3_p33-r2 ld GNU ld (Gentoo 2.25 p1.0) 2.25 app-shells/bash: 4.3_p33-r2::gentoo dev-java/java-config: 2.2.0::gentoo dev-lang/perl: 5.20.2::gentoo dev-lang/python: 2.7.9-r2::gentoo, 3.3.5-r1::gentoo dev-util/cmake: 3.1.0::gentoo dev-util/pkgconfig: 0.28-r2::gentoo sys-apps/baselayout: 2.2::gentoo sys-apps/openrc: 0.13.11::gentoo sys-apps/sandbox: 2.6-r1::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69::gentoo sys-devel/automake: 1.11.6-r1::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo sys-devel/binutils: 2.25::gentoo sys-devel/gcc: 4.9.2::gentoo sys-devel/gcc-config: 1.8::gentoo sys-devel/libtool: 2.4.6::gentoo sys-devel/make: 4.1-r1::gentoo sys-kernel/linux-headers: 3.19::gentoo (virtual/os-headers) sys-libs/glibc: 2.20-r2::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sporkbox location: /home/daniel/projects/g-overlay masters: gentoo gamerlay location: /var/lib/layman/gamerlay masters: gentoo priority: 50 steam-overlay location: /var/lib/layman/steam-overlay masters: gentoo priority: 50 sunrise location: /var/lib/layman/sunrise masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--ask-enter-invalid" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://ftp.wallawalla.edu/pub/mirrors/ftp.gentoo.org http://gentoo.osuosl.org/ http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://mirror.usu.edu/mirrors/gentoo/ http://www.gtlib.gatech.edu/pub/gentoo" INSTALL_MASK="/usr/lib/systemd/system /etc/systemd" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j7" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" USE="X acl alsa amd64 berkdb bzip2 cairo cli cracklib crypt cups cxx dbus dri fam ffmpeg flac fortran ftp g3dvl gdbm gpm gtk iconv imlib ipv6 jpeg lame lua mmx modules multilib ncurses netifrc nls nptl nvidia ogg opengl openmp openrc pam pcre png python qt4 readline samba sdl session sqlite sse sse2 ssl svg tcpd theora truetype udev unicode vdpau vorbis webm xft xinerama xpm xvmc zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev joystick wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US en_GB" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" SANE_BACKENDS="hp hp4200" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Created attachment 398576 [details] emerge --debug output I've attached `emerge --debug` output for others to inspect. The package is from my personal overlay (a simple Python utility). Package doesn't matter since it happens every time, but its source can be viewed at https://github.com/sporkbox/sporkbox-overlay/app-misc/dupekill/dupekill-9999.ebuild .
Created attachment 398578 [details] root's .bashrc Attached root's .bashrc
Created attachment 398580 [details] make.conf
(In reply to Daniel Campbell from comment #1) > Created attachment 398576 [details] > emerge --debug output > > I've attached `emerge --debug` output for others to inspect. The package is > from my personal overlay (a simple Python utility). Package doesn't matter > since it happens every time, but its source can be viewed at > https://github.com/sporkbox/sporkbox-overlay/app-misc/dupekill/dupekill-9999. > ebuild . Sorry, that's https://raw.githubusercontent.com/sporkbox/sporkbox-overlay/master/app-misc/dupekill/dupekill-9999.ebuild
I am having the same problems.
I just bisected portage and found the bad commit: $ git bisect log git bisect start # bad: [f44548b24df12f324b5560355e4e6947a9be5f3b] setup.py: Set version for new release git bisect bad f44548b24df12f324b5560355e4e6947a9be5f3b # good: [fc0567d794a4bf122baa3884f15c497d9d863734] setup.py: version bump, make sync modules selectable git bisect good fc0567d794a4bf122baa3884f15c497d9d863734 # bad: [c1489985f64443c4fba0b9661eee60f61e470d37] repoman: skip vcs calls for manifest modes (bug 540882) git bisect bad c1489985f64443c4fba0b9661eee60f61e470d37 # bad: [f1c1b8a77eebf7713b32e5f9945690f60f4f46de] Generate soname dependency metadata (bug 282639) git bisect bad f1c1b8a77eebf7713b32e5f9945690f60f4f46de # good: [06a4e7a09211c4a598a72a97c64daff13cd502ff] setup.py: Revert the select_plugins() changes git bisect good 06a4e7a09211c4a598a72a97c64daff13cd502ff # good: [7fab3aadb4cdca35ce0d81525af1256c745308ff] Add another check for broken /dev/s (bug 538980) git bisect good 7fab3aadb4cdca35ce0d81525af1256c745308ff # first bad commit: [f1c1b8a77eebf7713b32e5f9945690f60f4f46de] Generate soname dependency metadata (bug 282639)
(In reply to Karol Herbst from comment #6) > # first bad commit: [f1c1b8a77eebf7713b32e5f9945690f60f4f46de] Generate > soname dependency metadata (bug 282639) The bash changes in this commit are trivial, so I don't understand why they would trigger the "$: bad substitution" messages: http://gitweb.gentoo.org/proj/portage.git/commit/?id=f1c1b8a77eebf7713b32e5f9945690f60f4f46de I cannot reproduce the problem with bash-4.2_p53. Maybe it's a bug in bash-4.3_p33-r2?
Comment on attachment 398578 [details] root's .bashrc portage doesn't load the user's ~/.bashrc files, so this file shouldn't matter
the trace doesn't seem to be that enlightening ... the errors look like they come up *after* misc-functions.sh finishes fully executing i would try downgrading to some other bash versions in the tree as Zac suggested.
I tried downgrading to 4.3_p33-r1, 4.3_p33, and 4.2_p53, with restarting terminal and verifying that `bash --version` checks out, env-update, etc. and re-merging any package. The same "bad substitution" message appeared on each. I'm pretty much lost. I have nothing in p.use for bash so I'm using whatever USE flags it ships with.. same goes for portage.
That bisect seems wrong. Are you sure you don't have any malformed files in /etc/portage?
(In reply to Michał Górny from comment #11) > That bisect seems wrong. Are you sure you don't have any malformed files in > /etc/portage? I've pored over each file I have in /etc/portage and can't find anything out of place or malformed. I didn't know about repo.postsync.d and it's /usr/bin/q invocation, but given that I've not touched it I doubt it's a problem. It looks like it checks for PORTAGE_QUIET and if it's set, quietly forks. I can attach a file that has the content of all my /etc/portage files if you're interested in looking over it.
this also happens for me after whiping out /etc/portage
Just to add critical mass to the bug: I confirm that bug too. Although, I catched it only on my laptop, but do not on any of hundred servers. Although, /etc/portage is almost identical between all of them.
I'm trying to fix the title so that is turns up in searches for "bad substition", since I had a hard time finding this bug to mark bug 547154 as a duplicate.
*** Bug 547154 has been marked as a duplicate of this bug. ***
I can observe strange situation: - bug occurs on my desktop systems - bug does not occur on my server systems What differs both: - I have Polish locales on desktop sysmtems - I have systemd on desktop systems - I have openrc on server systems - I have nvidia GPU on desktop systems (yup, it was the cause of one of mysterious bugs in the past!) - I have ATI graphics on server systems I also checked that Python 2.7 is currently set as default python interpreter on both. I'm keeping all my Gentoo boxes up to date, so they share similar versions of everything.
Well, I have an nvidia card, using the nouveau driver, en_US.utf8 and no Errors
(In reply to Paul Osmialowski from comment #17) > What differs both: > - I have Polish locales on desktop sysmtems > - I have systemd on desktop systems > - I have openrc on server systems > - I have nvidia GPU on desktop systems (yup, it was the cause of one of > mysterious bugs in the past!) > - I have ATI graphics on server systems > > I also checked that Python 2.7 is currently set as default python > interpreter on both. I use openrc, radeon video driver (probably should not matter), Python 3.3 as system default and ru_RU.UTF8 locale and see this issue.
I forgot about I guess the most important difference: - I have these server systems for long long years and they still use baselayout1 (make.conf is in /etc) - On desktop systems which I change more frequently, I have baselayout2 (make.conf is in /etc/portage)
See https://forums.gentoo.org/viewtopic-t-1014842.html tl;dr: Broken libs have "$$ORIGIN" in runpaths which issues a warning/error since the commit (added varexpand).
There's a patch in the following branch: https://github.com/zmedico/portage/tree/bug_542796 I've posted it for review here: https://archives.gentoo.org/gentoo-portage-dev/message/b0ed5c8fd00f88ba4813777d557ac812
I've placed your patch as /etc/portage/patches/sys-apps/portage/2.2.18-bad_substitution.patch and now emerging packages ends like this: >>> Installing (1 of 1) media-video/gxine-0.5.907-r1::gentoo /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution * Updating desktop mime database ... * Updating shared mime info database ... * Updating icons cache ... [ ok ] * Updating desktop mime database ... * Updating shared mime info database ... * Updating icons cache ... [ ok ] /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution >>> Auto-cleaning packages...
(In reply to Paul Osmialowski from comment #23) > I've placed your patch as > /etc/portage/patches/sys-apps/portage/2.2.18-bad_substitution.patch and now > emerging packages ends like this: > > >>> Installing (1 of 1) media-video/gxine-0.5.907-r1::gentoo > /var/db/pkg/dev-db/tora-3.0.0_pre20140929/NEEDED.ELF.2: $: bad substitution Please file a new bug for dev-db/tora-3.0.0_pre20140929, and attach that file. In order silence the error message by manually edit the file, and then update the parent directory timestamps to invalidate portage's cache: touch /var/db/pkg/dev-db/tora-3.0.0_pre20140929 \ /var/db/pkg/dev-db \ /var/db/pkg
Added bug 547520
This is in the master branch: https://gitweb.gentoo.org/proj/portage.git/commit/?id=a47407e864d9a5a43d6f05ef06b46e34ff0e0c24
Hi, I also have the problem and have found something. In /var/db/pkg/dev-db/tora-9999/NEEDED.ELF.2 is: X86_64;/usr/bin/tora;;$ORIGIN/:$$ORIGIN/;libQtGui.so.4,libQtSql.so.4,libQtNetwork.so.4,libQtCore.so.4,libqscintilla2.so.11,libferrisloki.s o.3,libdl.so.2,libpq.so.5,libstdc++.so.6,libm.so.6,libgcc_s.so.1,libc.so.6 X86_64;/usr/lib64/libtrotl.so;libtrotl.so;$ORIGIN/instantclient/:$$ORIGIN/instantclient/;libclntsh.so.11.1,libstdc++.so.6,libgcc_s.so.1,li bc.so.6 X86_64;/usr/lib64/tora-3.0.0alpha/libporacle.so;libporacle.so;$ORIGIN/:$$ORIGIN/;libclntsh.so.11.1,libQtGui.so.4,libQtCore.so.4,libtrotl.s o,libstdc++.so.6,libgcc_s.so.1,libc.so.6 Look at the "$$ORIGIN". Portage since 2.2.18 looks at this and shows "$: bad substitution" error/warning, nevermind what it merges. After unmerge tora-9999 strange waring disapeared. Merging tora-9999 again or tora-3.0.0_pre20140929 - the same situation Looks like portage has a problem with parsing "$$ORIGIN". In my opinion there should be "$ORIGIN"
(In reply to Jacek from comment #27) > After unmerge tora-9999 strange waring disapeared. > Merging tora-9999 again or tora-3.0.0_pre20140929 - the same situation > Looks like portage has a problem with parsing "$$ORIGIN". In my opinion > there should be "$ORIGIN" Something about tora's build system produces the invalid $$ORIGIN thing. See bug 547520.
yes, you right. toras code sets that paths. SET(CMAKE_INSTALL_RPATH "$ORIGIN/:$$ORIGIN/") SET(CMAKE_INSTALL_RPATH "$ORIGIN/instantclient/:$$ORIGIN/instantclient/") tnx j
These seem to be workarounds. Some buildsystems mess(ed) up the $ORIGIN... http://www.cmake.org/pipermail/cmake/2008-January/019290.html
(In reply to Markus from comment #30) > These seem to be workarounds. > Some buildsystems mess(ed) up the $ORIGIN... > http://www.cmake.org/pipermail/cmake/2008-January/019290.html Apparently it's a workaround for a buggy old version of cmake. There are at least two possible ways to fix these packages: 1) Use sed to remove :$$ORIGIN from the source files in src_prepare. 2) Use patchelf to remove :$$ORIGIN from DT_RUNPATH of the binaries in src_install.
Update of tora to version dev-db/tora-3.0.0_pre20140929-r1 solves the problem.
from what i can tell, there's nothing here related to the shell, so dropping base-system from the list
(In reply to Zac Medico from comment #31) > (In reply to Markus from comment #30) > > These seem to be workarounds. > > Some buildsystems mess(ed) up the $ORIGIN... > > http://www.cmake.org/pipermail/cmake/2008-January/019290.html > > Apparently it's a workaround for a buggy old version of cmake. There are at > least two possible ways to fix these packages: > > 1) Use sed to remove :$$ORIGIN from the source files in src_prepare. > 2) Use patchelf to remove :$$ORIGIN from DT_RUNPATH of the binaries in > src_install. Although I think 1 is preferred, in my case that was adobe-air-sdk-bin, which has no "source" :-/
After applying the patch (which is now in the master branch), the culprit was dev-util/adobe-air-runtime-2.6: /var/db/pkg/dev-util/adobe-air-runtime-2.6/NEEDED.ELF.2: $: bad substitution Editing the file with vim (:%s/$$ORIGIN/$ORIGIN/g) and touching the directories fixed the issue for me.
yeah, same for me
(In reply to Daniel Campbell from comment #35) > After applying the patch (which is now in the master branch), the culprit > was dev-util/adobe-air-runtime-2.6: > > /var/db/pkg/dev-util/adobe-air-runtime-2.6/NEEDED.ELF.2: $: bad substitution > > Editing the file with vim (:%s/$$ORIGIN/$ORIGIN/g) and touching the > directories fixed the issue for me. Actually, I've edited it with vim too, but I think we need to think about patching policy and fix them all then ;)
(In reply to Vadim A. Misbakh-Soloviov (mva) from comment #37) > (In reply to Daniel Campbell from comment #35) > > :snip: > > Actually, I've edited it with vim too, but I think we need to think about > patching policy and fix them all then ;) I could cook up a quick script tonight, but I don't think it'd be the preferred way to fix the DB. Fixing it at the ebuild level is probably the most elegant right now.
(In reply to Vadim A. Misbakh-Soloviov (mva) from comment #34) > Although I think 1 is preferred, in my case that was adobe-air-sdk-bin, > which has no "source" :-/ It should be possible to use a combination of 'patchelf --print-rpath' and 'patchelf --set-rpath' to fix the binaries. Alternatively, maybe 'patchelf --shrink-rpath' will do the trick.
(In reply to Zac Medico from comment #39) > It should be possible to use a combination of 'patchelf --print-rpath' and > 'patchelf --set-rpath' to fix the binaries. Alternatively, maybe 'patchelf > --shrink-rpath' will do the trick. I suggest something like this: find "${ED}" -type f -print0 | while read -r -d '' x; do rpath=$(patchelf --print-rpath "${x}" 2>/dev/null) [[ ${rpath} =~ (.*:)?\$\$ORIGIN[^:]*(:.*)? ]] || continue patchelf --set-rpath "${BASH_REMATCH[1]}${BASH_REMATCH[2]#:}" "${x}" \ || die done
(In reply to Zac Medico from comment #40) > (In reply to Zac Medico from comment #39) > > It should be possible to use a combination of 'patchelf --print-rpath' and > > 'patchelf --set-rpath' to fix the binaries. Alternatively, maybe 'patchelf > > --shrink-rpath' will do the trick. > > I suggest something like this: > > find "${ED}" -type f -print0 | while read -r -d '' x; do > rpath=$(patchelf --print-rpath "${x}" 2>/dev/null) > [[ ${rpath} =~ (.*:)?\$\$ORIGIN[^:]*(:.*)? ]] || continue > patchelf --set-rpath "${BASH_REMATCH[1]}${BASH_REMATCH[2]#:}" "${x}" \ > || die > done while read -r -d '' x; do ... done < <(find "${ED}" -type f -print0) Otherwise the loop will be run in subprocess instead of the main process.
Possibly stupid question: During last update I've seen many similiar warnings. It's necessary to open a bug for each issued package or it will be better to report it upstream?
(In reply to Sergey S. Starikoff from comment #42) > Possibly stupid question: > During last update I've seen many similiar warnings. > It's necessary to open a bug for each issued package or it will be better to > report it upstream? It's probably best to file a gentoo bug, so that the problem can be corrected ASAP. Who knows if upstream still wants to support broken old versions of cmake?
(In reply to Zac Medico from comment #43) > Who knows if upstream still wants to support broken old > versions of cmake? Not only cmake. Issue seems to be more common. Working on ebuild for =app-editors/gobby-0.5.0 (see bug #545394) I've seen the same messages. Although it uses cmake only for docs (and I build it with default -doc). Fix from bug #547520 failes because no issued pattern found. Continue testing showed the same issue for =gnome-doc-utils/gnome-doc-utils-0.20.10-r1 (which seems not to use cmake at all). I expect not hundreds, but at least thousands of such packages only in the gentoo overlay. So, opening a bug for each issued package don't looks to be a good idea. Probably we should try to provide some generic check at eclass[es] level.
(In reply to Sergey S. Starikoff from comment #44) > I expect not hundreds, but at least thousands of such packages only in the > gentoo overlay. > So, opening a bug for each issued package don't looks to be a good idea. > Probably we should try to provide some generic check at eclass[es] level. I guess we'll have to put the check in portage, since it calls scanelf on all of the files after they are installed into ${D}. We can make it check for invalid substitutions at that time, and trigger a QA notice.
Released in portage-2.2.19
