Noticed by Ionen. Not clear yet if the packages are to blame, Portage, or bash 5.2. Needs (currently masked) bash 5.2 to reproduce. [22:04:59] <+ionen> there was the weird thing where it enables extglob if the ebuild has a statement like: echo $(usex test $(usex test $(usex test))) and then you get a QA notice about changed shopt [22:05:25] <+ionen> not that it cause real problems [22:05:53] <+ionen> I could never reproduce outside ebuilds, I assume portage's usex is related
Created attachment 815095 [details] build.log of app-i18n/uim-1.8.9 (up to src_configure) Portage 3.0.37 (python 3.11.0-candidate-2, default/linux/amd64/17.1/hardened, gcc-13.0.0, glibc-2.36-r4, 5.15.71-gentoo-dist-hardened x86_64) ================================================================= System uname: Linux-5.15.71-gentoo-dist-hardened-x86_64-AMD_Ryzen_9_3950X_16-Core_Processor-with-glibc2.36 KiB Mem: 65765024 total, 15625784 free KiB Swap: 8290300 total, 8226044 free Timestamp of repository gentoo: Sat, 01 Oct 2022 21:46:47 +0000 Head commit of repository gentoo: 172fc770d099e4815aab72b3aad9c3d0ebf5f9e9 Timestamp of repository emacs: Wed, 24 Aug 2022 05:02:36 +0000 Head commit of repository emacs: cfdb9df2b4b45d064912481b0fa29cd2c6965703 Timestamp of repository kde: Sun, 25 Sep 2022 19:02:33 +0000 Head commit of repository kde: bdff9eefb9c4db131551b459a7c0ef3cad86a03e Timestamp of repository qt: Thu, 29 Sep 2022 12:46:44 +0000 Head commit of repository qt: 28970ef1db83c3cebe032a13ecf7f9551218b5f1 Timestamp of repository sam_c: Thu, 22 Sep 2022 03:01:46 +0000 Head commit of repository sam_c: d13161c447f2cf16381c26fe6a89e10bb0a84094 Timestamp of repository steam-overlay: Wed, 07 Sep 2022 08:26:00 +0000 Head commit of repository steam-overlay: 78208ca83614c4cc031ca8f8ec9b023a5c9c3b1e sh dash 0.5.11.5 ld GNU ld (Gentoo 2.39 p4) 2.39.0 ccache version 4.6.3 [disabled] app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.2::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.36.0::gentoo dev-lang/python: 2.7.18_p15-r1::gentoo, 3.8.14::gentoo, 3.9.14::gentoo, 3.10.7::gentoo, 3.11.0_rc2::gentoo dev-lang/rust: 1.64.0-r1::gentoo dev-util/ccache: 4.6.3::gentoo dev-util/cmake: 3.24.2::gentoo dev-util/meson: 0.63.2-r1::gentoo sys-apps/baselayout: 2.9::gentoo sys-apps/sandbox: 2.29::gentoo sys-apps/systemd: 251.4::gentoo sys-devel/autoconf: 2.13-r3::gentoo, 2.69-r7::gentoo, 2.71-r4::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.39-r2::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/clang: 14.0.6-r1::gentoo, 15.0.1::gentoo, 16.0.0_pre20220930::gentoo sys-devel/gcc: 9.5.0::gentoo, 10.4.1_p20220922::gentoo, 11.3.1_p20220923::gentoo, 12.2.1_p20220924::gentoo, 13.0.0_pre20220918::gentoo sys-devel/gcc-config: 2.6::gentoo sys-devel/libtool: 2.4.7::gentoo sys-devel/lld: 14.0.6::gentoo sys-devel/llvm: 14.0.6-r2::gentoo, 15.0.1::gentoo, 16.0.0_pre20220930::gentoo sys-devel/make: 9999::gentoo sys-kernel/linux-headers: 5.19::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r4::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo.git priority: -1000 eclass-overrides: sam_c sync-git-clone-extra-opts: -b stable -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now sync-git-verify-commit-signature: yes crossdev location: /var/db/repos/crossdev masters: gentoo eclass-overrides: sam_c emacs location: /var/db/repos/emacs sync-type: git sync-uri: https://github.com/gentoo-mirror/emacs.git masters: gentoo eclass-overrides: sam_c kde location: /var/db/repos/kde sync-type: git sync-uri: https://github.com/gentoo-mirror/kde.git masters: gentoo eclass-overrides: sam_c local location: /var/db/repos/local masters: gentoo eclass-overrides: sam_c qt location: /var/db/repos/qt sync-type: git sync-uri: https://github.com/gentoo-mirror/qt.git masters: gentoo eclass-overrides: sam_c sam_c location: /var/db/repos/sam_c sync-type: git sync-uri: https://github.com/gentoo-mirror/sam_c.git masters: gentoo eclass-overrides: sam_c steam-overlay location: /var/db/repos/steam-overlay sync-type: git sync-uri: https://github.com/gentoo-mirror/steam-overlay.git masters: gentoo eclass-overrides: sam_c test location: /var/db/repos/test masters: gentoo eclass-overrides: sam_c ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches -D_GLIBCXX_ASSERTIONS" DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS="--keep-going --with-bdeps=y --complete-graph --deep --dynamic-deps=n --changed-deps=n --usepkg-exclude sys-fs/zfs --usepkg-exclude sys-fs/zfs-kmod --usepkg=y --implicit-system-deps=n --jobs=4 --load-average 32" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg cgroup clean-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox mount-sandbox multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms split-log strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches" GENTOO_MIRRORS="http://mirror.bytemark.co.uk/gentoo/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/ http://mirrors.soeasyto.com/distfiles.gentoo.org/" LANG="en_GB.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -Wl,-z,pack-relative-relocs" LINGUAS="en en_GB" PKGDIR="/var/cache/binpkgs" PORTAGE_COMPRESS="zstd" PORTAGE_COMPRESS_FLAGS="-9 --rm -q" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" RUSTFLAGS="-C target-cpu=native" SHELL="/bin/bash" USE="PIC X a52 aac acl acpi activities aes alsa amd64 avx avx2 bash-completion bluetooth branding bzip2 cairo caps cdda cdr clang cli crypt dbus declarative dist-kernel dri dts dvd dvdr emacs encode exif f16c filecaps firewalld flac fma3 fortran freetype gdbm gif gmp gpm graphite gtk gui hardened harfbuzz hunspell iconv icu ipv6 jit jpeg kde kdesu kwallet lcms libglvnd libnotify libtirpc llvm-libunwind lto mad mmx mmxext mng mp3 mp4 mpeg multilib ncurses nftables nls nptl ogg opengl openmp pam pango pclmul pcre pdf pgo pie pipewire plasma png policykit popcnt ppds pulseaudio qml qt5 rdrand readline screencast sdl seccomp semantic-desktop sha spell sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssp ssse3 startup-notification svg system-av1 system-binutils system-boost system-bootstrap system-cairo system-clang system-digest system-ffmpeg system-harfbuzz system-heimdal system-icu system-jpeg system-leveldb system-libevent system-libs system-libvpx system-libyaml system-llvm system-lz4 system-mitkrb5 system-sqlite system-ssl system-tbb system-uulib system-webp system-zlib systemd tiff truetype udev udisks unicode upower usb verify-sig vorbis vulkan wayland widgets x264 xattr xcb xml xtpax xv xvid zfs zlib zsh-completion" ABI_X86="32 64" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="AArch64 AMDGPU ARM AVR BPF Hexagon Lanai MSP430 Mips NVPTX PowerPC RISCV Sparc SystemZ WebAssembly X86 XCore" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10 pypy3 python3_11 python3_8 python3_9" QEMU_USER_TARGETS="aarch64 aarch64_be alpha armeb hppa m68k ppc64le sparc64" RUBY_TARGETS="ruby31" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, MAKEOPTS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
Hi Sam. Unsurprisingly, it is a bug in bash. I am attaching a reproducer. With BASH_COMPAT=50 in effect, it takes 3 levels of comsub nesting for extglob to be unexpectedly enabled in bash-5.2 at all levels, including that of the initial shell, despite never having requested that it be enabled. Further, test #3 demonstrates - as had already been reported for 5.2-rc3 - that BASHOPTS no longer reliably indicates whether extglob is enabled (shopt is being truthful). It was implied by upstream that the issues with 5.2-rc3 could be addressed with "an easy fix", which struck me as being rather blasé at the time. I had the distinct feeling that this wouldn't be the end of it.
Created attachment 815119 [details] gentoo-bug-873931-reproducer.bash
Created attachment 815125 [details] gentoo-bug-873931-reproducer-r1.bash Same as the initial attachment, only rectifying a typo for an informational message.
Created attachment 822688 [details, diff] bash-5.2-compat-extglob-comsub.patch This is the patch posted by Chet on the bug-bash list, regenerated so as to be a unified diff and to apply to the 5.2 release without fuzz. I tested it on top of 5.2_p2 and could find no fault with it.
I spoke too soon. While the patch is probably sufficient for this particular bug to be closed, the >=5.2-rc3 parser remains buggy. I have posted a response that goes into further detail: https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00029.html.
Created attachment 822832 [details, diff] bash-5.2-fixes-for-extglob-in-compat-mode.patch This is a backport of commit 22f21b760ed90eb77c3756e6ccf39b73c84f532a from the devel branch. It is intended to be applied to 5.2.2, with -p0. It addresses the issue discovered by Ionen, along with the issue mentioned by comment #6.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=47950445cddff736a1e6c0c1346a20ab6b326cc1 commit 47950445cddff736a1e6c0c1346a20ab6b326cc1 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-10-08 13:24:32 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-10-08 13:25:11 +0000 app-shells/bash: backport fixes for extglob in compat mode for 5.2 Bug: https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00029.html Closes: https://bugs.gentoo.org/873931 Thanks-to: Kerin Millar <kfm@plushkava.net> Signed-off-by: Sam James <sam@gentoo.org> app-shells/bash/bash-5.2_p2-r1.ebuild | 345 +++++++++++++++++++++ ...h-5.2_p2-fixes-for-extglob-in-compat-mode.patch | 320 +++++++++++++++++++ 2 files changed, 665 insertions(+)
Thanks. I should add that I inadvertently included an unrelated fix, though it's a valid one. Specifically, the hunk that touches subst.c addresses an issue with braced parameter expansion, reported at https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00022.html. I suppose that could be split out into a separate patch at some point.
Created attachment 822862 [details, diff] bash-5.2_p2-fixes-for-extglob-in-compat-mode.patch (properly scoped) Here's a revised patch that disentangles it from the fix for brackets appearing in nested parameter expansions, in case it is wanted.
Created attachment 822865 [details, diff] bash-5.2_p2-fix-for-brackets-in-nested-parameter-expansions.patch And here's the split-out fix for for brackets appearing in nested parameter expansions, in case it is wanted.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f311a77983e608c3218f492ece715bced27f77a0 commit f311a77983e608c3218f492ece715bced27f77a0 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-10-08 22:07:20 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-10-08 22:08:13 +0000 app-shells/bash: reorganise 5.2_p2-r2 patches Just splits the patches from 47950445cddff736a1e6c0c1346a20ab6b326cc1 into two. Closes: https://bugs.gentoo.org/873931 See: 47950445cddff736a1e6c0c1346a20ab6b326cc1 Thanks-to: Kerin Millar <kfm@plushkava.net> Signed-off-by: Sam James <sam@gentoo.org> app-shells/bash/bash-5.2_p2-r2.ebuild | 3 +- ...r-brackets-in-nested-parameter-expansions.patch | 41 ++++++++++++++++++++++ ...h-5.2_p2-fixes-for-extglob-in-compat-mode.patch | 38 -------------------- 3 files changed, 43 insertions(+), 39 deletions(-)
(In reply to Kerin Millar from comment #11) > Created attachment 822865 [details, diff] [details, diff] > bash-5.2_p2-fix-for-brackets-in-nested-parameter-expansions.patch > > And here's the split-out fix for for brackets appearing in nested parameter > expansions, in case it is wanted. Thanks for your help overall in this. It is very much appreciated. FYI: I'm intending to wait a bit longer given the recent churn, but my intention is to unmask Bash 5.2 & readline 8.2 probably in a week or so unless someone has strong objections or another bug pops up.
(In reply to Sam James from comment #13) > FYI: I'm intending to wait a bit longer given the recent churn, but my > intention is to unmask Bash 5.2 & readline 8.2 probably in a week or so > unless someone has strong objections or another bug pops up. That seems reasonable. Note that, in principle, 5.2 will be a bit slower for portage. That's because BASH_COMPAT=50 now disables the optimisation implemented in 5.1, whereby here documents and here strings can avoid the use of temporary files for payloads shorter than PIPE_BUF. It's probable that no one will notice but I mention it on the off chance that you do.
kerin notes that one of the patches (I think) doesn't survive in the latest patch release to 5.2_p9. We're going to wait until _p9 is officially announced and then ask upstream if they plan on backporting it.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9f77b77ef167cef8987cf639116541da4628b200 commit 9f77b77ef167cef8987cf639116541da4628b200 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-11-15 07:28:45 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-15 07:32:21 +0000 app-shells/bash: backport another extglob fix It's still not sufficient but with this in on top of _p9, it's very clear what the remaining issues are then, rather than me at least trying to remember exactly what's broken still as the saga continues. It is still not enough to unmask as there's still one extglob issue remaining (and this came up pretty early as far as testing goes). I don't see us being able to expose Bash 5.2 to users for a while yet. Bug: https://bugs.gentoo.org/873931 Signed-off-by: Sam James <sam@gentoo.org> app-shells/bash/bash-5.2_p9-r1.ebuild | 345 +++++++++++++++++++++ ...ash-5.2_p9-fix-for-extglob-in-compat-mode.patch | 42 +++ 2 files changed, 387 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c0322f1d5dbd5b8c0468d93800281923599cfe4b commit c0322f1d5dbd5b8c0468d93800281923599cfe4b Author: Sam James <sam@gentoo.org> AuthorDate: 2022-11-23 23:29:19 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-23 23:29:30 +0000 app-shells/bash: add 5.2_p12 Bug: https://bugs.gentoo.org/873931 Signed-off-by: Sam James <sam@gentoo.org> app-shells/bash/Manifest | 6 + app-shells/bash/bash-5.2_p12.ebuild | 344 ++++++++++++++++++++++++++++++++++++ 2 files changed, 350 insertions(+)
As far as I can tell, 5.2_p12 adequately addresses this issue. The initially reported comsub nesting bug no longer occurs (I tested up to 100 levels), nor does the bug referenced by comment 6.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=abc73485106492b6d0d20ff558c1a856ced1f79d commit abc73485106492b6d0d20ff558c1a856ced1f79d Author: Sam James <sam@gentoo.org> AuthorDate: 2022-11-24 02:38:23 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-24 02:40:54 +0000 profiles: unmask Bash 5.2 and Readline 8.2 There's still a few issues - their presence is subideal, but there's no specific regressions wrt Gentoo usage now, and they're things to take to upstream (and lobby them over) if it's a problem. That is, there's some changes in behaviour the Bash community is understandably not happy about, but there's nothing concrete which upstream would agree is a bug as things stand. Bug: https://bugs.gentoo.org/881379 Closes: https://bugs.gentoo.org/873931 Signed-off-by: Sam James <sam@gentoo.org> profiles/package.mask | 5 ----- 1 file changed, 5 deletions(-)
