When using more than one flag in BINPKG_COMPRESS_FLAGS, for example: BINPKG_COMPRESS="zstd" BINPKG_COMPRESS_FLAGS="-T0 -6" quickpkg fails: $ sudo quickpkg binutils * Building package for sys-devel/binutils-2.38-r2 ...Incorrect parameters Usage : zstd [args] [FILE(s)] [-o file] FILE : a filename with no FILE, or when FILE is - , read standard input Arguments : -# : # compression level (1-19, default: 3) -d : decompression -D DICT: use DICT as Dictionary for compression or decompression -o file: result stored into `file` (only 1 output file) -f : disable input and output checks. Allows overwriting existing files, input from console, output to stdout, operating on links, block devices, etc. --rm : remove source file(s) after successful de/compression -k : preserve source file(s) (default) -h/-H : display help/long help and exit Traceback (most recent call last): File "/usr/lib/python-exec/python3.10/quickpkg", line 338, in quickpkg_main atom = Atom(arg, allow_wildcard=True, allow_repo=True) File "/usr/lib/python3.10/site-packages/portage/dep/__init__.py", line 1525, in __init__ raise InvalidAtom(self) portage.exception.InvalidAtom: binutils During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/portage/dbapi/vartree.py", line 2194, in quickpkg tar_contents( File "/usr/lib/python3.10/site-packages/portage/dbapi/vartree.py", line 6544, in tar_contents tar.addfile(tarinfo) File "/usr/lib/python3.10/tarfile.py", line 2015, in addfile self.fileobj.write(buf) File "/usr/lib/python3.10/tarfile.py", line 437, in write self.__write(s) File "/usr/lib/python3.10/tarfile.py", line 445, in __write self.fileobj.write(self.buf[:self.bufsize]) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python-exec/python3.10/quickpkg", line 430, in <module> retval = quickpkg_main(options, args, eout) File "/usr/lib/python-exec/python3.10/quickpkg", line 341, in quickpkg_main quickpkg_atom(options, infos, arg, eout) File "/usr/lib/python-exec/python3.10/quickpkg", line 178, in quickpkg_atom excluded_config_files = dblnk.quickpkg( File "/usr/lib/python3.10/site-packages/portage/dbapi/vartree.py", line 2189, in quickpkg with tarfile.open( File "/usr/lib/python3.10/tarfile.py", line 2491, in __exit__ self.fileobj.close() File "/usr/lib/python3.10/tarfile.py", line 461, in close self.fileobj.write(self.buf) BrokenPipeError: [Errno 32] Broken pipe Reproducible: Always Steps to Reproduce: In make.conf, set: BINPKG_COMPRESS="zstd" BINPKG_COMPRESS_FLAGS="-T0 -6" and then try to quickpkg a package: sudo quickpkg binutils Actual Results: Fails. Expected Results: Should work.
$ emerge --info portage Portage 3.0.32 (python 3.10.5-final-0, default/linux/amd64/17.1/desktop/plasma/systemd, gcc-11.3.0, glibc-2.35-r8, 5.15.54-gentoo x86_64) ================================================================= System Settings ================================================================= System uname: Linux-5.15.54-gentoo-x86_64-AMD_Ryzen_7_3700X_8-Core_Processor-with-glibc2.35 KiB Mem: 16313784 total, 8478036 free KiB Swap: 5665600 total, 5665600 free Timestamp of repository gentoo: Thu, 14 Jul 2022 06:00:01 +0000 Head commit of repository gentoo: 0372ef7872544ba2d39e3aec30843681186c4183 Timestamp of repository steam-overlay: Thu, 14 Jul 2022 00:46:36 +0000 Head commit of repository steam-overlay: e71c67269ee996fc8ea0e76d0ae4d50e6292432c sh bash 5.1_p16 ld GNU ld (Gentoo 2.38 p4) 2.38 app-misc/pax-utils: 1.3.4::gentoo app-shells/bash: 5.1_p16::gentoo dev-lang/perl: 5.36.0::gentoo dev-lang/python: 3.9.13::gentoo, 3.10.5::gentoo dev-lang/rust-bin: 1.62.0::gentoo dev-util/cmake: 3.23.2::gentoo dev-util/meson: 0.63.0::gentoo sys-apps/baselayout: 2.8::gentoo sys-apps/sandbox: 2.29::gentoo sys-apps/systemd: 251.2::gentoo sys-devel/autoconf: 2.71-r1::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.38-r2::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/clang: 13.0.1::gentoo, 14.0.6::gentoo sys-devel/gcc: 11.3.0::gentoo sys-devel/gcc-config: 2.5-r1::gentoo sys-devel/libtool: 2.4.7::gentoo sys-devel/lld: 13.0.1::gentoo sys-devel/llvm: 13.0.1::gentoo, 14.0.6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.18-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.35-r8::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-max-age: 24 sync-rsync-extra-opts: sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 local location: /usr/local/portage masters: gentoo steam-overlay location: /var/db/repos/steam-overlay sync-type: git sync-uri: https://github.com/gentoo-mirror/steam-overlay.git masters: gentoo ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -mtune=native -march=native -ftree-vectorize" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/vmware-installer" CXXFLAGS="-O2 -pipe -mtune=native -march=native -ftree-vectorize" DISTDIR="/mnt/Data/cache/gentoo/distfiles" EMERGE_DEFAULT_OPTS="--backtrack=200 --with-bdeps=y" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe -mtune=native -march=native -ftree-vectorize" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live cgroup config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe -mtune=native -march=native -ftree-vectorize" GENTOO_MIRRORS="http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://ftp.ntua.gr/pub/linux/gentoo/" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en_US en" MAKEOPTS="-j8" PKGDIR="/mnt/Data/cache/gentoo/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="X a52 aac acl acpi activities amd64 avif branding bzip2 cairo cdr cli crypt dbus declarative dri dts dvdr emoji encode exif flac fortran gdbm gif gpm grub gtk gui iconv icu ipv6 jpeg kde kwallet lcms libglvnd libnotify libtirpc lto mad mng mp3 mp4 mpeg multilib ncurses networkmanager nptl nvidia offensive ogg opengl openmp pam pango pcre pdf pgo pipewire plasma png policykit ppds pulseaudio qml qt5 readline screencast sdl seccomp spell split-usr ssl startup-notification svg systemd tiff truetype udev udisks unicode upower usb vorbis vulkan wayland webp widgets wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="64 32" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en-US en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="WebAssembly" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10 python3_9" RUBY_TARGETS="ruby31" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS ================================================================= Package Settings ================================================================= sys-apps/portage-3.0.32::gentoo was built with the following: USE="(ipc) native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev (-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_10 python3_9 -pypy3 -python3_11 -python3_8" FEATURES="config-protect-if-modified sfperms qa-unresolved-soname-deps cgroup unmerge-logs xattr usersandbox binpkg-dostrip buildpkg-live news fixlafiles binpkg-multi-instance ebuild-locks ipc-sandbox userfetch network-sandbox usersync assume-digests unknown-features-warn merge-sync pid-sandbox binpkg-docompress sandbox multilib-strict userpriv protect-owned strict unmerge-orphans parallel-fetch preserve-libs distlocks binpkg-logs"
Please share 'zstd --version'.
(In reply to Sam James from comment #2) > Please share 'zstd --version'. *** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
Downgrading to =sys-apps/portage-3.0.30-r3 makes it work.
Bisected to: b094ba47368bb9b16fb17ee47a54644a09860823 is the first bad commit commit b094ba47368bb9b16fb17ee47a54644a09860823 Author: Rin Cat (鈴猫) <dev@rincat.ch> Date: Sun Jan 23 18:23:14 2022 -0500 */*: GPKG (new binpkg format) Support (GLEP 78, provisionally) Bug: https://bugs.gentoo.org/500630 Bug: https://bugs.gentoo.org/659864 Bug: https://bugs.gentoo.org/672672 Bug: https://bugs.gentoo.org/773259 Bug: https://bugs.gentoo.org/820578 Signed-off-by: Rin Cat (鈴猫) <dev@rincat.ch> Closes: https://github.com/gentoo/portage/pull/562 Signed-off-by: Sam James <sam@gentoo.org>
My suspicion is that it's treating the args in BINPKG_COMPRESS_FLAGS as a single string -- and a single arg as a result. With the following settings in make.conf: BINPKG_COMPRESS="xz" BINPKG_COMPRESS_FLAGS="-9 -e" I get the following error (extra printing of the command and args used): $ /home/oskari/coding/portage/bin/quickpkg sys-devel/binutils * Building package for sys-devel/binutils-2.37_p1-r2 ... cmd: 'xz' cmd: '-9 -e' xz: invalid option -- ' ' xz: Try `xz --help' for more information. Traceback (most recent call last): File "/home/oskari/coding/portage/lib/portage/dbapi/vartree.py", line 2194, in quickpkg tar_contents( File "/home/oskari/coding/portage/lib/portage/dbapi/vartree.py", line 6544, in tar_contents tar.addfile(tarinfo) File "/usr/lib/python3.10/tarfile.py", line 2015, in addfile self.fileobj.write(buf) File "/usr/lib/python3.10/tarfile.py", line 437, in write self.__write(s) File "/usr/lib/python3.10/tarfile.py", line 445, in __write self.fileobj.write(self.buf[:self.bufsize]) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/oskari/coding/portage/bin/quickpkg", line 435, in <module> retval = quickpkg_main(options, args, eout) File "/home/oskari/coding/portage/bin/quickpkg", line 351, in quickpkg_main quickpkg_atom(options, infos, atom, eout) File "/home/oskari/coding/portage/bin/quickpkg", line 183, in quickpkg_atom excluded_config_files = dblnk.quickpkg( File "/home/oskari/coding/portage/lib/portage/dbapi/vartree.py", line 2189, in quickpkg with tarfile.open( File "/usr/lib/python3.10/tarfile.py", line 2491, in __exit__ self.fileobj.close() File "/usr/lib/python3.10/tarfile.py", line 461, in close self.fileobj.write(self.buf) BrokenPipeError: [Errno 32] Broken pipe The "-9 -e" is indeed a single element in the array used. Although, what's surprising to me is that xz is complaining about ' ' being an argument. I believe subprocess.Popen() is used incorrectly [1]: proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=fobj) The Python docs say that the first element should be (an absolute path to) the program to execute, and subsequent elements should each be args [2]. It's similar to how execve(2) works. [1]: https://github.com/gentoo/portage/blob/1a796dafdb46f6ec1aeed1281f4097f205a91cef/bin/quickpkg#L177 [2]: https://docs.python.org/3/library/subprocess.html#popen-constructor
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=158ff662921150011fdd9929e8e7d5f4be946a5d commit 158ff662921150011fdd9929e8e7d5f4be946a5d Author: Sheng Yu <syu.os@protonmail.com> AuthorDate: 2022-07-17 23:00:14 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-07-18 20:41:57 +0000 quickpkg: Fix incorrect BINPKG_COMPRESS_FLAGS split. Bug: https://bugs.gentoo.org/857933 Signed-off-by: Sheng Yu <syu.os@protonmail.com> Closes: https://github.com/gentoo/portage/pull/857 Signed-off-by: Sam James <sam@gentoo.org> NEWS | 2 ++ bin/quickpkg | 5 +---- 2 files changed, 3 insertions(+), 4 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=12cec950b69ca41a94fdc756e4025c6ea85c41a1 commit 12cec950b69ca41a94fdc756e4025c6ea85c41a1 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-07-18 21:43:03 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-07-18 21:43:12 +0000 sys-apps/portage: add 3.0.33 Closes: https://bugs.gentoo.org/858824 Closes: https://bugs.gentoo.org/857933 Closes: https://bugs.gentoo.org/858407 Closes: https://bugs.gentoo.org/858818 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.33.ebuild | 273 +++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+)
