Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 933297 - Crash when creating binpkg (KeyError: 'bintree') with FEATURES="downgrade-backup"
Summary: Crash when creating binpkg (KeyError: 'bintree') with FEATURES="downgrade-bac...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 930802
Blocks:
  Show dependency tree
 
Reported: 2024-06-01 04:35 UTC by Sam James
Modified: 2024-06-04 01:31 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (file_933297.txt,170.49 KB, text/plain)
2024-06-01 04:35 UTC, Sam James
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-01 04:35:16 UTC
Created attachment 894764 [details]
build.log

```
$ ebuild antlr4-python3-runtime-4.9.3.ebuild clean merge
[...]
>>> Completed installing dev-python/antlr4-python3-runtime-4.9.3 into /var/tmp/portage/dev-python/antlr4-python3-runtime-4.9.3/image

 * Final size of build directory: 38908 KiB (37.9 MiB)
 * Final size of installed tree:   8936 KiB ( 8.7 MiB)

 * Verifying compiled files for python3.10
 * Verifying compiled files for python3.11
 * Verifying compiled files for python3.12
 * Verifying compiled files for python3.13
 * checking 914 files for package collisions
Process Process-18:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/site-packages/portage/util/_async/ForkProcess.py", line 326, in _bootstrap
    sys.exit(target(*(args or []), **(kwargs or {})))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/portage/dbapi/_MergeProcess.py", line 275, in _target
    rval = mylink.merge(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/portage/dbapi/vartree.py", line 1895, in wrapper
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/portage/dbapi/vartree.py", line 6041, in merge
    retval = self.treewalk(
             ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/portage/dbapi/vartree.py", line 4935, in treewalk
    rval = self._pre_merge_backup(self._installed_instance, downgrade)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/portage/dbapi/vartree.py", line 6190, in _pre_merge_backup
    return self._quickpkg_dblink(backup_dblink, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/portage/dbapi/vartree.py", line 6211, in _quickpkg_dblink
    bintree = trees["bintree"]
              ~~~~~^^^^^^^^^^^
KeyError: 'bintree'
```

----

Portage 3.0.64-29+g120b2ec98-dirty (python 3.12.3-final-0, default/linux/amd64/17.1/hardened, gcc-14, glibc-2.39-r7, 6.6.32-adry x86_64)
=================================================================
System uname: Linux-6.6.32-adry-x86_64-11th_Gen_Intel-R-_Core-TM-_i7-11800H_@_2.30GHz-with-glibc2.39
KiB Mem:    32585744 total,   2981776 free
KiB Swap:    8388604 total,   8139004 free
Timestamp of repository gentoo: Sat, 01 Jun 2024 03:03:58 +0000
Head commit of repository gentoo: 3440b979d5894d4dade7b38ce24f4ce6ae346933

Timestamp of repository emacs: Sun, 12 May 2024 05:06:59 +0000
Head commit of repository emacs: 75ecea1d19781f19e39f6de747f45658608f2e46

Timestamp of repository kde: Thu, 30 May 2024 17:33:47 +0000
Head commit of repository kde: f10c67775d120f33ca716c8e8086b1150c5f3d04

Timestamp of repository qt: Thu, 30 May 2024 13:03:51 +0000
Head commit of repository qt: 003dee833214a358a9cfa818e39ad37d7eeee8a6

sh bash 5.3_alpha-r2
ld GNU ld (Gentoo 9999 p1) 2.42.50.20240528
ccache version 4.10 [disabled]
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.3_alpha-r2::gentoo
dev-build/autoconf:        2.13-r8::gentoo, 2.72-r1::gentoo
dev-build/automake:        1.16.5-r2::gentoo
dev-build/cmake:           3.29.3::gentoo
dev-build/libtool:         2.4.7-r4::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.4.1::gentoo
dev-java/java-config:      2.3.3-r1::gentoo
dev-lang/perl:             5.38.2-r6::gentoo
dev-lang/python:           2.7.18_p16-r2::gentoo, 3.10.14_p1-r1::gentoo, 3.11.9-r1::gentoo, 3.12.3-r1::gentoo, 3.13.0_beta1_p3::gentoo
dev-lang/rust:             1.78.0::gentoo
dev-util/ccache:           4.10-r1::gentoo
sys-apps/baselayout:       2.15::gentoo
sys-apps/openrc:           0.54::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-devel/binutils:        2.42-r1::gentoo, 9999::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           16.0.6::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-devel/gcc:             11.4.1_p20240501::gentoo, 12.3.1_p20240530::gentoo, 13.3.1_p20240524::gentoo, 14.1.1_p20240525::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/lld:             16.0.6::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-devel/llvm:            16.0.6::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-kernel/linux-headers:  6.9::gentoo (virtual/os-headers)
sys-libs/glibc:            2.39-r7::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    priority: -1000
    volatile: False
    sync-git-clone-extra-opts: -b stable -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now
    sync-git-verify-commit-signature: yes

emacs
    location: /var/db/repos/emacs
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/emacs.git
    masters: gentoo
    volatile: False

kde
    location: /var/db/repos/kde
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/kde.git
    masters: gentoo
    volatile: False

local
    location: /var/db/repos/local
    masters: gentoo
    volatile: False

qt
    location: /var/db/repos/qt
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/qt.git
    masters: gentoo
    volatile: False

sam_c
    location: /home/sjames/git/overlay
    masters: gentoo
    volatile: True

Installed sets: @sam-emacs, @sam-fonts, @sam-fortune
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches -ggdb3"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches -ggdb3"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going=y --complete-graph --deep --changed-deps-report=y --usepkg=y --usepkg-exclude nvidia-drivers --usepkg-exclude gentoo-kernel --implicit-system-deps=n"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg clean-logs compressdebug config-protect-if-modified dedupdebug distlocks downgrade-backup ebuild-locks fixlafiles installsources ipc-sandbox merge-wait mount-sandbox multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms split-elog split-log splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches"
GENTOO_MIRRORS="https://www.mirrorservice.org/sites/distfiles.gentoo.org/ https://mirror.bytemark.co.uk/gentoo/ https://mirrors.soeasyto.com/distfiles.gentoo.org/"
LANG="en_GB.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -Wl,-z,pack-relative-relocs"
LEX="flex"
LINGUAS="en en_GB en_US"
MAKEOPTS="-j16"
PKGDIR="/var/cache/binpkgs"
PORTAGE_COMPRESS="xz"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RUSTFLAGS="-C target-cpu=native -C link-arg=-Wl,-z,pack-relative-relocs"
SHELL="/bin/bash"
USE="PIC X a52 aac acl acpi activities aes alsa amd64 avx avx2 avx512bw avx512cd avx512dq avx512f avx512vbmi avx512vl bash-completion bluetooth branding bzip2 cairo caps cdda cdr cet clang cli crypt dbus declarative dist-kernel dri dts dvd dvdr elogind emacs encode exif f16c flac fma3 fortran gdbm gif gles2 gmp gpm graphite gtk gui hardened hunspell iconv icu ipv6 iwd jit jpeg kde kf6compat kwallet lcms libnotify libtirpc lto mad mmx mmxext mng modules-compress mp3 mp4 mpeg multilib ncurses networkmanager ogg opengl openmp pam pango pclmul pcre pdf pgo pic pie pipewire plasma png policykit popcnt ppds pulseaudio qml qt5 rdrand readline screencast sdl seccomp semantic-desktop sha smartcard sound spell split-usr sse sse2 sse3 sse4_1 sse4_2 ssl ssp ssse3 startup-notification svg syslog system-av1 system-binutils system-boost system-bootstrap system-cairo system-clang system-digest system-ffmpeg system-harfbuzz system-heimdal system-icu system-jpeg system-jsoncpp system-leveldb system-libevent system-libs system-libvpx system-libyaml system-llvm system-lz4 system-mitkrb5 system-sqlite system-ssl system-tbb system-uulib system-webp system-zlib test-rust tiff truetype udev udisks unicode upower usb vaapi verify-sig vorbis vulkan wayland widgets x264 xattr xcb xft xml xtpax xv xvid zlib zsh-completion zstd" ABI_X86="32 64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 avx512f avx512dq avx512cd avx512bw avx512vl avx512vbmi f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-GB en-US" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11 python3_10 python3_12 python3_13" RUBY_TARGETS="ruby31 ruby32 ruby33" VIDEO_CARDS="intel i965 iris nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-01 04:36:28 UTC
Hit this after running 'cp antlr4-python3-runtime-{4.13.1,4.9.3.ebuild}' in dev-python/antlr-python3-runtime at commit 3f63c5bd7c8fd39ff70ca518db4f4d6bdce625f6.

I doubt it's relevant but I say it for completeness given nothing comes to mind for what would've changed recently for this?
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-01 04:36:46 UTC
Another merge is running, by the way, but it's currently mid-GCC build.
Comment 3 Zac Medico gentoo-dev 2024-06-01 04:49:12 UTC
Inside lib/portage/dbapi/_MergeProcess.py we need to add bintree here:

        # Since the entire QueryCommand._db is not required, only pass
        # in tree types that QueryCommand specifically requires.
        child_db = {}
        parent_db = portage.db if QueryCommand._db is None else QueryCommand._db
        for root in parent_db:
            child_db[root] = {}
            for tree_type in ("vartree", "porttree"):
                child_db[root][tree_type] = parent_db[root][tree_type]
Comment 4 Zac Medico gentoo-dev 2024-06-01 04:51:51 UTC
The flaw was in this commit:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b9a85ff987ea677e350fb70e2c3355197ee6c39b

commit b9a85ff987ea677e350fb70e2c3355197ee6c39b
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2023-10-22 08:08:02 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2023-10-22 21:29:49 +0000

    MergeProcess: Support QueryCommand with spawn start method
    
    Since QueryCommand._db does not propagate to the child process
    in MergeProcess with the multiprocessing spawn start method,
    handle it by passing QueryCommand._db as an explicit parameter.
    
    Since self.mydbapi can be a bindbapi instance, override the
    bindbapi __getstate__ method so that it is picklable (omit
    the unpicklable _aux_cache_slot_dict attribute and regenerate
    it after unpickling).
    
    Bug: https://bugs.gentoo.org/916106
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/dbapi/_MergeProcess.py | 19 ++++++++++++++++---
 lib/portage/dbapi/bintree.py       | 20 ++++++++++++++++++--
 2 files changed, 34 insertions(+), 5 deletions(-)
Comment 5 Larry the Git Cow gentoo-dev 2024-06-01 19:20:46 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=1a7fc63d20ad2e1292be3697c105c2d7e1691f91

commit 1a7fc63d20ad2e1292be3697c105c2d7e1691f91
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-06-01 04:56:45 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-06-01 19:18:35 +0000

    MergeProcess: Pass bintree to subprocess
    
    It's required for FEATURES=*-backup.
    
    Fixes: b9a85ff987ea ("MergeProcess: Support QueryCommand with spawn start method")
    Bug: https://bugs.gentoo.org/933297
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/dbapi/_MergeProcess.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Comment 6 Larry the Git Cow gentoo-dev 2024-06-04 01:31:40 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1dd08f05355b14c7594d0e0ccf46cdb2b1486cbe

commit 1dd08f05355b14c7594d0e0ccf46cdb2b1486cbe
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-06-04 01:31:07 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-06-04 01:31:07 +0000

    sys-apps/portage: add 3.0.65
    
    Closes: https://bugs.gentoo.org/930802
    Closes: https://bugs.gentoo.org/787623
    Closes: https://bugs.gentoo.org/922629
    Closes: https://bugs.gentoo.org/927241
    Closes: https://bugs.gentoo.org/930992
    Closes: https://bugs.gentoo.org/932739
    Closes: https://bugs.gentoo.org/932804
    Closes: https://bugs.gentoo.org/933297
    Closes: https://bugs.gentoo.org/933385
    Closes: https://bugs.gentoo.org/933442
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.65.ebuild | 242 +++++++++++++++++++++++++++++++++
 2 files changed, 243 insertions(+)