Using sys-apps/portage-3.0.57, updating always fail, here's full message: 10:49:33 ~ # emerge --sync gentoo >>> Syncing repository 'gentoo' into '/usr/portage'... /usr/bin/git fetch origin --depth 1 remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 /usr/bin/git reset --merge refs/remotes/origin/master === Sync completed for gentoo Performing Global Updates (Could take a couple of minutes if you have a lot of binary packages.) .='update pass' *='binary update' #='/var/db update' @='/var/db move' s='/var/db SLOT move' %='binary move' S='binary SLOT move' p='update /etc/portage/package.*' /usr/portage/profiles/updates/1Q-2023....................... /usr/portage/profiles/updates/2Q-2023............................... Traceback (most recent call last): File "/usr/lib/python-exec/python3.11/emerge", line 57, in main retval = emerge_main() ^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_emerge/main.py", line 1308, in emerge_main return run_action(emerge_config) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_emerge/actions.py", line 3832, in run_action return action_sync(emerge_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_emerge/actions.py", line 2413, in action_sync success, msgs = syncer.repo(options=options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/emaint/modules/sync/sync.py", line 122, in repo return self._sync(repos, return_messages, emaint_opts=options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/emaint/modules/sync/sync.py", line 261, in _sync self._do_pkg_moves() File "/usr/lib/python3.11/site-packages/portage/emaint/modules/sync/sync.py", line 272, in _do_pkg_moves if self.emerge_config.opts.get("--package-moves") != "n" and _global_updates( ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/_global_updates.py", line 41, in _global_updates return _do_global_updates( ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/_global_updates.py", line 257, in _do_global_updates bindb.update_ents(repo_map, onUpdate=onUpdate) File "/usr/lib/python3.11/site-packages/portage/dbapi/__init__.py", line 426, in update_ents aux_update(cpv, metadata_updates) File "/usr/lib/python3.11/site-packages/portage/dbapi/bintree.py", line 287, in aux_update raise KeyError(cpv) KeyError: 'dev-python/setuptools-65.0.0' Upgrade portage with "emerge -1 portage" also fails with the same messages. Reproducible: Always Steps to Reproduce: 1. With sys-apps/portage-3.0.57 2. emerge --sync gentoo Actual Results: Fail at 'Performing Global Updates' I identified the CPV at: 11:06:11 ~ # grep -n -R dev-python/setuptools-65.0.0 /usr/portage/ /usr/portage/packages/Packages:816:CPV: dev-python/setuptools-65.0.0 After remove the section, also fails with this section: 12:22:52 ~ # grep -n -R media-gfx/blender-3.2.2 /usr/portage/ /usr/portage/packages/Packages:1378:CPV: media-gfx/blender-3.2.2 After remove these two section, emerge works fine. 12:22:15 ~ # emerge --info Portage 3.0.58 (python 3.11.7-final-0, default/linux/amd64/17.1/desktop/gnome/systemd/merged-usr, gcc-13, glibc-2.38-r7, 6.5.7-gentoo x86_64) ================================================================= System uname: Linux-6.5.7-gentoo-x86_64-Intel-R-_Core-TM-_i7-6700HQ_CPU_@_2.60GHz-with-glibc2.38 KiB Mem: 24479536 total, 3392076 free KiB Swap: 8388604 total, 5645136 free Head commit of repository drunkard: d471aa1b6e573d19450509b26b68cdd2a6dd0a08 Timestamp of repository gamerlay: Fri, 15 Dec 2023 07:02:30 +0000 Head commit of repository gamerlay: ff23f0e4c76ae34badca3701de6e334c10408e68 Timestamp of repository gentoo-zh: Fri, 15 Dec 2023 10:46:49 +0000 Head commit of repository gentoo-zh: 3c14a2cb1cf53d60be838149631f417bee3ced38 Timestamp of repository gnome: Thu, 22 Dec 2022 05:17:16 +0000 Head commit of repository gnome: 57d2257b2b27ef52259de5ab28ddf7effc3af1a8 Timestamp of repository pentoo: Fri, 15 Dec 2023 07:02:32 +0000 Head commit of repository pentoo: d3679b89862919bf2c18aa34158c0405ef6a4241 Head commit of repository gentoo: 5d399078deaa87a26be77f2396bd523d7961d05b sh bash 5.2_p21-r1 ld GNU ld (Gentoo 2.41 p2) 2.41.0 app-misc/pax-utils: 1.3.7::gentoo app-shells/bash: 5.2_p21-r1::gentoo dev-java/java-config: 2.3.1-r1::gentoo dev-lang/perl: 5.38.2::gentoo dev-lang/python: 2.7.18_p16-r1::gentoo, 3.9.18::gentoo, 3.10.13::gentoo, 3.11.7::gentoo, 3.12.0_p1::gentoo dev-lang/rust-bin: 1.73.0::gentoo dev-util/cmake: 3.27.9::gentoo dev-util/meson: 1.3.0-r1::gentoo sys-apps/baselayout: 2.14-r1::gentoo sys-apps/sandbox: 2.38::gentoo sys-apps/systemd: 254.7::gentoo sys-devel/autoconf: 2.13-r8::gentoo, 2.71-r7::gentoo sys-devel/automake: 1.16.5-r1::gentoo sys-devel/binutils: 2.41-r2::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/clang: 15.0.7-r3::gentoo, 16.0.6::gentoo, 17.0.6::gentoo sys-devel/gcc: 12.3.1_p20230825::gentoo, 13.2.1_p20231014::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/lld: 15.0.7::gentoo, 16.0.6::gentoo, 17.0.6::gentoo sys-devel/llvm: 15.0.7-r3::gentoo, 16.0.6::gentoo, 17.0.6::gentoo sys-devel/make: 4.4.1-r1::gentoo sys-kernel/linux-headers: 6.6::gentoo (virtual/os-headers) sys-libs/glibc: 2.38-r7::gentoo Repositories: drunkard location: /var/db/repos/drunkard sync-type: git sync-uri: /git/DRUNKARD/overlay.git masters: gentoo volatile: False gamerlay location: /var/db/repos/gamerlay sync-type: git sync-uri: https://github.com/gentoo-mirror/gamerlay.git masters: gentoo volatile: False gentoo-zh location: /var/db/repos/gentoo-zh sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo-zh.git masters: gentoo volatile: False gnome location: /var/db/repos/gnome sync-type: git sync-uri: https://github.com/gentoo-mirror/gnome.git masters: gentoo volatile: False pentoo location: /var/db/repos/pentoo sync-type: git sync-uri: https://github.com/gentoo-mirror/pentoo.git masters: gentoo volatile: False gentoo location: /usr/portage sync-type: git sync-uri: git://anongit.gentoo.org/repo/gentoo.git priority: 9999 volatile: True ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=haswell -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d" CXXFLAGS="-O2 -march=haswell -pipe" DISTDIR="/dev/shm//distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks downgrade-backup ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news nodoc noinfo parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://mirrors.ustc.edu.cn/gentoo/ http://mirrors.aliyun.com/gentoo/ http://mirrors.sohu.com/gentoo/ http://mirrors.kernel.org/gentoo/ " LANG="en_US.utf8" LC_ALL="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" LINGUAS="zh zh-CN zh-HK zh-TW zh_CN zh_HK zh_TW en ru de bo ug" MAKEOPTS="-j8" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/dev/shm/" SHELL="/bin/bash" USE="X a52 aac aalib accessibility acl acpi aio alsa amd64 amr amrenc ao ap apm apng audit bash-completion bluetooth bluray branding broadway bzip2 cairo caps cdda cjk cleartype cli colord cracklib crypt cups dbus dga djvu dri dri3 drm dts dvd dvdr eds egl encode epoll evo exif ffmpeg flac flite fontconfig fortran gallium gconf gdbm gegl geoip gif gimp glamor gles gles2 gnome gnome-keyring gnome-online-accounts gnome-shell gpm gstreamer gtk gtk3 gtk4 gui hidpi http2 iconv icu idn imagemagick inotify introspection ipv6 jack jit jpeg jpeg2k kdbus keyring kms lcms libass libbluray libinput libnotify libtirpc linguas_bo linguas_de linguas_en linguas_ru linguas_ug linguas_zh linguas_zh_CN linguas_zh_HK linguas_zh_TW lto lz4 lzma lzo mad mime minizip mmap mng mp3 mp4 mpeg mtp multilib natspec nautilus ncurses networkmanager nls nptl ogg openal opencc opencl openexr opengl openmp opus pam pango pcre pdf png policykit postgres postscript ppds pulseaudio raw readline realtime sdl seccomp sftp smp sound ssh ssl startup-notification svg systemd test-rust theora threads tiff tracker truetype udev udisks unicode upower urandom usb usbredir v4l v4l2 vaapi vdpau vim-syntax vorbis vpx vulkan wayland webp wifi wxwidgets x264 x265 xattr xcb xft xinerama xkb xml xmp xpm xps xrandr xv xvid xvmc zlib" ABI_X86="64" ADA_TARGET="gnat_2021" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3" CURL_SSL="gnutls" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="evdev keyboard libinput mouse" KERNEL="linux" L10N="zh zh-CN zh-HK zh-TW zh_CN zh_HK zh_TW en ru de bo ug" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" QEMU_SOFTMMU_TARGETS="arm i386 x86_64" QEMU_USER_TARGETS="arm armeb i386 x86_64" RUBY_TARGETS="ruby31 ruby32" SANE_BACKENDS="hp hp3500" VIDEO_CARDS="nvidia intel i965 v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
I imagine 'emaint all -f' will fix this, but this shouldn't be happening to begin with: 1) We should catch the error and give some nicer output; 2) We shouldn't have a dead entry in the index anyway...
I guess the question for 2) is whether the previous fixes in e.g. portage-3.0.57 would prevent such a bad entry being created in the first place.
We should also add a lock around the global updates bit so that it's not possible to have multiple instances trying to do it at the same time (which could trigger a KeyError like this). We can use a global vardb lock here too since we don't want the installed packages mutating either.
I've just noticed that aux_update can interact badly with signed gpkg and FEATURES=pkgdir-index-trusted, leaving a stale entry in $PKGDIR/Packages and triggering a KeyError later. The issue is that aux_update for a signed gpkg will delete the gpkg file without updating internal state including but not limited to $PKGDIR/Packages: https://gitweb.gentoo.org/proj/portage.git/commit/?id=a7bbb4fc4d38f770fc943f3b856c5de56e315fe4 We have some code in the binarytree.inject method that we can reuse to fixup the state. Specifically, the part where it calls cpv_remove, and also _inject_file deletes a stale package index entry from pkgindex.packages before pkgindex is re-written by _pkgindex_write.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=d31f0fee089eb0883c7b48faabee4c47d80cbdf8 commit d31f0fee089eb0883c7b48faabee4c47d80cbdf8 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-12-24 18:40:57 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-12-24 18:57:49 +0000 bindbapi: Update state for package remove in aux_update When removing a signed gpkg in aux_update, update internal state including $PKGDIR/Packages (important especially for FEATURES=pkgdir-index-trusted). Bug: https://bugs.gentoo.org/920095 Fixes: a7bbb4fc4d38 ("Fix move_ent with signed binpkg") Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/bintree.py | 61 +++++++++++++++++++++---- lib/portage/tests/update/test_move_slot_ent.py | 19 +++++--- lib/portage/tests/update/test_update_dbentry.py | 13 ++++-- 3 files changed, 73 insertions(+), 20 deletions(-)
(In reply to Zac Medico from comment #3) > We should also add a lock around the global updates bit so that it's not > possible to have multiple instances trying to do it at the same time (which > could trigger a KeyError like this). We can use a global vardb lock here too > since we don't want the installed packages mutating either. Done in: commit 7100dfb769b9d98eb40a92c37d154cb4bc83292f Author: Zac Medico <zmedico@gentoo.org> Date: Tue Dec 26 14:24:59 2023 -0800 _global_updates: Acquire global vardbapi lock Also use bindbapi.writable attribute in case the PKGDIR is missing for some reason. Bug: https://bugs.gentoo.org/587088 Signed-off-by: Zac Medico <zmedico@gentoo.org>
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0a1f19cdd7a598070b7eb08b3954e677aa4868ad commit 0a1f19cdd7a598070b7eb08b3954e677aa4868ad Author: Sam James <sam@gentoo.org> AuthorDate: 2023-12-27 21:27:55 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-12-27 21:28:01 +0000 sys-apps/portage: add 3.0.59 Closes: https://bugs.gentoo.org/587088 Closes: https://bugs.gentoo.org/822033 Closes: https://bugs.gentoo.org/915494 Closes: https://bugs.gentoo.org/916135 Closes: https://bugs.gentoo.org/917120 Closes: https://bugs.gentoo.org/919862 Closes: https://bugs.gentoo.org/920095 Closes: https://bugs.gentoo.org/920258 Closes: https://bugs.gentoo.org/920537 Closes: https://bugs.gentoo.org/920654 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.59.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
(In reply to Sam James from comment #1) > I imagine 'emaint all -f' will fix this, but this shouldn't be happening to > begin with: > 1) We should catch the error and give some nicer output; > 2) We shouldn't have a dead entry in the index anyway... I've filed bug 920828 for 1).
