Under some conditions aborted merges are listed, when running qlop -r Reproducible: Always Steps to Reproduce: 1. emerge <package 1> 2. emerge <package 2> 3. abort package 2 4. run qlop -r 5. package 1 is finished/canceled 6. run qlop -r Actual Results: qlop -r in line 4 reports both package 1 and package 2 as running. qlop -r in line 6 is fine and reports nothing Expected Results: qlop -r in line 4 should only report package 1 as running. # emerge --info portage-utils Portage 2.3.76 (python 3.6.9-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-8.3.0, glibc-2.29-r2, 5.2.16-gentoo x86_64) ================================================================= System Settings ================================================================= System uname: Linux-5.2.16-gentoo-x86_64-Intel-R-_Core-TM-2_Duo_CPU_T8300_@_2.40GHz-with-gentoo-2.6 KiB Mem: 4027464 total, 838648 free KiB Swap: 6291452 total, 4662208 free Timestamp of repository gentoo: Tue, 08 Oct 2019 19:05:57 +0000 Head commit of repository gentoo: 37e466a54627f9231d2481dd80be37848c93989d sh bash 4.4_p23-r1 ld GNU ld (Gentoo 2.32 p2) 2.32.0 app-shells/bash: 4.4_p23-r1::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.28.2-r1::gentoo dev-lang/python: 2.7.16::gentoo, 3.6.9::gentoo dev-util/cmake: 3.14.6::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.41.2::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.15.1-r2::gentoo, 1.16.1-r1::gentoo sys-devel/binutils: 2.32-r1::gentoo sys-devel/gcc: 8.3.0-r1::gentoo sys-devel/gcc-config: 2.0::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 4.19::gentoo (virtual/os-headers) sys-libs/glibc: 2.29-r2::gentoo Repositories: gentoo location: /usr/portage sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo.git priority: -1000 sync-git-verify-commit-signature: true shared_overlay location: /opt/conf/common/var/db/repos/shared_overlay masters: gentoo priority: 100 local_overlay location: /var/db/repos/local_overlay masters: gentoo priority: 200 Installed sets: @system ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -ftree-vectorize -ggdb" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.6/conf" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/portage/package.accept_keywords/99-autounmask /etc/portage/package.unmask/99-autounmask /etc/portage/package.use/99-autounmask /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe -ftree-vectorize -ggdb" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask-write --autounmask-continue --jobs=2 --load-average=2 --backtrack=100" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned sandbox sfperms sign splitdebug strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j2" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi activities alsa amd64 apng bash-completion berkdb bluetooth branding brotli bzip2 cairo cdaudio cdda cddb cdparanoia cdr chm cli consolekit crypt cups cxx dbus declarative dell djvu dnssec dri dri3 dts dvd dvdr emboss encode epub evdev exif fam ffmpeg flac fortran gdbm gif glamor glib gpg gpm gstreamer gtk gzip iconv icu id3tag idn ieee1394 imagemagick inotify ipv6 irc jpeg jpeg2k kde kipi kwallet lame lcms ldap libnotify libsamplerate libtirpc lvm lzma mad matroska mmx mmxext mng mp3 mp4 mpeg mplayer mtp multilib musicbrainz ncurses nls nptl ntp ogg opengl openmp opus otr pam pango pcre pdf phonon plasma pm-utils png policykit ppds pulseaudio qml qt5 quicktime rar readline real rss sdl seccomp semantic-desktop spell split-usr sse sse2 sse3 sse4_1 ssl ssse3 startup-notification svg taglib tcpd theora threads thumbnail tiff truetype twolame udev udisks unicode upower usb v4l v4l2 vaapi vcd vim-syntax visualization vorbis vpx wavpack widgets wifi wma wmf wxwidgets x264 x265 xattr xcb xcomposite xinerama xml xv xvid xvidv xvmc xz zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 sse4_1 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS ================================================================= Package Settings ================================================================= app-portage/portage-utils-0.80::gentoo was built with the following: USE="nls openmp qmanifest qtegrity -libressl -static" ABI_X86="(64)"
The problem goes further: 1. emerge <package 1> 2. emerge <package 2> 3. package 1 is finished 4. run qlop -r In the above sequence package 2 is not listed in line 4, although the merge process is still running.
I'll have a look, but I fear this is impossible. Portage's emerge.log is very poorly designed, resulting in all kinds of guesswork.
At a quick glance, genlop -c seems to handle this correctly. Thus, it might be worth looking at their logic for this case.
could you share your emerge.log and the time on which you observed this problem?
Created attachment 592868 [details] tail of emerge.log emerge log tail (tail -n 10000 emerge.log) with reproduced errors. Case 1 happened at 2019-10-16T15:54:01+02:00 Case 2 happened at 2019-10-16T16:00:23+02:00
(In reply to Till Schäfer from comment #5) > Created attachment 592868 [details] > tail of emerge.log > > emerge log tail (tail -n 10000 emerge.log) with reproduced errors. > > Case 1 happened at 2019-10-16T15:54:01+02:00 > Case 2 happened at 2019-10-16T16:00:23+02:00 ups, case 1 was @ 2019-10-16T14:54:24+02:00
thanks!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=e526bb8f26e2065bf990a0783ef3b1a9ad1fdb3d commit e526bb8f26e2065bf990a0783ef3b1a9ad1fdb3d Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2019-10-20 09:07:31 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2019-10-20 09:07:31 +0000 qlop: don't "forget" merges when a parallel merge finishes Stop seeing finishing emerge processes double, thereby wrongly calculating the current parallel merges. Bug: https://bugs.gentoo.org/697068 Signed-off-by: Fabian Groffen <grobian@gentoo.org> qlop.c | 3 +-- tests/qlop/dotest | 4 ++++ tests/qlop/parallel.log | 53 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 2 deletions(-)
(In reply to Till Schäfer from comment #0) > Under some conditions aborted merges are listed, when running qlop -r > > Reproducible: Always > > Steps to Reproduce: > 1. emerge <package 1> > 2. emerge <package 2> > 3. abort package 2 > 4. run qlop -r > 5. package 1 is finished/canceled > 6. run qlop -r > Actual Results: > qlop -r in line 4 reports both package 1 and package 2 as running. > qlop -r in line 6 is fine and reports nothing > > Expected Results: > qlop -r in line 4 should only report package 1 as running. Unless I misunderstand, reporting only package 1 in step 4 is impossible, because the log never shows package 2 was aborted. The log also doesn't show which of the two emerge processes was terminated, so the best one can do is to show both package 1 and 2 (thus lie about 2). The only way to check this is to look at processes, but since emerge/portage itself doesn't identify what it is doing, it would rely on external tools such as sandbox which happen to identify themselves. But sandbox is likely going away, and not used everywhere. (In reply to Till Schäfer from comment #1) > The problem goes further: > > 1. emerge <package 1> > 2. emerge <package 2> > 3. package 1 is finished > 4. run qlop -r > > In the above sequence package 2 is not listed in line 4, although the merge > process is still running. This is a bug I think I found the cause for (and the fix). (In reply to Till Schäfer from comment #6) > (In reply to Till Schäfer from comment #5) > > Created attachment 592868 [details] > > tail of emerge.log > > > > emerge log tail (tail -n 10000 emerge.log) with reproduced errors. > > > > Case 1 happened at 2019-10-16T15:54:01+02:00 > > Case 2 happened at 2019-10-16T16:00:23+02:00 > > ups, case 1 was @ 2019-10-16T14:54:24+02:00 These timestamps don't appear in the logs, something that perhaps resembles your desciption actually is reported in the log as a successfully completed merge. I can't use genlop to verify anything here, because genlop uses sandbox to determine running merges.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0b04f542749e2fb95c627364bf58b075c164c4f6 commit 0b04f542749e2fb95c627364bf58b075c164c4f6 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2019-10-20 10:25:14 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2019-10-20 10:25:38 +0000 app-portage/portage-utils: version bump to v0.81 - 697094: qfile incorrectly matching /usr/lib - 697068: qlop -r not showing ongoing merges from parallel merges - 696078: qgrep not matching revisioned ebuilds - 695586: qpkg double frees and incorrect unpacking - 694972: qlop -r/-a speedups - 692224: qlop support for alternate ROOT - 677982: qfile report matches from prune lib registry Closes: https://bugs.gentoo.org/697094 Closes: https://bugs.gentoo.org/697068 Closes: https://bugs.gentoo.org/696078 Closes: https://bugs.gentoo.org/695586 Closes: https://bugs.gentoo.org/694972 Closes: https://bugs.gentoo.org/692224 Closes: https://bugs.gentoo.org/677982 Package-Manager: Portage-2.3.76, Repoman-2.3.16 Signed-off-by: Fabian Groffen <grobian@gentoo.org> app-portage/portage-utils/Manifest | 1 + .../portage-utils/portage-utils-0.81.ebuild | 87 ++++++++++++++++++++++ 2 files changed, 88 insertions(+)
now, qlop -r seems to list all aborted merges -> see Bug 698196
