I am using app-admin/metalog for a long time. Since I updated to app-admin/metalog-20200113, I noticed, that some of my Gentoo systems becomes unbootable. Unbootable means, that the system hangs on various daemons to start during boot. It affects every service, which needs to log into the system logger. If I do disable admin/metalog from boot completly or downgrade back to app-admin/metalog-20181125, everything goes back to normal. Note: This problem seems not to affect every of my Gentoo machines! Some of them, which got also updated to app-admin/metalog-20200113, just work fine. The only difference I see, that "ps aux" reports a defunct on the new metalog version on those systems, where it hangs. So it makes sense, that somehow every service hangs forever, when it wants to log into the defunct logger. System working: root 8029 0.0 0.0 3020 1224 ? Ss 12:21 0:00 metalog [MASTER] root 8030 0.0 0.0 3020 120 ? S 12:21 0:00 metalog [KERNEL] System failing: root 4403 0.0 0.0 3020 1356 ? Ss 12:20 0:00 metalog [MASTER] root 4404 0.0 0.0 3020 120 ? S 12:20 0:00 metalog [KERNEL] root 4558 0.0 0.0 0 0 ? Z 12:20 0:00 [metalog] <defunct>
Created attachment 613942 [details] strace.log This is an strace of running /usr/sbin/metalog, which is hanging forever.
emerge --info Portage 2.3.87 (python 3.8.1-final-0, default/linux/amd64/17.1/developer, gcc-9.2.0, glibc-2.30-r3, 5.5.2-gentoo-r1 x86_64) ================================================================= System uname: Linux-5.5.2-gentoo-r1-x86_64-Intel-R-_Core-TM-_i7-6600U_CPU_@_2.60GHz-with-glibc2.4 KiB Mem: 32592136 total, 28979828 free KiB Swap: 17301500 total, 17301500 free Timestamp of repository gentoo: Fri, 14 Feb 2020 01:45:01 +0000 Head commit of repository gentoo: 542ac4493fb6ec1442a58cf4f29efa8433e096af Head commit of repository tlp: 118404c26c8d17ae69ab4ebd57f5455b58f489c2 sh bash 5.0_p16 ld GNU ld (Gentoo 2.34 p1) 2.34.0 distcc 3.3.3 x86_64-pc-linux-gnu [disabled] ccache version 3.7.7 [enabled] app-shells/bash: 5.0_p16::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.17-r1::gentoo, 3.6.10::gentoo, 3.7.6::gentoo, 3.8.1::gentoo, 3.9.0_alpha2::gentoo dev-util/ccache: 3.7.7::gentoo dev-util/cmake: 3.16.4::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.7::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.18::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r5::gentoo sys-devel/automake: 1.11.6-r3::gentoo, 1.13.4-r2::gentoo, 1.16.1-r2::gentoo sys-devel/binutils: 2.33.1-r1::gentoo, 2.34::gentoo sys-devel/gcc: 9.2.0-r3::gentoo sys-devel/gcc-config: 2.2.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.5::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r3::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: rsync sync-uri: rsync://rsync.de.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-jobs: 1 sync-rsync-verify-max-age: 24 sync-rsync-extra-opts: --exclude-from=/etc/portage/rsync_excludes sync-rsync-verify-metamanifest: yes crossdev location: /usr/local/crossdev masters: gentoo serenity location: /usr/local/portage masters: gentoo sattvik location: /var/lib/layman/sattvik sync-type: webrsync sync-uri: https://anongit.gentoo.org/git/user/sattvik.git masters: gentoo priority: 50 steam-overlay location: /var/lib/layman/steam-overlay sync-type: webrsync sync-uri: https://github.com/anyc/steam-overlay.git masters: gentoo priority: 50 tlp location: /var/lib/layman/tlp sync-type: git sync-uri: git://github.com/dywisor/tlp-portage.git masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -mfpmath=sse,387 -mtune=intel -O3 -frecord-gcc-switches -fomit-frame-pointer -malign-data=abi -mtls-dialect=gnu2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -mfpmath=sse,387 -mtune=intel -O3 -frecord-gcc-switches -fomit-frame-pointer -malign-data=abi -mtls-dialect=gnu2 -pipe -fvisibility-inlines-hidden" DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS="--autounmask=n --quiet-build=n --with-bdeps=y" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-frecord-gcc-switches" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs ccache collision-protect config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict news parallel-fetch pid-sandbox protect-owned sandbox sfperms splitdebug strict strict-keepdir test-fail-continue unknown-features-warn unmerge-logs unmerge-orphans userpriv usersandbox xattr" FFLAGS="-frecord-gcc-switches" GENTOO_MIRRORS="https://mirror.netcologne.de/gentoo/" INSTALL_MASK="/etc/systemd/* /etc/xinetd.d/* /usr/lib/systemd/*" LANG="de_DE.UTF-8" LC_ALL="de_DE.UTF-8" LDFLAGS="-Wl,--as-needed -Wl,-O1 -Wl,--sort-common -Wl,-z,now" LINGUAS="de de_DE" MAKEOPTS="-j5 -l4" PKGDIR="/var/cache/binpkgs" PORTAGE_BUNZIP2_COMMAND="pbunzip2" PORTAGE_BZIP2_COMMAND="pbzip2" PORTAGE_COMPRESS="xz" PORTAGE_COMPRESS_FLAGS="-9e" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac aalib accessibility acl adns alsa amd64 berkdb bidi bluetooth bzip2 cairo calendar caps cdda cddb cdinstall cgi cjk colord cracklib crypt cups curl cvs cxx dbus dga djvu dri dts dv dvb dvd eds elogind encode exif fbcon ffmpeg fftw flac fontconfig fortran ftp gd gdbm geoip geolocation gif git gmp gnome-keyring gnutls gphoto2 gpm graphviz gsm gstreamer gtk iconv icu idn ieee1394 imagemagick imap imlib inotify introspection ipv6 java javascript jbig jemalloc jingle jit joystick jpeg jpeg2k lame latex lcms ldap libass libcaca libnotify libsamplerate lm-sensors lua luajit lzma lzo mad maildir matroska mhash mikmod milter mmap mms mng modplug mono motif mp3 mpeg mplayer mtp musepack mysql mysqli ncurses networkmanager nls nntp nptl nsplugin offensive ogg openal openexr opengl openmp opus orc oss pam pcre pdf perl pie png policykit posix postscript pulseaudio python qt5 quicktime raw rdp readline rtmp samba sasl scanner sctp sdl seccomp session slang slp smartcard sndfile snmp soap sockets sound sox speex spell split-usr sqlite ssl startup-notification subversion suid svg symlink syslog taglib tcmalloc tcpd theora threads tiff timidity truetype udev udisks unicode upnp usb v4l vaapi vcd vdpau vim-syntax vnc vorbis vpx wavpack wayland webkit webp wifi wmf wxwidgets x264 xattr xcb xcomposite xface xft xinetd xml xmp xpm xscreensaver xv xvid zeroconf zip zlib zsh-completion zstd" ABI_X86="32 64" COLLECTD_PLUGINS="apcups battery cgroups conntrack contextswitch cpu cpufreq cpusleep df disk dns entropy ethstat exec fhcount filecount gps hddtemp hugepages interface ipc ipmi iptables irq load lvm java mcelog md memory mysql netlink nfs nginx ntpd openldap openvpn ping postgresql processes protocols rrdcached sensors serial smart snmp swap syslog tcpconns thermal turbostat unixsock uptime users wireless" CPU_FLAGS_X86="aes avx avx2 mmx mmxext sse sse2 sse3 sse4_1 sse4_2 ssse3" CURL_SSL="openssl" ELIBC="glibc" GPSD_PROTOCOLS="garmin garmintxt nmea0183" INPUT_DEVICES="libinput" KERNEL="linux" L10N="de de_DE" LCD_DEVICES="cfontzpacket crystalfontz glcd2usb hd44780 lcd2usb" LIBREOFFICE_EXTENSIONS="nlpsolver scripting-beanshell scripting-javascript wiki-publisher" NGINX_MODULES_HTTP="access auth_basic autoindex brotli browser charset fancyindex fastcgi geo geoip geoip2 gunzip gzip gzip_static headers_more image_filter javascript limit_conn limit_req map metrics proxy realip referer rewrite secure_link sticky stub_status upload_progress upstream_keepalive vhost_traffic_status xslt" NGINX_MODULES_MAIL="imap pop3 smtp" NGINX_MODULES_STREAM="access geo geoip geoip2 javascript limit_conn map realip return ssl_preread" PHP_TARGETS="php7-4" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6 python3_7 python3_8" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby26 ruby27" SANE_BACKENDS="hp net" USERLAND="GNU" VIDEO_CARDS="i965 intel iris" Unset: CC, CPPFLAGS, CTARGET, CXX, PORTAGE_BINHOST
I found the cause! The new ebuild is doing that: dosbin "${FILESDIR}"/consolelog.sh-r1 This means, that consolelog.sh is now installed as "consolelog.sh-r1". I consider that wrong, since the metalog.conf still references consolelog.sh. The shipped metalog.conf also uses consolelog.sh. So that should be fixed with an newsbin, to install consolelog.sh-r1 as consolelog.sh. Strace shows, that metalog tries all time to execute /sbin/consolelog.sh with all my kernellog. Since that does not exist, it fails and goes into defunct as a service.
After logging through logs, I can see, all my systems were affected. Systems, which did not locked up during boot, still did not logged much. This is an example of a system, which did boot, but did not logged: Feb 15 03:19:30 [metalog] Child [1091] exited with return code 127. Feb 15 03:19:30 [metalog] Child [1092] exited with return code 127. Feb 15 03:20:50 [su] pam_unix(su:session): session closed for user man Feb 15 03:20:50 [metalog] Child [13244] exited with return code 127. Feb 15 10:15:35 [uptimed] moving up to position 40: 0 days, 10:56:01 Feb 15 10:15:35 [metalog] Child [6662] exited with return code 127. Feb 15 12:00:01 [start-stop-daemon] Will stop /usr/bin/lcd4linux Feb 15 12:00:01 [start-stop-daemon] Will stop PID 3593 Feb 15 12:00:01 [metalog] Child [32369] exited with return code 127. Feb 15 12:00:01 [metalog] Child [32370] exited with return code 127. Feb 15 12:00:01 [start-stop-daemon] Sending signal 15 to PID 3593 Feb 15 12:00:01 [metalog] Child [32371] exited with return code 127. Feb 15 12:00:01 [start-stop-daemon] Will stop /usr/bin/lcd4linux Feb 15 12:00:01 [metalog] Child [32458] exited with return code 127. So basically, I think, everyone with enabled consolelog.sh should be affected.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ca959af4ffe4122207405b15a039ea8d91652967 commit ca959af4ffe4122207405b15a039ea8d91652967 Author: Patrick McLean <chutzpah@gentoo.org> AuthorDate: 2020-02-15 18:00:05 +0000 Commit: Patrick McLean <chutzpah@gentoo.org> CommitDate: 2020-02-15 18:00:05 +0000 app-admin/metalog-20200113: Revbump, fix name of consolelog (bug 709726) Closes: https://bugs.gentoo.org/709726 Package-Manager: Portage-2.3.89, Repoman-2.3.20 Signed-off-by: Patrick McLean <chutzpah@gentoo.org> .../metalog/{metalog-20200113.ebuild => metalog-20200113-r1.ebuild} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Also reported to upstream.
