After updating to sysklogd-2.0 I started wondering why my logs went silent, and it turned out that the daemon was never started in the first place. Debugging with strace and -F shows that - contrary to what the syslogd man page claims - the default compiled-in pid file path is not /var/run (typically symlinked to /run) but rather /var/lib/run, which does not exist, thus making syslogd fail silently. Reproducible: Always Steps to Reproduce: 1. update to sysklogd-2.0 with working config 2. restart sysklogd 3. syslogd has silently gone away Actual Results: No syslogd running Expected Results: Running syslogd The problem can be fixed by explicitly adding the pid file path to the SYSLOGD args in /etc/conf.d/sysklogd: "-P /run/syslogd.pid" (or /var/run to be in line with the start-stop-daemon args). This immediately fixes the problem and returns the whole setup to a working state. I suspect that the discrepancy between compiled-in path and man page is an upstream and/or platform problem (Linux vs. BSD and such).
To wit: $strace syslogd -F execve("/usr/sbin/syslogd", ["syslogd", "-F"], 0x7ffc0c939888 /* 51 vars */) = 0 ..snip.. getpid() = 14465 openat(AT_FDCWD, "/var/lib/run/syslogd.pid", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 ENOENT (No such file or directory) getpid() = 14465 exit_group(1) = ? +++ exited with 1 +++ Maybe the easiest fix is to --configure a proper default path.
Are you using systemd or openrc to start sysklogd? Please post the output of emerge --info sysklogd to this bug.
Using openrc. $emerge --info sysklogd Portage 2.3.79 (python 3.7.5-final-0, default/linux/amd64/17.1/desktop, gcc-9.2.0, glibc-2.30-r2, 5.3.13 x86_64) ================================================================= System Settings ================================================================= System uname: Linux-5.3.13-x86_64-Intel-R-_Core-TM-_i7-2600K_CPU_@_3.40GHz-with-gentoo-2.6 KiB Mem: 24620020 total, 10345716 free KiB Swap: 0 total, 0 free Head commit of repository gentoo: eec0b46b4bb3ebbf484378c6cca4f18573e34f75 sh bash 5.0_p11 ld GNU ld (Gentoo 2.33.1 p1) 2.33.1 ccache version 3.7.6 [disabled] app-shells/bash: 5.0_p11::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.17::gentoo, 3.7.5-r1::gentoo dev-util/ccache: 3.7.6::gentoo dev-util/cmake: 3.15.5::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.18::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.16.1-r2::gentoo sys-devel/binutils: 2.33.1::gentoo sys-devel/gcc: 9.2.0-r2::gentoo sys-devel/gcc-config: 2.1::gentoo sys-devel/libtool: 2.4.6-r5::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.3-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r2::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: git://anongit.gentoo.org/repo/gentoo.git priority: -1000 sync-git-pull-extra-opts: --quiet hhoffstaette location: /home/holger/Projects/portage masters: gentoo priority: 100 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-pipe -O2 -march=native" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-3.6/conf" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-pipe -O2 -march=native" DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS="--deep --keep-going --nospinner --quiet-build --with-bdeps=y" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news noinfo parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS=" http://mirror.eu.oneandone.net/linux/distributions/gentoo/gentoo/ http://mirror.netcologne.de/gentoo/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.fau.de/gentoo http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/" INSTALL_MASK="/usr/share/gtk-doc /usr/share/locale" LDFLAGS="-Wl,-O1,--as-needed,-z,now" LINGUAS="en" MAKEOPTS="-j8" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/tmp" USE="X a52 aac acl acpi aio alsa amd64 branding btrfs bzip2 cacert cairo caps cdda cddb cdr cli crypt cups cxx dbus dri drm dts dvd dvdr elogind emboss encode epoll exif fam flac geoip gif gold gpg graphicsmagick gtk highlight hipe http2 hwloc iconv icu ipv6 jit jitterentropy jpeg kpoll lcms ldap libdrm libev libevent libnotify libtirpc logrotate lz4 lzma mad markdown maxminddb minizip mng mp3 mp4 mpeg multilib native-extensions natspec ncurses netlink network nptl nscd nss ntfs objc objc++ offensive ogg ogg123 opengl openmp openssl opus pam pango pcre pcre16 pdf png policykit postproc postscript ppds pulseaudio qt5 rar raw readline realtime rocksdb seccomp smi smp split-usr ssl startup-notification svg system-llvm tcpd threads tiff truetype udev udisks unicode upower urandom usb uuid vdpau vorbis vpx webp wxwidgets x264 xattr xcb xml xmp xv xvid xvmc zlib zsh-completion zstd" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="apcups cgroups chrony contextswitch cpu cpufreq df disk entropy fhcount interface irq load memory nfs smart swap tcpconns thermal uptime vmem write_prometheus" CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" L10N="en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python3_7" RUBY_TARGETS="ruby26" USERLAND="GNU" VIDEO_CARDS="radeon r600" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, LANG, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS ================================================================= Package Settings ================================================================= app-admin/sysklogd-2.0::gentoo was built with the following: USE="logrotate -klogd -logger -systemd" ABI_X86="(64)"
Did you run etc-update (or dispatch-conf) after installing sysklogd-2.0? It comes with a new openrc init script because sysklogd-2.0 doesn't work with the old init script.
(In reply to Lars Wendler (Polynomial-C) from comment #4) > Did you run etc-update (or dispatch-conf) after installing sysklogd-2.0? It > comes with a new openrc init script because sysklogd-2.0 doesn't work with > the old init script. Yes, sure. The new init script clearly has references to the changes in 2.0. Also if the installation was messed up it wouldn't work by adding the -P arg, to make pid-path expectations of init script and binary consistent. $diff /etc/init.d/sysklogd /var/db/repos/gentoo/app-admin/sysklogd/files/sysklogd.rc8 $echo $? 0 So no diff, latest version.
Liek I suspected there's something wrong with non-systemd configuration and/or platform directories. When running the ebuild compile phase manually: ------------------ Summary ------------------ sysklogd version 2.0 Prefix.........: /usr Sysconfdir.....: /etc Localstatedir..: /var/lib ^^^^^^^^^^^^^^^^^^^^^^^^^ The derived runstatedir is based on this and that's where it goes wrong, since the openrc init script assumes /var/run and the problem can be fixed by explicitly passing a matching value to the binary, or passing --runstatedir=/run to configure.
Ah, I see where the /var/lib path comes from. Now I just need to find out, why this breaks your sysklogd but not mine. Can you please attach the /etc/conf.d/sysklogd file from the affected system to this bug?
(In reply to Lars Wendler (Polynomial-C) from comment #7) > Ah, I see where the /var/lib path comes from. Now I just need to find out, > why this breaks your sysklogd but not mine. You probably have systemd installed, which during configuration may resolve localstatedir & friends differently; I don't, so configure hiccups and uses some odd builtin default. > Can you please attach the /etc/conf.d/sysklogd file from the affected system > to this bug? What does that have to do with wrong built-in defaults? I updated the config to be in line with the new version, removed log rotation (since I use logrotate) and only added the explicit pid path to make it work: $diff /var/db/repos/gentoo/app-admin/sysklogd/files/sysklogd.confd2 /etc/conf.d/sysklogd 3c3 < SYSLOGD="-m 0 -s -s -r 10M:10" --- > SYSLOGD="-m 0 -s -s -P /run/syslogd.pid"
(In reply to Holger Hoffstätte from comment #8) > (In reply to Lars Wendler (Polynomial-C) from comment #7) > > Ah, I see where the /var/lib path comes from. Now I just need to find out, > > why this breaks your sysklogd but not mine. > > You probably have systemd installed, which during configuration may resolve > localstatedir & friends differently; I don't, so configure hiccups and > uses some odd builtin default. No, I use openrc on all of my systems. > > Can you please attach the /etc/conf.d/sysklogd file from the affected system > > to this bug? > > What does that have to do with wrong built-in defaults? > I updated the config to be in line with the new version, removed log rotation > (since I use logrotate) and only added the explicit pid path to make it work: > > $diff /var/db/repos/gentoo/app-admin/sysklogd/files/sysklogd.confd2 > /etc/conf.d/sysklogd > 3c3 > < SYSLOGD="-m 0 -s -s -r 10M:10" > --- > > SYSLOGD="-m 0 -s -s -P /run/syslogd.pid" Thanks. I wanted to know the possible differences between your and my configuration because I have no problem running syslogd without the -P option. Anyway, I don't think upstream did us a great service by abusing the --localstatedir option for the pid file location. I will come up with a fix shortly.
(In reply to Lars Wendler (Polynomial-C) from comment #9) > Thanks. I wanted to know the possible differences between your and my > configuration because I have no problem running syslogd without the -P > option. Interesting. What happens when you run "strace syslogd -F" as root - where does it try to write the pid file? Apparently *something* is different somehwere. > Anyway, I don't think upstream did us a great service by abusing the > --localstatedir option for the pid file location. I will come up with a fix > shortly. Thanks :)
Created attachment 597416 [details] config.log from manually running ebuild configure In case it helps here's my config.log. I haven't done anything too weird to my systems and the same symptom happens on both my server and my desktop (with different profiles).
(In reply to Holger Hoffstätte from comment #10) > (In reply to Lars Wendler (Polynomial-C) from comment #9) > > Thanks. I wanted to know the possible differences between your and my > > configuration because I have no problem running syslogd without the -P > > option. > > Interesting. What happens when you run "strace syslogd -F" as root - where > does it try to write the pid file? Apparently *something* is different > somehwere. It successfully tries to write the pid file to /var/lib/run/syslogd.pid and then logs events. Most likely because I have a /var/lib/run directory. But that should not matter because openrc's start-stop-daemon creates its own pid file in /var/run/syslog.pid and that's the one the openrc init script is using. > > Anyway, I don't think upstream did us a great service by abusing the > > --localstatedir option for the pid file location. I will come up with a fix > > shortly. > > Thanks :) This is a really annoying flaw. I am curious what upstream says to this behavior.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0d3159ce3a0ba9d0d7d6daaabd4da6052fbc5784 commit 0d3159ce3a0ba9d0d7d6daaabd4da6052fbc5784 Author: Lars Wendler <polynomial-c@gentoo.org> AuthorDate: 2019-11-24 13:25:02 +0000 Commit: Lars Wendler <polynomial-c@gentoo.org> CommitDate: 2019-11-24 13:25:33 +0000 app-admin/sysklogd: Revbump to set correct localstatedir path This is required to not make syslogd bail out on startup if /var/lib/run directory does not exist. Reported-by: Holger Hoffstätte <holger@applied-asynchrony.com> Closes: https://bugs.gentoo.org/701048 Package-Manager: Portage-2.3.79, Repoman-2.3.18 Signed-off-by: Lars Wendler <polynomial-c@gentoo.org> app-admin/sysklogd/files/sysklogd.rc8 | 2 +- app-admin/sysklogd/{sysklogd-2.0.ebuild => sysklogd-2.0-r1.ebuild} | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)
