Summary: | sys-apps/openrc-0.48 is spinning cpu when stopping smbd process with >=sys-kernel/gentoo-sources-6.6 | ||
---|---|---|---|
Product: | Gentoo Hosted Projects | Reporter: | Marcin Mirosław <bug> |
Component: | OpenRC | Assignee: | OpenRC Team <openrc> |
Status: | RESOLVED FIXED | ||
Severity: | critical | CC: | 85nrdzgv, digitalaudiorock, gentoo3, hydrapolic, ian, kernel, sam |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: |
https://github.com/OpenRC/openrc/pull/668 https://bugzilla.kernel.org/show_bug.cgi?id=217681 https://github.com/landley/toybox/issues/306 https://github.com/landley/toybox/issues/448 https://bugs.gentoo.org/show_bug.cgi?id=920816 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- |
Description
Marcin Mirosław
2023-11-05 19:20:30 UTC
Could you attach gdb to start-stop-daemon and get a backtrace too? I have the same issue as well, also only with samba. Haven't had to time to dig into it. openrc v0.51 (newest as of writing) affecting two seperate systems. I have the same issue after upgrading to >=sys-kernel/gentoo-sources-6.6.0. It does not reproduce when I booted with <sys-kernel/gentoo-sources-6.6.0. It does not reproduce when I run `start-stop-daemon --stop --exec /usr/sbin/smbd` directly in a shell. This is the output of step execution by gdb. $ pgrep start-stop 29563 $ sudo gdb --pid=29563 GNU gdb (Gentoo 13.2 vanilla) 13.2 : Attaching to process 29563 Reading symbols from /sbin/start-stop-daemon... Reading symbols from /usr/lib/debug//sbin/start-stop-daemon.debug... Reading symbols from /lib64/libeinfo.so.1... Reading symbols from /usr/lib/debug//lib64/libeinfo.so.1.debug... Reading symbols from /lib64/librc.so.1... Reading symbols from /usr/lib/debug//lib64/librc.so.1.debug... Reading symbols from /lib64/libpam.so.0... (No debugging symbols found in /lib64/libpam.so.0) Reading symbols from /lib64/libcap.so.2... (No debugging symbols found in /lib64/libcap.so.2) Reading symbols from /lib64/libc.so.6... (No debugging symbols found in /lib64/libc.so.6) Reading symbols from /lib64/libtinfo.so.6... (No debugging symbols found in /lib64/libtinfo.so.6) Reading symbols from /lib64/ld-linux-x86-64.so.2... (No debugging symbols found in /lib64/ld-linux-x86-64.so.2) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007fad166b4647 in rename () from /lib64/libc.so.6 (gdb) n Single stepping until exit from function rename, which has no line number information. rc_service_daemon_set (service=service@entry=0x7ffd86f4ce7f "samba", exec=exec@entry=0x7ffd86f4c8a7 "/usr/sbin/smbd", argv=argv@entry=0x7ffd86f4aae0, pidfile=pidfile@entry=0x0, started=started@entry=false) at ../openrc-0.51/src/librc/librc-daemon.c:434 434 strlcpy(oldfile, file, sizeof(oldfile)); (gdb) 436 free(file); (gdb) 419 while ((d = readdir(dp))) { (gdb) 420 if (d->d_name[0] == '.') (gdb) 423 xasprintf(&file, "%s/%s", dirpath, d->d_name); (gdb) 424 nfiles++; (gdb) 426 if (!*oldfile) { (gdb) 433 rename(file, oldfile); (gdb) 434 strlcpy(oldfile, file, sizeof(oldfile)); (gdb) 436 free(file); (gdb) 419 while ((d = readdir(dp))) { (gdb) 420 if (d->d_name[0] == '.') : (repeat) I also have exactly this issue.
<<<<<
I have the same issue after upgrading to >=sys-kernel/gentoo-sources-6.6.0.
It does not reproduce when I booted with <sys-kernel/gentoo-sources-6.6.0.
It does not reproduce when I run `start-stop-daemon --stop --exec /usr/sbin/smbd` directly in a shell.
>>>>>
After /etc/init.d/samba start
ls -l /run/openrc/daemons/samba/
total 8
-rw-r--r-- 1 root root 61 Nov 10 11:39 001
-rw-r--r-- 1 root root 61 Nov 10 11:39 002
After /etc/init.d/samba stop
ls -l /run/openrc/daemons/samba/ | wc -l
270
Repeating this always shows a different number of line (files) ranging from 200'ish to 1263
Also displayed are many repeats of
ls: cannot access '/run/openrc/daemons/samba/002': No such file or directory
ls: /run/openrc/daemons/samba/001: No such file or directory
ls: /run/openrc/daemons/samba/002: No such file or directory
ls: cannot access '/run/openrc/daemons/samba/001': No such file or directory
.....................
290
I can take a quick look later. Just in case it's relevant emerge --info Portage 3.0.51 (python 3.11.5-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-13, glibc-2.37-r7, 6.6.1-gentoo x86_64) ================================================================= System uname: Linux-6.6.1-gentoo-x86_64-AMD_Ryzen_5_5600X_6-Core_Processor-with-glibc2.37 KiB Mem: 32809340 total, 30903972 free KiB Swap: 33554424 total, 33554424 free Timestamp of repository gentoo: Thu, 09 Nov 2023 02:45:01 +0000 Head commit of repository gentoo: b7bc44f3b35169406f0c54f86fd45c8f5d02deeb sh bash 5.1_p16-r6 ld GNU ld (Gentoo 2.40 p5) 2.40.0 distcc 3.4 x86_64-pc-linux-gnu [enabled] ccache version 4.8.2 [disabled] app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r6::gentoo dev-java/java-config: 2.3.1-r1::gentoo dev-lang/perl: 5.38.0-r1::gentoo dev-lang/python: 3.10.13::gentoo, 3.11.5::gentoo dev-lang/rust: 1.71.1::gentoo dev-util/ccache: 4.8.2::gentoo dev-util/cmake: 3.26.5-r2::gentoo dev-util/meson: 1.2.1-r1::gentoo sys-apps/baselayout: 2.14::gentoo sys-apps/openrc: 0.48::gentoo sys-apps/sandbox: 2.38::gentoo sys-devel/autoconf: 2.13-r7::gentoo, 2.71-r6::gentoo sys-devel/automake: 1.16.5-r1::gentoo sys-devel/binutils: 2.40-r5::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/clang: 16.0.6::gentoo sys-devel/gcc: 13.2.1_p20230826::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/lld: 16.0.6::gentoo sys-devel/llvm: 16.0.6::gentoo sys-devel/make: 4.4.1-r1::gentoo sys-kernel/linux-headers: 6.1::gentoo (virtual/os-headers) sys-libs/glibc: 2.37-r7::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://datastoreone/gentoo-portage priority: -1000 volatile: True sync-rsync-extra-opts: sync-rsync-verify-jobs: 1 sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes local location: /usr/local/portage masters: gentoo priority: 0 volatile: True Installed sets: @kernels ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=amdfam10 -mtune=amdfam10 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.1/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/vmware-installer" CXXFLAGS="-march=amdfam10 -mtune=amdfam10 -O2 -pipe" DISTDIR="/mnt/portage.autofs/distfiles/ddata" EMERGE_DEFAULT_OPTS="--with-bdeps=y -j20 --load-average=18 --quiet-build=y" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg buildpkg-live config-protect-if-modified distcc distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://mirrors.linuxant.fr/distfiles.gentoo.org/ http://www.mirrorservice.org/sites/www.ibiblio.org/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo ftp://ftp.free.fr/mirrors/ftp.gentoo.org/ http://mirror.leaseweb.com/gentoo/ http://gentoo.modulix.net/gentoo/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ ftp://mirror.bytemark.co.uk/gentoo/" LANG="en_GB.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" LINGUAS="en en_GB" MAKEOPTS="-j20 --load-average=18" PKGDIR="/mnt/portage.autofs/packages/pdata" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git --exclude=lost+found/" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="X a52 aac acl acpi activities aes alsa amarok amd64 amr authdaemond avx avx2 bash-completion bluetooth bluray branding bzip2 cairo caps cdda cdr cli crypt cups dbus declarative dri dts dv dvd dvdr dvdread elogind encode exif f16c fam flac fma3 fortran gdbm gif gpm gtk gui iconv icu ipv6 jpeg kde kwallet lcms libnotify libtirpc lm_sensors lock lvm mad mmx mmxext mng mp3 mp4 mpeg multilib mysql ncurses network networkmanager nls nptl ofx ogg opengl openmp oss pam pango pclmul pcre pdf pipewire plasma png policykit popcnt ppds pulseaudio qml qt3 qt5 rdrand readline samba sasl screencast sdl seccomp semantic-desktop session sha sound sound-server spell split-usr sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssse3 startup-notification svg test-rust thunar tiff truetype udev udisks uk_bleb uk_rt unicode upower usb v4l v4l2 vdpau vorbis vulkan wayland widgets wxwidgets x264 xattr xcb xft xinerama xml xv xvid xvmc zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgi cgid slotmem_shm unixd access_compat authz_core socache_shmcb authn_core" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-32 efi-64 pc multiboot mount" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby31" SANE_BACKENDS="hp5590 hp net abaton agfafocus apple artec artec_eplus48u as6e avision bh canon canon630u canon_dr canon_pp cardscan coolscan coolscan2 coolscan3 dc210 dc240 dc25 dell1600n_net dmc epjitsu epson epson2 fujitsu genesys gt68xx hp3500 hp3900 hp4200 hp5400 hpljm1005 hpsj5s hs2p ibm kodak kvs1025 kvs20xx leo lexmark ma1509 magicolor matsushita microtek microtek2 mustek mustek_pp mustek_usb nec niash p5 pie pixma plustek plustek_pp pnm qcam ricoh rts8891 s9036 sceptre sharp sm3600 sm3840 snapscan sp15c st400 stv680 tamarack teco1 teco2 teco3 test u12 umax umax1220u umax_pp xerox_mfp" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS Interestingly, after the following sequence '/etc/init.d/samba stop' has hung and
needs aborting with Ctrl/C, even though smbd & nmbd have already been stopped manually.
<<<<<
# /etc/init.d/samba start
* samba -> start: smbd ... [ ok ]
* samba -> start: nmbd ... [ ok ]
# start-stop-daemon --stop --exec /usr/sbin/smbd
# start-stop-daemon --stop --exec /usr/sbin/nmbd
# ps -ef | egrep "smbd|nmbd"
root 7697 7517 0 20:30 pts/11 00:00:00 /bin/grep -E smbd|nmbd
# /etc/init.d/samba stop
* samba -> stop: smbd ...
* start-stop-daemon: no matching processes found
^C * start-stop-daemon: caught SIGINT, aborting
* samba: caught SIGINT, aborting [ !! ]
* samba -> stop: nmbd ...
* start-stop-daemon: no matching processes found
>>>>
Also reproduced in sys-kernel/gentoo-kernel-6.6.1 without `savedconfig`. (to confirm if the issue was caused by my kernel config.) The plot thickens. While samba is running normally # ps -ef | grep -E "smbd|nmbd" root 7766 1 0 09:52 ? 00:00:00 /usr/sbin/smbd -D root 7771 7766 0 09:52 ? 00:00:00 smbd: notifyd . root 7772 7766 0 09:52 ? 00:00:00 smbd: cleanupd root 7774 1 0 09:52 ? 00:00:00 /usr/sbin/nmbd -D root 7794 7789 0 09:53 pts/13 00:00:00 grep --colour=auto -E smbd|nmbd After /etc/init.d/samba stop # ps -ef | grep -E "smbd|nmbd" root 7774 1 0 09:52 ? 00:00:00 /usr/sbin/nmbd -D root 7811 7796 86 09:53 pts/12 00:00:03 start-stop-daemon --stop --exec /usr/sbin/smbd root 7815 7789 0 09:53 pts/13 00:00:00 grep --colour=auto -E smbd|nmbd Note that smbd has been successfully stopped by the start-stop-daemon but it is still trying to delete the 'daemon info' but none of the files in /run/openrc/daemons/samba match smbd cat /run/openrc/daemons/samba/001 exec=/usr/sbin/nmbd argv_0=/usr/sbin/nmbd argv_1=-D pidfile= # cat /run/openrc/daemons/samba/002 exec=/usr/sbin/nmbd argv_0=/usr/sbin/nmbd argv_1=-D pidfile= Interestingly both files have the same inode number! In fact 'ls -li /run/openrc/daemons/samba/' reports the same inode number for all the files it finds. I see the same problem with net-misc/xrdp as well. Running a test program doing something very similar to `rc_service_daemon_set` (scan directory, rename file) on tmpfs (this part is important; /run is tmpfs) gives the same infinite loop. Basically `rename` triggers re-reading new renamed file from `readdir` and calling rename again, infinitely. The difference from other daemons appears to be this: $ ls /run/openrc/daemons/sshd/ 001 $ ls /run/openrc/daemons/samba/ 001 002 On my system, only samba and xrdp have two files in daemon directory. So for other daemons openrc doesn't attempt this rename. I've created https://github.com/OpenRC/openrc/pull/668 that should fix that problem. Could you please test it with that patch: diff --git a/src/librc/librc-daemon.c b/src/librc/librc-daemon.c index 50245129..e23593b9 100644 --- a/src/librc/librc-daemon.c +++ b/src/librc/librc-daemon.c @@ -402,7 +402,7 @@ rc_service_daemon_set(const char *service, const char *exec, bool retval = false; DIR *dp; struct dirent *d; - RC_STRINGLIST *match; + RC_STRINGLIST *match, *renamelist; int i = 0; FILE *fp; @@ -416,11 +416,17 @@ rc_service_daemon_set(const char *service, const char *exec, /* Regardless, erase any existing daemon info */ if ((dp = opendir(dirpath))) { match = _match_list(exec, argv, pidfile); + renamelist = rc_stringlist_new(); while ((d = readdir(dp))) { if (d->d_name[0] == '.') continue; xasprintf(&file, "%s/%s", dirpath, d->d_name); + if (rc_stringlist_find(renamelist, file)) { + free(file); + continue; + } + nfiles++; if (!*oldfile) { @@ -432,11 +438,15 @@ rc_service_daemon_set(const char *service, const char *exec, } else { rename(file, oldfile); strlcpy(oldfile, file, sizeof(oldfile)); + /* Add renamed file to renamelist, as this new file name could + * be read again from readdir() */ + rc_stringlist_add(renamelist, oldfile); } free(file); } closedir(dp); rc_stringlist_free(match); + rc_stringlist_free(renamelist); } /* Now store our daemon info */ The patch works fine and start-stop-daemon stops smbd successfully. Thank you! Same for me, patch fixes issues with samba & kea dhcp Patch worked for me on sys-apps/openrc-0.48 with kernel 6.6.1. just ftr/completeness, the relevant change I think is: commit a229cf67ab851a6e92395f37ed141d065176575a Merge: 5d2f53532ecc 8e7f82deb0c0 Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Wed Sep 20 11:03:45 2023 -0700 Merge tag 'for-6.6-rc2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux Pull btrfs fixes from David Sterba: "A few more followup fixes to the directory listing. People have noticed different behaviour compared to other filesystems after changes in 6.5. This is now unified to more "logical" and expected behaviour while still within POSIX. And a few more fixes for stable. - change behaviour of readdir()/rewinddir() when new directory entries are created after opendir(), properly tracking the last entry - fix race in readdir when multiple threads can set the last entry index for a directory [...] and https://lore.kernel.org/all/2c8c55ec-04c6-e0dc-9c5c-8c7924778c35@landley.net/. Patch worked for me on sys-apps/openrc-0.51 with kernel 6.6.1. This will be included in 0.52. *** Bug 924178 has been marked as a duplicate of this bug. *** |