/etc/init.d/lvm fails when booting with a thinly provisioned logical volume attached. This is similar to bug #878353 but occurs when thinly provisioned logical volume is attached. After creating a thin provisioned lv, /etc/init.d/lvm fails. I don't need to be booting via a thinly provisioned logical volume, it just needs to be found during boot. I tried changing /etc/init.d/lvm so that it ignored the failure but then dmeventd failed to start, due to is already running. I used 'ps -ef | grep dmeventd' to find the errant daemon and using kill to remove it. The I was able to '/etc/init.d/dmeventd start' and '/etc/init.d/lvm-monitoring start' successfully. It apears that when thinly provisioned lv is present vgscan causes dmeventd to start. I can enabled the system to boot cleanly by: 1. Changing /etc/lvm/lvm.conf and adding 'executable = "/sbin/dmeventd-disabled" ' to the deventd section. 2. Changing /etc/init.d/lvm so that it ignored vgscan errors. When started with unchanged /etc/init.d/lvm and /etc/lvm/lvm.conf From /etc/rc.log , redacting noise. ......... * Setting system clock using the hardware clock [UTC] ... [ ok ] * Starting the Logical Volume Manager ... Found volume group "vg_ssd4_1" using metadata type lvm2 ..... Command failed with status code 5. 33 logical volume(s) in volume group "vg_ssd4_1" now active ....... * Failed to start the Logical Volume Manager [ !! ] * ERROR: lvm failed to start * Checking local filesystems ... .... After changing /etc/init.d/lvm to ignore the error. vg_ssd7_6 is the thinly provisioned logical volume ............... * Starting the Logical Volume Manager ... Found volume group "vg_ssd7_6" using metadata type lvm2 ......... Found volume group "syssda4" using metadata type lvm2 Command failed with status code 5. * vgscan completed with error 5 ...... * Starting dmeventd ... * start-stop-daemon: /sbin/dmeventd is already running * Failed to start dmeventd [ !! ] * ERROR: dmeventd failed to start ...... After changing /etc/init.d/lvm to ignore the error and disabling the dmeventd path in /etc/lvm/lvm.conf. vg_ssd7_6 is the thinly provisioned logical volume ................ * Starting the Logical Volume Manager ... Found volume group "vg_ssd7_6" using metadata type lvm2 Found volume group "sys_ssd5_5" using metadata type lvm2 ................ Found volume group "syssda4" using metadata type lvm2 Command failed with status code 5. * vgscan completed with error 5 ............... * Starting dmeventd ... [ ok ] ................ * Starting LVM monitoring for VGs ......... 51 logical volume(s) in volume group "vg_ssd7_6" monitored .......... [ ok ] .......... /var/log/messages now contains dmeventd[4883]: Monitoring thin pool vg_ssd7_6-vg_ssd7_6_pool-tpool. ......... dmeventd[4883]: WARNING: Thin pool vg_ssd7_6-vg_ssd7_6_pool-tpool data is now 85.28% full. ......... Reproducible: Always Steps to Reproduce: 1. Create a thinly provisioned logical volume 2. Reboot 3. Actual Results: /etc/init.d/lvm fails Expected Results: The system boot normally.
emerge --info Portage 3.0.41 (python 3.10.9-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-11, glibc-2.36-r5, 6.1.2-gentoo x86_64) ================================================================= System uname: Linux-6.1.2-gentoo-x86_64-AMD_Ryzen_9_3900X_12-Core_Processor-with-glibc2.36 KiB Mem: 65797596 total, 37028580 free KiB Swap: 58720236 total, 58720236 free Timestamp of repository gentoo: Tue, 03 Jan 2023 03:15:01 +0000 Head commit of repository gentoo: 820964b57265ac7e869419b0297a278c5fdde787 sh bash 5.1_p16-r2 ld GNU ld (Gentoo 2.39 p5) 2.39.0 distcc 3.4 x86_64-pc-linux-gnu [disabled] ccache version 4.7.4 [disabled] app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r2::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.36.0-r1::gentoo dev-lang/python: 3.10.9::gentoo, 3.11.1::gentoo dev-lang/rust: 1.65.0::gentoo dev-util/ccache: 4.7.4::gentoo dev-util/cmake: 3.24.3::gentoo dev-util/meson: 0.64.1::gentoo sys-apps/baselayout: 2.9::gentoo sys-apps/openrc: 0.45.2-r2::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.13-r7::gentoo, 2.71-r5::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.39-r4::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/clang: 15.0.6::gentoo sys-devel/gcc: 11.3.1_p20221209::gentoo sys-devel/gcc-config: 2.8::gentoo sys-devel/libtool: 2.4.7::gentoo sys-devel/lld: 15.0.6::gentoo sys-devel/llvm: 15.0.6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.15-r3::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r5::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://datastoreone/gentoo-portage priority: -1000 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 local location: /usr/local/portage masters: gentoo priority: 0 eclipse location: /var/lib/layman/eclipse masters: gentoo priority: 50 Installed sets: @kernels, @openwrt-prerequisites ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=amdfam10 -mtune=amdfam10 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.4/ext-active/ /etc/php/apache2-php8.1/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/fpm-php7.4/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/phpdbg-php7.4/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/vmware-installer" CXXFLAGS="-march=amdfam10 -mtune=amdfam10 -O2 -pipe" DISTDIR="/mnt/portage.autofs/distfiles/ddata" EMERGE_DEFAULT_OPTS="--with-bdeps=y -j52 --load-average=52 --quiet-build=y" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://www.mirrorservice.org/sites/distfiles.gentoo.org/ http://mirror.bytemark.co.uk/gentoo/ http://mirrors.gethosted.online/gentoo http://mirrors.soeasyto.com/distfiles.gentoo.org/ http://gentoo.mirrors.ovh.net/gentoo-distfiles/ http://ftp.free.fr/mirrors/ftp.gentoo.org/ https://mirrors.ircam.fr/pub/gentoo-distfiles/" LANG="en_GB.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" LINGUAS="en en_GB" MAKEOPTS="-j52 --load-average=52" PKGDIR="/mnt/portage.autofs/packages/pdata" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git --exclude=lost+found/" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="X a52 aac accessibility acl acpi activities aes alsa amarok amd64 amr authdaemond avx avx2 bash-completion bluetooth bluray branding bzip2 cairo caps cdda cdr cli crypt cups dbus declarative dri dts dv dvd dvdr dvdread elogind encode exif f16c fam flac fma3 fortran gdbm gif gpm gtk gui iconv icu ieee1394 ipv6 jpeg jumbo-build kde kwallet lcms libglvnd libnotify libtirpc lm-sensors lm_sensors lock lvm mad mmx mmxext mng mp3 mp4 mpeg multilib mysql ncurses network nls nptl ofx ogg opencl opengl openmp oss pam pango pclmul pcre pdf plasma png policykit popcnt ppds pulseaudio qml qt3 qt5 rdrand readline samba sasl sdl seccomp semantic-desktop session sha sound sound-server spell split-usr sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssse3 startup-notification svg test-rust thunar tiff truetype udev udisks uk_bleb uk_rt unicode upower usb v4l v4l2 vdpau vorbis widgets wxwidgets x264 xattr xcb xft xinerama xml xv xvid xvmc zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgi cgid slotmem_shm unixd access_compat authz_core socache_shmcb authn_core" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-32 efi-64 pc multiboot mount" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby27" SANE_BACKENDS="hp5590 hp net abaton agfafocus apple artec artec_eplus48u as6e avision bh canon canon630u canon_dr canon_pp cardscan coolscan coolscan2 coolscan3 dc210 dc240 dc25 dell1600n_net dmc epjitsu epson epson2 fujitsu genesys gt68xx hp3500 hp3900 hp4200 hp5400 hpljm1005 hpsj5s hs2p ibm kodak kvs1025 kvs20xx leo lexmark ma1509 magicolor matsushita microtek microtek2 mustek mustek_pp mustek_usb nec niash p5 pie pixma plustek plustek_pp pnm qcam ricoh rts8891 s9036 sceptre sharp sm3600 sm3840 snapscan sp15c st400 stv680 tamarack teco1 teco2 teco3 test u12 umax umax1220u umax_pp xerox_mfp" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
I can provide the full rc.logs if required. It apears that when thinly provisioned lv is present vgscan causes dmeventd to start. This prevents dmeventd and lvm-monitoring from starting.
Please show the enabled USE flags for sys-fs/lvm2.
Also, please try starting lvm with rc_verbose=yes in /etc/rc.conf or /etc/conf.d/lvm. That will cause the lvm init script to run the vgscan command with the "-v" option.
(In reply to Mike Gilbert from comment #3) > Please show the enabled USE flags for sys-fs/lvm2. equery u sys-fs/lvm2 [ Legend : U - final flag setting for installation] [ : I - package is installed with flag ] [ Colors : set, unset ] * Found these USE flags for sys-fs/lvm2-2.03.17-r1: U I + + lvm : Build all of LVM2 including daemons and tools like lvchange, not just the device-mapper library (for other packages to use). If your system uses LVM2 for managing disks and partitions, enable this flag. - - lvm2create-initrd : Install lvm2create_initrd script and pull in sys-apps/makedev for the /sbin/MAKEDEV command + + readline : Enable support for libreadline, a GNU line-editing library that almost everyone wants - - sanlock : Enable lvmlockd with support for sanlock - - static : !!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically - - static-libs : Build static versions of dynamic libraries as well - - systemd : Enable use of systemd-specific libraries and features like socket activation or session tracking + + thin : Support for thin volumes + + udev : Enable virtual/udev integration (device discovery, power and storage device support, etc)
(In reply to Mike Gilbert from comment #4) > Also, please try starting lvm with rc_verbose=yes in /etc/rc.conf or > /etc/conf.d/lvm. That will cause the lvm init script to run the vgscan > command with the "-v" option. I tried changing the vgscan call in /etc/init.d/lvm to ${lvm_path} vgscan${verbose_command} --config "${config}" --mknodes -vvvv -dddddd 2> /root/vgscan-mknodes-vvvv-dddddd-boot.log but I could find no errors reported and resulting log file is huge. 'lvs | wc -l' displays 279 on this box. I will try and recreate this in an environment with less lvs present.
Created attachment 847044 [details] log of vgscan --mknodes -vvvv -dddddd vgscan log of the issue replicated in a virtualbox vm with only 31 lvs. /etc/init.d/lvm modified, changing the vgscan call to: ${lvm_path} vgscan${verbose_command} --config "${config}" --mknodes -vvvv -dddddd 2> /root/vgscan-mknodes-vvvv-dddddd-boot.log Two volume labels and lv names have been redacted, replacing them with redacted1 and redacted2.
As far as I can tell the log shows no errors occurring.
The pvs, vgs and lvs were created: pvcreate $DEVICE vgcreate -s 32M $VG $DEVICE lvcreate -l 100%free --type thin-pool --thinpool "${THIN_POOL}" $VG LV_PREFIX="sys1." lvcreate --wipesignatures y --virtualsize 4G -n ${LV_PREFIX}slash --thinpool $VG/$THIN_POOL .......etc
All that is needed to provoke this issue is to add a thin pool in the vg. i.e. the error still occurs after all thin lvs are removed, so the issue still when vg only contains: lvs vg_ssd7_6 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert vg_ssd7_6_pool vg_ssd7_6 twi-aotz-- 59.84g 0.00 10.42
yomo-db2 [16:12:10] ~ # /etc/init.d/lvm start * Caching service dependencies ... [ ok ] * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/lvm start * Starting the Logical Volume Manager ... Reloading config files Found volume group "lvm" using metadata type lvm2 Reloading config files Command failed with status code 5. Reloading config files 11 logical volume(s) in volume group "lvm" already active Activating logical volume lvm/thin. activation/volume_list configuration setting not defined: Checking only host tags for lvm/thin. Activating logical volume lvm/tmp. activation/volume_list configuration setting not defined: Checking only host tags for lvm/tmp. Activating logical volume lvm/usr. activation/volume_list configuration setting not defined: Checking only host tags for lvm/usr. Activating logical volume lvm/opt. activation/volume_list configuration setting not defined: Checking only host tags for lvm/opt. Activating logical volume lvm/var. activation/volume_list configuration setting not defined: Checking only host tags for lvm/var. Activating logical volume lvm/var_spool. activation/volume_list configuration setting not defined: Checking only host tags for lvm/var_spool. Activating logical volume lvm/home. activation/volume_list configuration setting not defined: Checking only host tags for lvm/home. Activating logical volume lvm/swap. activation/volume_list configuration setting not defined: Checking only host tags for lvm/swap. Activating logical volume lvm/uls_report. activation/volume_list configuration setting not defined: Checking only host tags for lvm/uls_report. Activating logical volume lvm/firmware. activation/volume_list configuration setting not defined: Checking only host tags for lvm/firmware. Activating logical volume lvm/mysql. activation/volume_list configuration setting not defined: Checking only host tags for lvm/mysql. Activated 11 logical volumes in volume group lvm. 11 logical volume(s) in volume group "lvm" now active Reloading config files * Failed to start the Logical Volume Manager [ !! ] * ERROR: lvm failed to start A bit of debugging shows vgscan terminating with exit code 5. vgscan -vvvvv -mknodes states: 16:19:42.531534 vgscan[26257] lvmcmdline.c:3352 Completed: vgscan -vvvv --mknodes 16:19:42.531558 vgscan[26257] lvmcmdline.c:3855 Internal error: Failed command did not use log_error 16:19:42.531567 vgscan[26257] lvmcmdline.c:3856 Command failed with status code 5. https://listman.redhat.com/archives/lvm-devel/2009-July/msg00058.html Implies that whatever is causing that failure code to be returned isn't actually internally calling log_error, thus the error isn't being logged. I thought this may be due to: 16:27:02.352153 vgscan[3063] device_mapper/libdm-config.c:1085 global/notify_dbus not found in config: defaulting to 1 However adding --config "global/notify_dbus=0" to the command gets rid of that but doesn't solve the issue. This is somewhat problematic. DIAGNOSTICS All tools return a status code of zero on success or non-zero on failure. The non-zero codes distinguish only between the broad categories of unrecognised commands, problems processing the command line arguments and any other failures. As LVM remains under active development, the code used in a specific case occasionally changes between releases. Mes‐ sage text may also change. That's not of much help either. Marking issue as confirmed, but will likely need upstream fix. Can we perhaps at least ignore return code 5 for vgscan in the meantime in the init script?
This is the same issue as bug 878353, so I'm going to dupe this to that one. See the comment I just added there for more information. *** This bug has been marked as a duplicate of bug 878353 ***
