Whenever I try to start the lvm service (openrc) if fails with the following output. It should be noted though, that the lvm volumes are all there and lvm is working without any observable issues. # /etc/init.d/lvm start * Starting the Logical Volume Manager ... Found volume group "vgdata" using metadata type lvm2 Found volume group "vgsystem" using metadata type lvm2 Command failed with status code 5. 12 logical volume(s) in volume group "vgdata" now active 7 logical volume(s) in volume group "vgsystem" now active * Failed to start the Logical Volume Manager [ !! ] * ERROR: lvm failed to start It seems that the erroneous return code is issued by the vgscan command with --mknode parameter. When this parameter is removed, the issue is not present. # vgscan --config 'global { locking_dir = "/run/lock/lvm" }' --mknodes Found volume group "vgdata" using metadata type lvm2 Found volume group "vgsystem" using metadata type lvm2 Command failed with status code 5. A log with -vvvv is attached to the bug report. However, I cannot find any message about a failed node creation or a similar issue. The only thing special to that system seems to be that it runs a software raid 5, containing a luks container which then contains the physical volume of the volume group vgdata. However, the lvm command already fails at boot time. when the luks container is still closed. Thus, this contradicts this point. I found an old similar report here https://gitlab.alpinelinux.org/alpine/aports/-/issues/3543 but the issue is refereed to as fixed by this commit https://github.com/lvmteam/lvm2/commit/4dc602f79bd6579eef15a9227aee99fe832a7610 Reproducible: Always # emerge --info Portage 3.0.38.1 (python 3.10.8-final-0, default/linux/amd64/17.1/no-multilib/hardened, gcc-11.3.0, glibc-2.35-r8, 5.15.74-gentoo x86_64) ================================================================= System uname: Linux-5.15.74-gentoo-x86_64-Intel-R-_Core-TM-_i5-4570S_CPU_@_2.90GHz-with-glibc2.35 KiB Mem: 16253124 total, 2972612 free KiB Swap: 16777212 total, 16777212 free Timestamp of repository gentoo: Wed, 26 Oct 2022 07:31:49 +0000 Head commit of repository gentoo: 3e19a7d9285d53513ebf2a4282312c13b9fe00c3 sh bash 5.1_p16-r1 ld GNU ld (Gentoo 2.38 p4) 2.38 app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r1::gentoo dev-lang/perl: 5.34.1-r3::gentoo dev-lang/python: 3.9.15::gentoo, 3.10.8::gentoo dev-lang/rust: 1.64.0-r1::gentoo dev-util/cmake: 3.24.2::gentoo dev-util/meson: 0.63.2-r1::gentoo sys-apps/baselayout: 2.8::gentoo sys-apps/openrc: 0.45.2-r1::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.71-r1::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.38-r2::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/gcc: 11.3.0::gentoo sys-devel/gcc-config: 2.5-r1::gentoo sys-devel/libtool: 2.4.7::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.15-r3::gentoo (virtual/os-headers) sys-libs/glibc: 2.35-r8::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo.git priority: -1000 sync-git-verify-commit-signature: true shared_overlay location: /opt/conf/common/var/db/repos/shared_overlay masters: gentoo priority: 100 local_overlay location: /var/db/repos/local_overlay masters: gentoo priority: 200 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -ftree-vectorize -fforce-addr" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/portage/package.accept_keywords/99-autounmask /etc/portage/package.unmask/99-autounmask /etc/portage/package.use/99-autounmask /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe -ftree-vectorize -fforce-addr" DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask=y --autounmask-write --autounmask-continue --jobs=2 --load-average=4 --backtrack=100 --alert" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms sign splitdebug strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j4" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" RUSTFLAGS="-C target-cpu=native -O" SHELL="/bin/bash" USE="acl aes amd64 avx avx2 bash-completion bzip2 cli crypt cups dnssec dri f16c fma3 fortran gdbm glib hardened iconv icu idn ipv4 ipv6 libglvnd libtirpc lvm mmx mmxext ncurses nfsv4 nfsv41 nls nptl ntp openmp opensslcrypt pam pclmul pcre pie popcnt rdrand readline samba seccomp split-usr sse sse2 sse3 sse4_1 sse4_2 ssl ssp ssse3 syslog test-rust threads unicode vim-syntax xattr xtpax zlib" ABI_X86="64" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
Created attachment 825487 [details] output of vgscan -vvvv --config 'global { locking_dir = "/run/lock/lvm" }' --mknodes
I am currently using =sys-fs/lvm2-2.03.14-r3, but the issue was already present at version 2.02.188-r3.
# pvdisplay --- Physical volume --- PV Name /dev/mapper/crypt_raid VG Name vgdata PV Size 32.74 TiB / not usable 3.75 MiB Allocatable yes PE Size 4.00 MiB Total PE 8583098 Free PE 3347130 Allocated PE 5235968 PV UUID BordYg-Qpir-Ksm1-iOeK-v2hA-HQya-A4Hmug --- Physical volume --- PV Name /dev/sda4 VG Name vgsystem PV Size 414.78 GiB / not usable <2.82 MiB Allocatable yes PE Size 4.00 MiB Total PE 106184 Free PE 40009 Allocated PE 66175 PV UUID witdMP-cVc1-IycP-YHHB-ergZ-05c2-95zjhC # vgdisplay --- Volume group --- VG Name vgdata System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 14 VG Access read/write VG Status resizable MAX LV 0 Cur LV 12 Open LV 12 Max PV 0 Cur PV 1 Act PV 1 VG Size 32.74 TiB PE Size 4.00 MiB Total PE 8583098 Alloc PE / Size 5235968 / 19.97 TiB Free PE / Size 3347130 / <12.77 TiB VG UUID Ahewgi-Osnw-r7gf-a3SA-AzT7-URAj-XHmaMJ --- Volume group --- VG Name vgsystem System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 36 VG Access read/write VG Status resizable MAX LV 0 Cur LV 7 Open LV 6 Max PV 0 Cur PV 1 Act PV 1 VG Size 414.78 GiB PE Size 4.00 MiB Total PE 106184 Alloc PE / Size 66175 / <258.50 GiB Free PE / Size 40009 / <156.29 GiB VG UUID W7knUd-A0yQ-0BSn-wDHZ-D2O7-g4D8-tG58mD
I have the same symptoms here, temporary solution is removing the --mknodes option from the startup script. The system runs on md-raid1 PVs without initramfs. Interestingly, there is no startup script for lvmetad installed. Reproducible: Always emerge --info Portage 3.0.38.1 (python 3.10.8-final-0, default/linux/amd64/17.1, gcc-11.3.0, glibc-2.36-r5, 5.15.80-gentoo-dom0-Hyperjanus x86_64) ================================================================= System uname: Linux-5.15.80-gentoo-dom0-Hyperjanus-x86_64-AMD_Opteron-tm-_X3418_APU-with-glibc2.36 KiB Mem: 930720 total, 695632 free KiB Swap: 32804576 total, 32804576 free Timestamp of repository gentoo: Sat, 03 Dec 2022 08:30:01 +0000 Head commit of repository gentoo: c2ad6974286ce29bae310d0ff67c773cfe123acc sh bash 5.1_p16-r2 ld GNU ld (Gentoo 2.38 p4) 2.38 distcc 3.4 x86_64-pc-linux-gnu [disabled] ccache version 4.6.3 [disabled] app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r2::gentoo dev-lang/perl: 5.34.1-r4::gentoo dev-lang/python: 3.10.8_p3::gentoo, 3.11.0_p2::gentoo dev-util/ccache: 4.6.3::gentoo dev-util/cmake: 3.24.3::gentoo dev-util/meson: 0.63.3::gentoo sys-apps/baselayout: 2.9::gentoo sys-apps/openrc: 0.45.2-r1::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.71-r5::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.38-r2::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/gcc: 11.3.0::gentoo sys-devel/gcc-config: 2.8::gentoo sys-devel/libtool: 2.4.7::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.15-r3::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r5::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=athlon64 -fomit-frame-pointer -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=athlon64 -fomit-frame-pointer -pipe" DISTDIR="/var/cache/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en" MAKEOPTS="-j8" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="amd64 bzip2 cli crypt dri gdbm iconv libglvnd libtirpc minimal mmx multilib ncurses nptl openmp pam pcre readline seccomp split-usr sse sse2 ssl test-rust unicode xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LANG, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Interesting that you are also using mdraid. Since mdraid and lvm are both using the device mapper, my hypothesis is that mdraid is creating some device mapper nodes prior to lvm and that this causes lvm to fail when it tries to create the nodes itself. Maybe return code 5 is not an issue at all and should be simply ignored by the init script? However, i cannot find a documentation regarding the return code and what is exactly means or if it is also used for more severe situation. (lvmetad was deprecated and is gone since version 2.03. Thus, missing lvmetad is expected and should not be of concern)
I do not use LVM2 to manage the RAID setup, this is left to the boot time kernel auto-raid feature and the mdadm. But my LVM uses RAID devices /dev/md* as PVs. So I'm not longer sure, if the bug title relates to my situation, but the symptoms are the very same.
I have confused md and dm devices... My setup is also using md devices onthe bottom of the stack. More precisely mdraid -> luks -> PV.
