At boot, in the initrd phase, the LVM partition device nodes are not being created, resulting in /sysroot/usr failing to mount. When I run /sbin/lvm_scan, after I am being dropped to a rescue shell, the device nodes are being created, I can mount /sysroot/usr and continue the boot process. I am using sys-kernel/dracut-022-r5 and also tried sys-kernel/dracut-023 with the same result. Udev is at sys-fs/udev-171-r6 and the Kernel is sys-kernel/hardened-sources-2.6.32-r116. Reproducible: Always Portage 2.2.0_alpha121 (hardened/linux/ia64/server, gcc-4.5.3, glibc-2.14.1-r3, 2.6.32-hardened-r116 ia64) ================================================================= System uname: Linux-2.6.32-hardened-r116-ia64-31-with-gentoo-2.1 Timestamp of tree: Fri, 24 Aug 2012 08:15:01 +0000 distcc 3.1 ia64-unknown-linux-gnu [enabled] app-shells/bash: 4.2_p20 dev-lang/python: 2.7.3-r2, 3.2.3-r1 dev-util/cmake: 2.8.7-r5 dev-util/pkgconfig: 0.26 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.9.8.4 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.68 sys-devel/automake: 1.11.6 sys-devel/binutils: 2.21.1-r1 sys-devel/gcc: 4.5.3-r2 sys-devel/gcc-config: 1.6 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r1 sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers) sys-libs/glibc: 2.14.1-r3 Repositories: gentoo local sunrise ACCEPT_KEYWORDS="ia64" ACCEPT_LICENSE="* -@EULA" CBUILD="ia64-unknown-linux-gnu" CFLAGS="-pipe -mtune=mckinley -O2" CHOST="ia64-unknown-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi- php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sand box.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/tex mf/updmap.d /etc/texmf/web2c" CXXFLAGS="-pipe -mtune=mckinley -O2" DISTDIR="/var/cache/portage/distfiles" EMERGE_DEFAULT_OPTS="--usepkg --buildpkg --binpkg-respect-use --with-bdeps y --k eep-going" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distcc distcc-pump distlocks ebuild-locks fixlafiles news parallel-fetch parallel-insta ll parse-eapi-ebuild-head preserve-libs protect-owned sandbox sfperms strict unk nown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox u sersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/ http://ftp-st ud.fht-esslingen.de/pub/Mirrors/gentoo/ http://ftp.spline.inf.fu-berlin.de/mirro rs/gentoo/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ http://distfiles .gentoo.org" LANG="en_GB.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -Wl,--hash-style=gnu" MAKEOPTS="-j6" PKGDIR="/var/cache/portage/packages" PORTAGE_COMPRESS="xz" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/ distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/var/cache/portage/gentoo" PORTDIR_OVERLAY="/var/cache/portage/local /var/cache/portage/overlays/sunrise" [\342\200\246] Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAN D, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Please append rd.debug to kernel cmdline and attach the log which you should find in /run/initramfs. Please also paste the content of /etc/cmdline.d/*.conf files, especially base.conf and 90lvm.conf: lsinitrd /boot/initramfs.img /etc/cmdline.d/base.conf lsinitrd /boot/initramfs.img /etc/cmdline.d/90lvm.conf
Please also append -L=6 to dracut args and paste the output.
Created attachment 322460 [details] dracut -L 6 (In reply to comment #2) > Please also append -L=6 to dracut args and paste the output. -L=6 produced syntax errors, hence I ran it with -L 6 instead.
(In reply to comment #1) > Please also paste the content of /etc/cmdline.d/*.conf files, especially > base.conf and 90lvm.conf: > > lsinitrd /boot/initramfs.img /etc/cmdline.d/base.conf > lsinitrd /boot/initramfs.img /etc/cmdline.d/90lvm.conf # lsinitrd kernel/initramfs-2.6.32-hardened-r116.img /etc/cmdline.d/base.conf ro # lsinitrd kernel/initramfs-2.6.32-hardened-r116.img /etc/cmdline.d/90lvm.conf # lsinitrd kernel/initramfs-2.6.32-hardened-r116.img /etc/cmdline.d/* ro
(In reply to comment #1) > Please append rd.debug to kernel cmdline and attach the log which you should > find in /run/initramfs. Please expect some more delay. The whole system is extremely slow over a serial console, and atm it appears impossible to interact with the debug shell at all.
Created attachment 322464 [details] boot.log (In reply to comment #5) > (In reply to comment #1) > > Please append rd.debug to kernel cmdline and attach the log which you should > > find in /run/initramfs. > Please expect some more delay. The whole system is extremely slow over a > serial console, and atm it appears impossible to interact with the debug > shell at all. By the time that the output stopped (the debug shell seems to be active minutes before that), the the shell would no longer accept my input. Hence I captured the output using tmux, instead of using the file in /run/initramfs.
(In reply to comment #6) > Created attachment 322464 [details] > boot.log > > (In reply to comment #5) > > (In reply to comment #1) > > > Please append rd.debug to kernel cmdline and attach the log which you should > > > find in /run/initramfs. > > Please expect some more delay. The whole system is extremely slow over a > > serial console, and atm it appears impossible to interact with the debug > > shell at all. > By the time that the output stopped (the debug shell seems to be active > minutes before that), the the shell would no longer accept my input. Hence I > captured the output using tmux, instead of using the file in /run/initramfs. The remaining bit (a new boot without rd.debug): <30>dracut: Mounted root filesystem /dev/sda2 <30>dracut: Mounting /usr with -o defaults,noatime,nodev,ro,defaults,noatime,nod ev,ro <30>dracut: mount: special device /dev/black/usr does not exist <28>dracut Warning: Mounting /usr to /sysroot/usr failed dracut Warning: Mounting /usr to /sysroot/usr failed <28>dracut Warning: *** Dropping you to a shell; the system will continue dracut Warning: *** Dropping you to a shell; the system will con<28>dracut Warni ng: *** when you leave the shell. tinue dracut Warning: *** when you leave the shell. <28>dracut Warning: dracut Warning: Dropping to debug shell. /bin/sh: 0: can't access tty; job control turned off dracut:/# lvm_scan <30>dracut: Scanning devices sda3 sdc1 for LVM volume groups <30>dracut: Reading all physical volumes. This may take a while... <30>dracut: Found volume group "black" using metadata type lvm2 <30>dracut: 5 logical volume(s) in volume group "black" now active dracut:/# mount /sysroot mount: /dev/sda2 already mounted or /sysroot busy mount: according to mtab, /dev/sda2 is already mounted on /sysroot dracut:/# mount /sysroot/usr kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with writeback data mode. grsec: mount of /dev/mapper/black-usr to /sysroot/usr by /bin/mount[mount:2049] uid/euid:0/0 gid/egid:0/0, parent /bin/dash[sh:1966] uid/euid:0/0 gid/egid:0/0 dracut:/# <30>dracut: Switching root grsec: mount of /dev to /sysroot/dev by /sbin/switch_root[switch_root:1] uid/eui d:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 grsec: mount of /proc to /sysroot/proc by /sbin/switch_root[switch_root:1] uid/e uid:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 grsec: mount of /sys to /sysroot/sys by /sbin/switch_root[switch_root:1] uid/eui d:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 grsec: mount of /run to /sysroot/run by /sbin/switch_root[switch_root:1] uid/eui d:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 grsec: mount of /sysroot to / by /sbin/switch_root[switch_root:1] uid/euid:0/0 g id/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 INIT: version 2.88 booting
Could you try to add following options to kernel cmdline? rd.lvm.lv=black/usr
(In reply to comment #8) > Could you try to add following options to kernel cmdline? > > rd.lvm.lv=black/usr Works. Does this mean that the usrmount module needs to inform the lvm module about the needed volumes?
I've also encountered this problem on one of my gentoo systems with sys-kernel/dracut-023. But sys-kernel/dracut-022-r5 works fine! I also have another two gentoo systems with LVM which boots fine with both dracut 022-r5 and 023. The main difference of them is the lack of mdraid. $ emerge --info Portage 2.1.11.9 (hardened/linux/amd64, gcc-4.6.3, glibc-2.15-r2, 3.4.5-hardened x86_64) ================================================================= System uname: Linux-3.4.5-hardened-x86_64-AMD_E-350_Processor-with-gentoo-2.1 Timestamp of tree: Tue, 02 Oct 2012 00:45:01 +0000 app-shells/bash: 4.2_p37 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.3-r2, 3.2.3 dev-util/cmake: 2.8.9 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.10.5 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.13, 2.68 sys-devel/automake: 1.9.6-r3, 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers) sys-libs/glibc: 2.15-r2 Repositories: gentoo x-local ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=btver1 -mtune=btver1 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=btver1 -mtune=btver1 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps y" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles/ ftp://mirror.yandex.ru/gentoo-distfiles/ " LANG="ru_RU.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="ru en" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local" SYNC="" USE="X aac acl acpi aio alsa amd64 amr ape asyncns avahi bash-completion berkdb bluetooth bluray branding bzip2 cairo caps cdda cddb cdparanoia cdr cli colord consolekit cracklib crypt cups cxx dbus device-mapper djvu dri dts dvd dvdr examples exif faac faad ffmpeg flac fontconfig fuse gdbm gif gnome gnome-keyring gpm gtk gtk3 hardened iconv icu idn ieee1394 ipv6 jpeg jpeg2k justify libnotify libsamplerate logrotate lzma mac mad maildir mms mmx mmxext modules mp3 mp4 mudflap multilib musepack musicbrainz nautilus ncurses nls nptl nptlonly ogg openal opengl openmp pam pax_kernel pcre perl pmu png policykit postscript pppd pulseaudio python raw readline samba session speex sse sse2 sse3 sse4a ssl ssse3 startup-notification svg tcpd theora tiff tls truetype udev unicode upnp urandom usb vaapi vdpau vim-syntax vorbis vpx wavpack webp x264 xattr xfs xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="*" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="*" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DRACUT_MODULES="lvm mdraid" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="ru en" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" QEMU_SOFTMMU_TARGETS="*" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="radeon vesa v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
(In reply to comment #10) > I've also encountered this problem on one of my gentoo systems with > sys-kernel/dracut-023. But sys-kernel/dracut-022-r5 works fine! > > I also have another two gentoo systems with LVM which boots fine with both > dracut 022-r5 and 023. The main difference of them is the lack of mdraid. The machines that work do not use mdraid? I think I am not using that, but just plain LVM - how do I check?
> The machines that work do not use mdraid? Yes. I'll try to obtain more complete info when I have free time for experiments.
(In reply to comment #9) > (In reply to comment #8) > > Could you try to add following options to kernel cmdline? > > > > rd.lvm.lv=black/usr > Works. Does this mean that the usrmount module needs to inform the lvm > module about the needed volumes? It was needed in earlier versions, later it wasn't needed anymore afaik, and it looks there's some bug now. Please do not attach emerge --info anymore for dracut package, because I don't need it at all with except of information about what flags are enabled.
(In reply to comment #11) > (In reply to comment #10) > > I've also encountered this problem on one of my gentoo systems with > > sys-kernel/dracut-023. But sys-kernel/dracut-022-r5 works fine! > > > > I also have another two gentoo systems with LVM which boots fine with both > > dracut 022-r5 and 023. The main difference of them is the lack of mdraid. > The machines that work do not use mdraid? I think I am not using that, but > just plain LVM - how do I check? Check if mdadm is included in your initramfs.
# which mdadm which: no mdadm in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/ia64-unknown-linux-gnu/gcc-bin/4.5.3) # locate mdadm [empty] Hence I assume it is not contained in my initramfs. I do not even know what it is...
Created attachment 325678 [details] dracut-023.log dracut -L 6 output
Created attachment 325680 [details] init.log rd.debug added to kernel args
Adding rd.lvm.lv=vg_system/root to the kernel cmdline did not help in my case.
*** Bug 437700 has been marked as a duplicate of this bug. ***
(In reply to comment #13) > (In reply to comment #9) > > (In reply to comment #8) > > > Could you try to add following options to kernel cmdline? > > > > > > rd.lvm.lv=black/usr > > Works. Does this mean that the usrmount module needs to inform the lvm > > module about the needed volumes? > > It was needed in earlier versions, later it wasn't needed anymore afaik, and > it looks there's some bug now. > As I understand it, LVM activation hooks are inserted into "hooks/initqueue/settled" and executed in main loop only if some hooks from "hooks/initqueue" not finished (for example if they call `wait_for_dev DEV` and device DEV is not ready). wait_for_dev() function is called only from this hooks: pre-udev/30resume-genrules.sh pre-udev/30block-genrules.sh cmdline/30parse-lvm.sh Or in other words, only if root=, rd.lvm.{vg,lv}= and/or resume= are appended to kernel cmdline. So if the root is on ordinary partition and /usr is on lvm then LVM is not automatically activated. In this case you need to append rd.lvm.{vg,lv}= to kernel cmdline.
(In reply to comment #20) > wait_for_dev() function is called only from this hooks: > > pre-udev/30resume-genrules.sh > pre-udev/30block-genrules.sh > cmdline/30parse-lvm.sh > This is not entirely true and is valid if only lvm and mdraid are enabled.
Please test dracut-024-r1.
(In reply to comment #22) > Please test dracut-024-r1. I am using dracut-024-r3 now (not exactly sure about the -r, will report again later if I was wrong), and now the rd.lvm.lv=black/usr workaround does not work anymore. Instead I get: [ 34.818954] dracut: Scanning devices sda3 sdc1 for LVM logical volumes black/usr [ 34.934143] dracut: inactive '/dev/black/usr' [8.00 GiB] inherit [ 34.934568] dracut: inactive '/dev/black/var' [16.00 GiB] inherit [ 34.935182] dracut: inactive '/dev/black/home' [10.00 GiB] inherit [ 34.935585] dracut: inactive '/dev/black/opt' [200.00 MiB] inherit [ 34.943483] dracut: inactive '/dev/black/vtmp' [2.00 GiB] inherit [ 35.033242] bio: create slab <bio-1> at 1 [ 35.042329] dracut: /dev/mapper/black-usr: open failed: No such file or directory And no debug console, so I cannot do anything about it (i.e. activate it myself) anymore.
P.S: This was using hardened-sources-3.7.0 and udev-171-r9 (again, will report back if I guessed the wrong -r).
hardened-sources-3.7.0 + udev-196-r1 + lvm2-2.02.97-r1 + dracut-024-r3 - works well for me now. $ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.7.0-hardened root=/dev/mapper/vg_system-root ro rd.lvm.lv=vg_system/root rd.lvm.lv=vg_system/usr resume=UUID=beabe4a8-484a-4614-8e3e-7c8fa75e8c0e Before that udev-171-r9 + lvm2-2.02.88 + dracut-024-r2 also worked without problems.
(In reply to comment #23) > I am using dracut-024-r3 now (not exactly sure about the -r, will report > again later if I was wrong), and now the rd.lvm.lv=black/usr workaround does > not work anymore. This is not a workaround but a requirement as of dracut-024 http://article.gmane.org/gmane.linux.kernel.initramfs/2963
(In reply to comment #23) > [ 35.042329] dracut: /dev/mapper/black-usr: open failed: No such file or > directory This issue is gone now (no clue what caused it — I found too many issues in my build process to name it exactly). Therefore I close this bug.
