After upgrading to sys-fs/lvm2-2.02.100-r2 the system behaves rather strangely. There are multiple symptoms which seem related: 1) I have to build my own initrd, which contains statically linked versions of lvm, busybox etc. In the init script contained in the same initrd, up to now, I did the following: /bin/lvm vgscan >/dev/null 2>&1 /bin/lvm vgchange -ay linux This used to create the symlinks under /dev/mapper that could then be mounted and pivoted to. With the "new" lvm, I had to add the line /bin/vgscan --mknodes Does autoactivation no longer create the links under /dev/mapper? 2) After getting the system back to bootable with the above change, lvm fails to see any volume groups in the running system: # lvm vgscan Reading all physical volumes. This may take a while... No volume groups found # lvm version LVM version: 2.02.100(2) (2013-08-13) Library version: 1.02.79 (2013-08-13) Driver version: 4.25.0 But: # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/linux-gentoo_root 30832636 18207264 11036124 63% / tmpfs 777692 3032 774660 1% /run udev 10240 0 10240 0% /dev shm 3888444 0 3888444 0% /dev/shm cgroup_root 10240 0 10240 0% /sys/fs/cgroup /dev/mapper/linux-ubuntu_root 15350768 6431648 8116308 45% /mnt/ubuntu ... with gentoo_root and ubuntu_root being logical volumes in group "linux". An older version of lvm displays them correctly: # mount -t proc proc /mnt/ubuntu/proc # mount -o bind /dev/ /mnt/ubuntu/dev # chroot /mnt/ubuntu/ # lvm vgscan Reading all physical volumes. This may take a while... Found volume group "linux" using metadata type lvm2 # lvm lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert gentoo_root linux -wi-ao-- 30.00g ubuntu_root linux -wi-ao-- 15.00g # lvm version LVM version: 2.02.95(2) (2012-03-06) Library version: 1.02.74 (2012-03-06) Driver version: 4.25.0 Happens with both kernel 3.10.2 and 3.11.1. BTW: This is a SSD-only UEFI system. Reproducible: Always # emerge --info Portage 2.2.7 (default/linux/amd64/13.0/desktop, gcc-4.7.3, glibc-2.17, 3.11.1-gentoo x86_64) ================================================================= System uname: Linux-3.11.1-gentoo-x86_64-Intel-R-_Core-TM-_i7-3517U_CPU_@_1.90GHz-with-gentoo-2.2 KiB Mem: 7776892 total, 5800524 free KiB Swap: 7999484 total, 7999484 free Timestamp of tree: Fri, 27 Sep 2013 09:15:01 +0000 ld GNU ld (GNU Binutils) 2.23.2 app-shells/bash: 4.2_p45 dev-java/java-config: 2.2.0::java-overlay dev-lang/python: 2.7.5-r2, 3.2.5-r2, 3.3.2-r2 dev-util/cmake: 2.8.11.2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.6, 1.12.6, 1.13.4, 1.14 sys-devel/binutils: 2.23.2 sys-devel/gcc: 4.5.4, 4.6.3, 4.7.3-r1 sys-devel/gcc-config: 1.8 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.11 (virtual/os-headers) sys-libs/glibc: 2.17 Repositories: gentoo java-overlay x-portage ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=native -pipe -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/polkit-1/actions" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=native -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/ ftp://sunsite.cnlab-switch.ch/mirror/gentoo/ http://de-mirror.org/gentoo/" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/java-overlay /usr/local/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="3dnow 3dnowext X a52 aac accessibility acl acpi alsa amd64 arts avahi bash-completion berkdb bluetooth branding bzip2 cairo caps cdda cddax cddb cdparanoia cdr cli consolekit cracklib crypt css cups cxx dbus device-mapper dga dirac dns dri dts dv dvb dvd dvdnav dvdr dvdread emboss encode esd exif faac faad fam ffmpeg firefox flac fortran ftp gd gdbm gdu ggi gif gimp git gmp gnome gphoto2 gpm gstreamer gtk gtkhtml gzip h323 hal hdmi http iconv icu imagemagick imap imlib ipv6 java jce jpeg jpeg2k kde kdrive kontact lame laptop lcms ldap libnotify live lzo mad matroska mdnsresponder-compat mikmod mime mmap mmx mmxext mng modplug modules mozilla mp3 mp4 mpeg mplayer mudflap multilib mysql ncurses network networkmanager nls nptl nsplugin nss obex ogg opencore-amr opengl openmp openssl osdmenu oss otr pam pango pcmcia pcre pcsc pcsc-lite pda pdf perl php pipechan plasma png pnm policykit ppds pvr qt3 qt3support qt4 radio raw rdesktop readline real rss sasl scanner sdl secure-delete semantic-desktop session sip smartcard smp sockets spell sql sqlite sse sse2 ssl ssse3 startup-notification static-libs subversion svg tcpd theora threads tiff timidity toolame tremor truetype twolame udev udisks unicode upower usb utils v4l v4l2 vcd vnc vorbis wav wavpack webkit wifi win32codecs wmf wxwidgets x264 xanim xattr xcb xcomposite xine xinerama xml xpm xscreensaver xv xvid zeroconf zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="all" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="keyboard mouse evdev synaptics void alps" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en de" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7" RUBY_TARGETS="ruby19 ruby18 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USE_PYTHON="2.7" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
2) - looks like the same problem as in bug 454746
Created attachment 359794 [details] initramfs init script If you see something that gives you suspsious comment and I will be more then happy to test changes.
(In reply to Jory A. Pratt from comment #2) > Created attachment 359794 [details] > initramfs init script > > If you see something that gives you suspsious comment and I will be more > then happy to test changes. You should not umount /proc, /dev and /sys in the initramfs. From the switch_root man page: "switch_root moves already mounted /proc, /dev and /sys to newroot and makes newroot the new root filesystem and starts init process."
(In reply to Alexander Tsoy from comment #3) > (In reply to Jory A. Pratt from comment #2) > > Created attachment 359794 [details] > > initramfs init script > > > > If you see something that gives you suspsious comment and I will be more > > then happy to test changes. > > You should not umount /proc, /dev and /sys in the initramfs. From the > switch_root man page: > "switch_root moves already mounted /proc, /dev and /sys to newroot and makes > newroot the new root filesystem and starts init process." And if device nodes not get created, consider using "vgscan --mknodes" as mentioned in comment 0.
(In reply to Alexander Tsoy from comment #4) > (In reply to Alexander Tsoy from comment #3) > > (In reply to Jory A. Pratt from comment #2) > > > Created attachment 359794 [details] > > > initramfs init script > > > > > > If you see something that gives you suspsious comment and I will be more > > > then happy to test changes. > > > > You should not umount /proc, /dev and /sys in the initramfs. From the > > switch_root man page: > > "switch_root moves already mounted /proc, /dev and /sys to newroot and makes > > newroot the new root filesystem and starts init process." > > And if device nodes not get created, consider using "vgscan --mknodes" as > mentioned in comment 0. Even with vgscan --mknodes devices nodes are still not created, I have to do a double scan/activate in order for them to appear and be usable.
(In reply to Jory A. Pratt from comment #5) > Even with vgscan --mknodes devices nodes are still not created, I have to do > a double scan/activate in order for them to appear and be usable. Try this sequence of commands: pvscan > /dev/null vgscan --mknodes > /dev/null vgchange --sysinit -ay > /dev/null
(In reply to Alexander Tsoy from comment #6) > (In reply to Jory A. Pratt from comment #5) > > Even with vgscan --mknodes devices nodes are still not created, I have to do > > a double scan/activate in order for them to appear and be usable. > > Try this sequence of commands: > > pvscan > /dev/null > vgscan --mknodes > /dev/null > vgchange --sysinit -ay > /dev/null Same failure `vgscan --mknodes` is having to be run twice before the device can be seen.
(In reply to Alexander Tsoy from comment #1) > 2) - looks like the same problem as in bug 454746 Well, not exactly. The symlinks in the running system ARE there: # ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 Sep 29 11:07 control brw------- 1 root root 252, 2 Sep 29 11:07 linux-gentoo_root brw------- 1 root root 252, 0 Sep 29 11:07 linux-ubuntu_root It's just lvm2 that fails to read the lvm partitions. I umount /proc and /sys in my initrd script, but leave /dev mounted (not a devfs, obviously), because it's just a directory in the initrd's root file system. But apparently udev takes care of this, as in the running system, /dev is mounted as devfs. Should I leave /proc and /sys mounted before calling switch_root?
(In reply to Markus Wernig from comment #8) > Should I leave /proc and /sys mounted before calling switch_root? OK, I tried that, with the following results: - no network interfaces started (no such device) - some kernel modules fail to load (virtualbox drivers, maybe others) - X starts but does not accept any keyboard or mouse input and is thus unusable Changing the initrd script back (i.e. calling "umount /proc; umount /sys" again, solves the problem.
Created attachment 359924 [details] init from initrd Maybe the problem is not with lvm2, but with initrd? But then, why does another lvm-Version on the same system successfully read the lvm volumes?
(In reply to Markus Wernig from comment #10) > Created attachment 359924 [details] > init from initrd > > Maybe the problem is not with lvm2, but with initrd? > But then, why does another lvm-Version on the same system successfully read > the lvm volumes? Try mounting /dev as devtmpfs in initramfs. This way all necessary symlinks should be preserved after switch_root. This worked for reporter of bug 454746
(In reply to Markus Wernig from comment #8) > (In reply to Alexander Tsoy from comment #1) > > 2) - looks like the same problem as in bug 454746 > > Well, not exactly. The symlinks in the running system ARE there: > > # ls -l /dev/mapper > total 0 > crw------- 1 root root 10, 236 Sep 29 11:07 control > brw------- 1 root root 252, 2 Sep 29 11:07 linux-gentoo_root > brw------- 1 root root 252, 0 Sep 29 11:07 linux-ubuntu_root /dev/mapper normally contains symlinks to "../dm-*", but you have device nodes here.
(In reply to Alexander Tsoy from comment #11) > (In reply to Markus Wernig from comment #10) > > Created attachment 359924 [details] > > init from initrd > > > > Maybe the problem is not with lvm2, but with initrd? > > But then, why does another lvm-Version on the same system successfully read > > the lvm volumes? > > Try mounting /dev as devtmpfs in initramfs. This way all necessary symlinks > should be preserved after switch_root. This worked for reporter of bug 454746 Not only the initramfs should have /dev with devtmpfs but it should use udev which makes it mandatory in any case.
I'm sorry, but for all I can find out, this really IS a problem in lvm2. Or in switch_root, the kernel, or somewhere. I know that this is not a support forum, but for me this is not a pebcak. I would still very much appreciate any help. I now have changed my init script in initramfs: #!/bin/sh mount -t proc none /proc CMDLINE='cat /proc/cmdline' mount -t sysfs none /sys mount -t devtmpfs none /dev [... skipped irrelevant stuff] # lvm - commands really need to be executed in that order, else no devices are # created /bin/lvm vgscan >/dev/null 2>&1 /bin/lvm vgchange -ay linux || rescue_shell "lvm vgchange -ay linux" /bin/vgscan --mknodes || rescue_shell "vgscan --mknodes" ls -l /dev/mapper sleep 10 #root filesystem mount -t ext4 -r /dev/mapper/linux-gentoo_root /newroot || rescue_shell ls -l /dev/mapper sleep 10 # unmount pseudo FS - else many devices are missing in the booted # system afterwards umount /proc umount /sys umount /dev #root switch exec /bin/busybox switch_root /newroot /sbin/init ${CMDLINE} ----- This gives me a working system, but still the original problem persists: # lvm vgscan Reading all physical volumes. This may take a while... No volume groups found # lvm pvscan No matching physical volumes found The annoying effect of this is that I am no longer able to manage any lvm devices. /sbin/lvm.static gives the same result. ----- When watching the output from the above init script during boot, I see the message "Found volume group "linux", using metadata format lvm2" (from memory) The "ls -l /dev/mapper" during the init script gives the exacty same output as in my running system afterwards (not sure about the "control"): # ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 Oct 13 19:39 control brw------- 1 root root 252, 2 Oct 13 19:40 linux-gentoo_root brw------- 1 root root 252, 3 Oct 13 19:40 linux-home brw------- 1 root root 252, 1 Oct 13 19:39 linux-swap_1 brw------- 1 root root 252, 0 Oct 13 19:40 linux-ubuntu_root So it seems that lvm (or the kernel via devtmpfs?) creates the devices as such, and not as symlinks. ----- On the same system, same running kernel, an older version of the lvm2 tools (from Ubuntu) DOES see the devices. So I would rather not assume it's the kernel. Please let me know if you need more info. thx /markus
Ok, my bad. Pebcak after all ... :-[ I had the following in lvm.conf: filter = [ "r|/dev/nbd.*|", "a|/dev/sd.*/|", "r/.*/" ] Removing the typo solved the "problem" filter = [ "r|/dev/nbd.*|", "a|/dev/sd.*|", "r/.*/" ] # lvm pvscan PV /dev/sda3 VG linux lvm2 [238.05 GiB / 0 free] Total: 1 [238.05 GiB] / in use: 1 [238.05 GiB] / in no VG: 0 [0 ] # lvm vgscan Reading all physical volumes. This may take a while... Found volume group "linux" using metadata type lvm2 Sorry for the noise and thanks for bearing with me ... PS: The fact that there are device nodes instead of symlinks in /dev/mapper is strange, but seems of no effect.
