pvmove fails and confuses the kernel. I was attempting to expand an important logical volume, but was out of extents on my RAID physical volumes. I decided to move some extents from a less important LV off a RAID PV onto a plain PV to make room. When I issued the pvmove command it failed: alexandria ~ # pvmove -v -n vg00/lv_dvds /dev/md3:527-1059 /dev/hdd9 Wiping cache of LVM-capable devices Finding volume group "vg00" Archiving volume group "vg00" metadata (seqno 715). Creating logical volume pvmove0 Moving 533 extents of logical volume vg00/lv_dvds Found volume group "vg00" Updating volume group metadata Creating volume group backup "/etc/lvm/backup/vg00" (seqno 716). Found volume group "vg00" Found volume group "vg00" Suspending vg00-lv_dvds (253:4) Found volume group "vg00" Creating vg00-pvmove0 Loading vg00-pvmove0 table device-mapper: reload ioctl failed: Invalid argument ABORTING: Temporary mirror activation failed. Run pvmove --abort. Found volume group "vg00" Loading vg00-pvmove0 table device-mapper: reload ioctl failed: Invalid argument Loading vg00-lv_dvds table device-mapper: reload ioctl failed: Invalid argument dmesg reveals the following: device-mapper: unrecognised sync argument to mirror log: 2 device-mapper: dm-mirror: Error creating mirror dirty log device-mapper: error adding target to table device-mapper: unrecognised sync argument to mirror log: 2 device-mapper: dm-mirror: Error creating mirror dirty log device-mapper: error adding target to table device-mapper: device 253:4 too small for target device-mapper: dm-linear: Device lookup failed device-mapper: error adding target to table From this point on any command which attempts to read or write LVM information gets stuck with a D flag in ps (disk wait?). lvdisplay, pvdisplay, vgdisplay. When I attempt to reboot the machine it gets stuck deactivating the LVM system and I have to press the reset button. As it boots back up it fails during file system check and I have to type the root password and perform a "pvmove --abort" to return the system to normal. My research leads me to believe there may be a version mismatch between device-mapper in the kernel and the user-space libraries. I have not been able to find a web page that specifies which software versions I should have installed to match my kernel. I have attempted with both 2.6.12-gentoo-r10 and 2.6.16-gentoo-r13. The failure mode is the same in both. (I tried a 2.6.17 kernel but its USB system was malfunctioning too horribly for me to actually issue a pvmove command) alexandria ~ # uname -a Linux alexandria 2.6.12-gentoo-r10 #11 Fri Jan 13 22:43:46 EST 2006 i686 AMD Athlon(tm) XP 2600+ GNU/Linux alexandria ~ # epm -qa | egrep 'device-mapper|lvm2' device-mapper-1.02.07 lvm2-2.02.06 I was able to accomplish a long string of pvmoves (moving data off failing disk and onto RAIDs) back in June, so this has worked in the past. Based on a clumsy scan of /var/log/emerge.log I think I was running lvm2-2.01.09 and device-mapper-1.00.19-r2 back then, but I have not tried downgrading yet. alexandria ~ # emerge --info Portage 2.1.1 (default-linux/x86/2006.0, gcc-3.4.6, glibc-2.4-r3, 2.6.12-gentoo-r10 i686) ================================================================= System uname: 2.6.12-gentoo-r10 i686 AMD Athlon(tm) XP 2600+ Gentoo Base System version 1.12.5 Last Sync: Fri, 22 Sep 2006 08:00:08 +0000 app-admin/eselect-compiler: [Not Present] dev-java/java-config: 1.2.11-r1 dev-lang/python: 2.3.5-r2, 2.4.3-r1 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: [Not Present] dev-util/confcache: [Not Present] sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.13-r3 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r5 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=i686 -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb /var/bind" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-O2 -march=i686 -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks metadata-transfer sandbox sfperms strict" GENTOO_MIRRORS="ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo http://mirror.datapipe.net/gentoo ftp://gentoo.chem.wisc.edu/gentoo/" LINGUAS="en" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/home/thoth/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 3dnow 3dnowex X Xaw3d aac acl alsa apache2 apm bash-completion berkdb bitmap-fonts bzilb bzip2 cli crypt cups divx4linux dri dvb dvd dvdr edl elibc_glibc emboss encode foomaticdb gdbm gif gnome gpm gstreamer gtk gtk2 imlib input_devices_keyboard input_devices_mouse ipv6 isdnlog java jpeg kde kernel_linux lcd libg++ libwww linguas_en lirc lirc_devices_hauppauge mad maildir matroska mikmod mjpeg mmx mmx2 mng motif mozilla mozsvg mp3 mpeg mysql mythtv ncurses nls nojython nptl ogg opengl pam pcre perl png pppd python qt3 qt4 quicktime readline real reflection sdl session spell spl sse ssl svg tcltk tcpd theora transcode truetype truetype-fonts type1-fonts udev unicode userland_GNU v4l v4l2 video_cards_fbdev video_cards_nv video_cards_nvidia video_cards_vga vorbis win32codecs wmf x264 xinerama xml xmms xorg xprint xv xvid zlib" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, MAKEOPTS, PORTAGE_RSYNC_EXTRA_OPTS
Please, try w/ device-mapper-1.02.09
Created attachment 98061 [details] vgdisplay; lvdisplay -m ; pvdisplay
I keyworded and emerged device-mapper-1.02.09. I also re-emerged lvm2 in case it was statically linked against device-mapper. revdep-rebuild --library libdevmapper.so.1.01 -p reveals nothing dynamically linked against the old libdevmapper. alexandria ~ # pvmove -v -n vg00/lv_dvds /dev/md3:527-1059 /dev/hdd9 Wiping cache of LVM-capable devices Finding volume group "vg00" Archiving volume group "vg00" metadata (seqno 721). Creating logical volume pvmove0 Moving 533 extents of logical volume vg00/lv_dvds Found volume group "vg00" Updating volume group metadata Creating volume group backup "/etc/lvm/backup/vg00" (seqno 722). Found volume group "vg00" Found volume group "vg00" Suspending vg00-lv_dvds (253:5) Found volume group "vg00" Creating vg00-pvmove0 Loading vg00-pvmove0 table Resuming vg00-pvmove0 (253:4) There is nothing in dmesg. I left for an hour and came back. Still no progress. I don't see any disk activity in "vmstat 10", but I can't remember if that is normal. strace -f lvdisplay reveals that lvdisplay gets stuck during: stat64("/proc/lvm/VGs/vg00", 0xbfbf6410) = -1 ENOENT (No such file or directory) rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0 open("/var/lock/lvm/V_vg00", O_RDWR|O_APPEND|O_CREAT|O_LARGEFILE, 0777) = 3 rt_sigaction(SIGINT, {0x80a0680, [INT], SA_RESTART}, {SIG_DFL}, 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[INT RTMIN RT_1], NULL, 8) = 0 rt_sigaction(SIGINT, NULL, {0x80a0680, [INT], SA_RESTART}, 8) = 0 rt_sigaction(SIGINT, {0x80a0680, [INT], 0}, NULL, 8) = 0 flock(3, LOCK_SH
maybe this could help http://blog.charlies-server.no-ip.com/2006/09/26/lvm-moving-around-drives-unhappy/
Upgrading to device-mapper-1.02.10 lvm2-2.02.10 allowed me to perform pvmoves again. Although the machine crashed several hours into a particularly long one last night, I'm inclined to blame that on general pvmove immaturity. I'm not sure how you want to signal to the user that certain versions of the ebuild are incompatible with certain versions of the kernel
I had the same symptoms on a freshly installed system, device-mapper-1.02.07 lvm2-2.02.06 would fail to pvmove, unmasking device-mapper-1.02.10 & lvm2-2.02.10 with gentoo-sources-2.6.17-r8 allows pvmove to function. --- Portage 2.1.1 (default-linux/x86/2006.1, gcc-4.1.1, glibc-2.4-r3, 2.6.17-gentoo-r8 i686) ================================================================= System uname: 2.6.17-gentoo-r8 i686 Intel(R) Pentium(R) 4 CPU 2.00GHz Gentoo Base System version 1.12.5 Last Sync: Sun, 29 Oct 2006 15:00:02 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled] app-admin/eselect-compiler: [Not Present] dev-java/java-config: [Not Present] dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: [Not Present] dev-util/confcache: [Not Present] sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.13-r4 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r1 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=i686 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/splash /etc/terminfo" CXXFLAGS="-O2 -march=i686 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distcc distlocks metadata-transfer sandbox sfperms strict" GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://distro.ibiblio.org/pub/linux/distributions/gentoo/ http://www.gtlib.gatech.edu/pub/gentoo " LINGUAS="en" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X a52 aac alsa apache2 audiofile berkdb binary-drivers bitmap-fonts cdparanoia cdr cli cracklib crypt dlloader dri dvd dvdread elibc_glibc encode ffmpeg fortran gdbm gif iconv input_devices_evdev input_devices_keyboard input_devices_mouse isdnlog jpeg kernel_linux libg++ linguas_en lirc lirc_devices_tira lzo mad mmx mng mp3 mpeg ncurses nls nptl nptlonly ogg opengl pam pcre perl png ppds pppd python quicktime readline reflection session spl sse sse2 ssl tcpd tga theora tiff truetype truetype-fonts type1-fonts udev unicode userland_GNU video_cards_nvidia video_cards_v4l video_cards_vesa video_cards_vga vorbis xorg xv xvid xvmc yv12 zlib" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, PORTAGE_RSYNC_EXTRA_OPTS ---
2.02.10 is stable on all arches now.
