lvcreate'ing a snapshot volume hangs (and also hangs other processes) if the volume being snapshotted is being written to heavily. It doesn't always hang, but it usually only takes a few minutes to reproduce. Once it hangs badly, it generally starts to hang anything that accesses the volume, so the only way to get out is "reboot -nf". Reproducible: Sometimes Steps to Reproduce: 1. lvcreate -L256M -nfoo vg0 && mkfs.ext2 /dev/vg0/foo && mount /dev/vg0/foo /mnt 2. In one console: while true; do dd if=/dev/zero of=/mnt/big; rm /mnt/big; done 3. In another console: while true; do lvcreate -s -nsnap -L64M vg0/foo; lvremove -f vg0/snap; done Actual Results: Creates and removes some snapshots (some of the operations fail claiming the snapshot is busy). After a while, hangs an lvcreate process and a dd process. Invoking other programs that access that volume hangs them too. There is NO hard drive activity at this time! Expected Results: Should not hang. Should just go creating and removing snapshots forever. [I--] [ ] sys-kernel/gentoo-sources-2.6.22-r9 (2.6.22-r9) [I--] [ ] sys-fs/lvm2-2.02.28-r2 (0) [I--] [ ] sys-fs/device-mapper-1.02.22-r5 (0) [I--] [ -] sys-kernel/linux-headers-2.6.22-r2 (0) [I--] [ -] sys-devel/gcc-4.1.2 (4.1) [I--] [ ] sys-libs/glibc-2.6.1 (2.2) # emerge --info Portage 2.1.3.19 (default-linux/x86/2007.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.22-gentoo-r9 i686) ================================================================= System uname: 2.6.22-gentoo-r9 i686 Intel(R) Pentium(R) 4 CPU 2.80GHz Timestamp of tree: Mon, 03 Dec 2007 02:16:01 +0000 app-shells/bash: 3.2_p17 dev-java/java-config: 1.3.7, 2.0.33-r1 dev-lang/python: 2.4.4-r6 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 1.12.9-r2 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.61-r1 sys-devel/automake: 1.5, 1.7.9-r1, 1.9.6-r2, 1.10 sys-devel/binutils: 2.18-r1 sys-devel/gcc-config: 1.3.16 sys-devel/libtool: 1.5.24 virtual/os-headers: 2.6.22-r2 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium4 -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-O2 -march=pentium4 -pipe -fomit-frame-pointer" DISTDIR="/tmp/distfiles" FEATURES="distlocks metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://adelie.polymtl.ca/ ftp://gentoo.arcticnetwork.ca/pub/gentoo/ http://gentoo.arcticnetwork.ca/ ftp://mirrors.tera-byte.com/pub/gentoo http://gentoo.mirrors.tera-byte.com/ " LANG="en_CA.utf8" LC_ALL="en_CA.utf8" LINGUAS="en en_CA" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="X acpi alsa bash-completion bitmap-fonts bzip2 crypt cscope dhcp directfb dri expat fam fat fbcon firefox flac fuse gif glibc-omitfp gmp gtk gtk2 hddtemp hpn iconv isdnlog java jce jpeg kdeenablefinal kdehiddenvisibility lm_sensors mad midi mikmod mmx mp3 mudflap ncurses nls no-old-linux nolvm1 nptl nptlonly nsplugin ntfs ogg opengl openmp pam pdf png readline reiserfs scanner sdl server smp sse sse-filters sse2 ssl symlink timidity truetype truetype-fonts type1-fonts unicode usb vim vim-syntax vorbis x86 xcb xml xorg xv zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_CA" USERLAND="GNU" VIDEO_CARDS="radeon" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 138195 [details] kernel configuration
Folks could you create yourself an lvm-bugs or whatnot alias? This gets a huge PITA to assign.
Sounds like your hardware doesn't support enough IO for what you are doing. Your snapshot is also too small for the amount of writing you are doing - it should be large enough to handle the quantity of writes expected in the lifetime of the snapshot. I cannot reproduce this on my serious LVM box (that runs lvm snapshots every 4 hours for backups): 16-disk 10k RPM SAS, 16Gb RAM, 2x dual-core Opteron 2214 HE. (presently on 2.6.19.7, uptime 90 days) - the snapshoting is done on the MySQL datadir, and it takes a full minute with the amount of write IO that is seen by that LV. For testing that it's an IO problem. Create a system with TWO sets of disks. First is the system set that you are running off. Second is/are disk(s) for the test - absolutely nothing else should be on them, not mounted even until the test. Make VG 'vgtest' consisting of the test disks. Make LV 'foo' for your test. Open two windows for monitoring. In the first, run top. In the second run 'iostat -x -t 5 -m DEV' (with some set of DEV that are the real devices for vgtest). Run the test, and keep an eye on iowait cpu, system cpu, and the disk utilization. I tried on a smaller box of mine as well: 2x 750Gb in software RAID1, quad G5, 12Gb of RAM. system cpu was 12%, iowait was 30%, disk utilization was 5%. This was a 4-way box, so that implies a single-cpu box does not have enough CPU to run the test with my RAID1 disks.
I'm afraid I don't have a system with a second disk to test with. However, I do have some news: I've reduced the "steps to reproduce" to a much simpler and more deterministic sequence: 1. lvcreate -L256M -nfoo vg0 2. lvcreate -s -L64M -nsnap vg0/foo 3. dd if=/dev/zero of=/dev/vg0/foo bs=1M count=256 A short way into the dd, syslog gets "device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.". Perfect! This is precisely what I expected. Here's the problem: the dd process also hangs, in state D+ (according to ps aux), aka "uninterruptible sleep". As the state suggests, it is unkillable. This is not so perfect: while I expected the snapshot to stop working once it runs out of space, it doesn't seem at all sensible for processes accessing the *original volume* to start hanging. Everything I've ever read about LVM suggests that the snapshot will start returning errors when you try to read from it, but the original volume will not be affected.
Your new instructions, are you running #2 and #3 in parallel or series? I do get the kernel message you mention, but my boxes never lock up (again, they are all SMP with plenty of disk etc). [835297.942101] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Series. No parallel operations this time. Also, new news: I haven't done extensive enough testing to be sure, but I think the problem may be gone as of 2.6.23. I just upgraded and can't reproduce.
Can't reproduce any more under 2.6.23 on a bunch of machines. Thanks for your time!
