Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 438262

Summary: sys-fs/lvm2-2.02.97 /etc/init.d/device-mapper does not support multi-line volumes in /etc/dmtab
Product: Gentoo Linux Reporter: Mads <mads>
Component: [OLD] Core systemAssignee: Robin Johnson <robbat2>
Status: RESOLVED FIXED    
Severity: normal CC: agk, cardoe, proxy-maint
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: new /etc/init.d/device-mapper, non-working
new /etc/init.d/device-mapper, working
new device-mapper init.d in patch form

Description Mads 2012-10-13 19:59:58 UTC
The method "get_new_dm_volumes()" in the init script device-mapper in LVM2-2.02.97 works by reading one and one line in the file /etc/dmtab, and then building dmsetup arguments manually from those lines. That makes volumes that requires several lines in /etc/dmtab to not work, because device-mapper then tries to create the same volume twice.

A good example of a multi line volume definition can be seen under EXAMPLES in `man dmsetupĀ“:

> EXAMPLES
>        # A table to join two disks together
>        0 1028160 linear /dev/hda 0
>        1028160 3903762 linear /dev/hdb 0

In /etc/dmtab, you would write that like this:
volname: 0 1028160 linear /dev/hda 0
volname: 1028160 3903762 linear /dev/hdb 0

A configuration like that will make device-mapper fail to set up volname correctly (it will not append /dev/hdb to /dev/hda because it only runs the first appearance of "volname").

Reproducible: Always

Steps to Reproduce:
1. Place those two mentioned volname-lines in /etc/dmtab
2. Run /etc/init.d/device-mapper start

Actual Results:  
 *  Setting up device-mapper volumes:
 *   Creating volume: volname ...                                                                                         [ ok ]
 *   Creating volume: volname ...
 *   Error creating volume: volname                                                                                       [ !! ]
 * ERROR: device-mapper failed to start


Expected Results:  
Volname should be created successfully.

Portage 2.2.0_alpha134 (default/linux/amd64/10.0, gcc-4.7.2, glibc-2.15-r3, 3.6.0-gentoo x86_64)
=================================================================
System uname: Linux-3.6.0-gentoo-x86_64-AMD_FX-tm-8120_Eight-Core_Processor-with-gentoo-2.2
Timestamp of tree: Sat, 13 Oct 2012 19:30:02 +0000
app-shells/bash:          4.2_p37
dev-java/java-config:     2.1.12
dev-lang/python:          2.7.3-r2, 3.2.3-r1
dev-util/cmake:           2.8.9-r1
dev-util/pkgconfig:       0.27.1
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.10.5
sys-apps/sandbox:         2.6
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.6, 1.12.4
sys-devel/binutils:       2.22.90
sys-devel/gcc:            4.7.2
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.6 (virtual/os-headers)
sys-libs/glibc:           2.15-r3
Repositories: gentoo rion local
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA Oracle-BCLA-JavaSE AdobeFlash-10.3 Nero-EULA-US"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--quiet-build=n"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp.ds.karen.hj.se/gentoo/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/rion /usr/portage/local"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X acl aes-ni alsa amd64 apng avx bzip2 cairo cli consolekit cracklib crypt cups cxx dbus dri dvb egl flac fortran gdbm gif gles gpm hvm iconv icu jpeg kde libnotify lxde lzma mmx mmxext modules mp3 mudflap multilib ncurses nls nptl ogg opengl openmp optimized-qmake pam pcre pgo png policykit pppd pvr qt3support qt4 readline samba session sse sse2 sse3 sse4 sse4_1 sse4_2 sse4a sse5 ssl ssse3 startup-notification svg system-sqlite tcpd threads tiff truetype udev unicode vaapi vdpau vorbis xcb xcomposite xinerama xv xvmc zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby19" SANE_BACKENDS="net" USERLAND="GNU" VIDEO_CARDS="fglrx" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 1 Mads 2012-10-13 20:04:08 UTC
A solution would be to change the script to create a temporary file for each volume, and then call dmsetup create <volname> <tempfile> in the init script. I'm not exactly sure how this should be solved if a temp file is not possible...
Comment 2 Mads 2012-10-16 10:41:33 UTC
I figured out that it was possible to create multi-line volumes in one dmsetup command. The example given in the initial description can be created like:

> echo -e "0 1028160 linear /dev/hda 0\n1028160 3903762 linear /dev/hdb 0" | \ 
> dmsetup create volname /dev/stdin

I haven't managed yet to edit the init script to reflect this yet, I suspect it will need a restructure of the get_new_dm_volumes function.
Comment 3 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2013-01-17 23:43:46 UTC
patches welcome
Comment 4 Mads 2013-01-18 16:25:24 UTC
Created attachment 336058 [details]
new /etc/init.d/device-mapper, non-working

Ah, I forgot about this one, thank you for reminding me!

I kinda got an idea of how it could be solved, but I hit a road block that I might need some help with.

> # /etc/init.d/device-mapper start
> /etc/init.d/device-mapper: line 40: syntax error near unexpected token `<'
> /etc/init.d/device-mapper: line 40: `   done < <(grep -v -e '^[[:space:]]*\(#\|$\)' /etc/dmtab)'
>  * ERROR: device-mapper failed to start

The updated function get_new_dm_volumes tries to create a associative array "dmsetup_cmds" where the keys are volnames:, and values are the finished dmsetup command. The array building cannot take place in a subshell which are done in the original script (line 29&30) because then array variable would not be available in the for loop afterward (not the same process). This can be solved by using process substitution as I have done in the attached script, but OpenRC won't parse it, giving that error message I pasted earlier...

I don't know if my escaping in line 35 is correct, I haven't come so far that I've been able to test it. But it seemingly puts together the dmsetup commands correctly at least...
Comment 5 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2013-01-18 17:26:34 UTC
(In reply to comment #4)
> The updated function get_new_dm_volumes tries to create a associative array
> "dmsetup_cmds" where the keys are volnames:, and values are the finished
> dmsetup command. The array building cannot take place in a subshell which
> are done in the original script (line 29&30) because then array variable
> would not be available in the for loop afterward (not the same process).
> This can be solved by using process substitution as I have done in the
> attached script, but OpenRC won't parse it, giving that error message I
> pasted earlier...
> 
> I don't know if my escaping in line 35 is correct, I haven't come so far
> that I've been able to test it. But it seemingly puts together the dmsetup
> commands correctly at least...
We can't do an associative array, it needs to be compatible with POSIX sh. Bash-only features cannot be used.
Comment 6 Mads 2013-01-18 17:38:21 UTC
I guess it'll be the same with that process substitution technique?

I'll try to rewrite it without fancyness.
Comment 7 Mads 2013-01-20 17:49:52 UTC
Created attachment 336228 [details]
new /etc/init.d/device-mapper, working

I managed to rewrite the init file making it POSIX compliant, I think - but it is a bit ugly - I had to make a new function, build_dmsetup_command, that takes volume name as an argument, and then builds the command from data from /etc/dmtab, making the algorithm complexity something like O(n^3). get_new_dm_volumes are adjusted to only return unique volume names, and no parameters (it still checks if the volume is already set up).

It has lots of greps and awks, so if you know how to clean it up, then feedback is appreciated! But as far as I have tested, it seems to work :)
Comment 8 Mads 2013-01-20 19:48:38 UTC
Created attachment 336244 [details, diff]
new device-mapper init.d in patch form

Here's a patch for the new device-mapper init.d
Comment 9 Mads 2013-01-28 12:11:27 UTC
A typical use case for linear device mapper setup, is if you want to mount dynamic Windows-created extended NTFS volume consisting of several dynamic LDM partitions. I came over a case where an NTFS volume consisted of a large /dev/sda2 extended with a small /dev/sda1-partition. To mount something like that in Linux, you first have to set up device mapper to append sda1 to sda2 - then it will mount just fine with ntfs3g.

In my current setup, I use a local.d-script running dmsetup to combine a 1TB drive with a 3TB drive creating two virtual 2TB drives, and then use those in a raid consisting of other 2TB drives. With this new device-mapper init.d, I could set everything up the right way.

Just thought I should post this FYI, to get some use cases where it is useful.
Comment 10 Mads 2013-03-13 13:12:17 UTC
Even if the algorithm has lousy complexity, I don't think it is that easy to optimize away those recursive grep calls. Maybe if you store the dmtab output in a big variable with another IFS so you can have a multi-line variable, so you don't need those grep-calls parsing /etc/dmtab several times - but I don't know if the environment OpenRC needs to run on has a max size on how big environment variables can be.
Comment 11 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2014-02-02 19:51:39 UTC
InCVS 2.02.105-r2