Bug 578350

Summary:	sys-apps/systemd-229 - systemd-remount-fs.service (systemd-udevd) '/sbin/dmsetup splitname --nameprefixes --noheadings --rows vg0-var' failed
Product:	Gentoo Linux	Reporter:	Juergen Rose <rose>
Component:	[OLD] Core system	Assignee:	Gentoo systemd Team <systemd>
Status:	RESOLVED TEST-REQUEST
Severity:	normal
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
Whiteboard:
Package list:		Runtime testing required:	---
Attachments:	The output of 'journalctl -xb'

Description Juergen Rose 2016-03-27 11:24:49 UTC

During booting I get several messages related to a failing unmounting of the /var filesystem. As the result the multiuser target is not reached and all logical volumens with except of /dev/mapper/vg0-var is not mounted.

When I login just after booting, I see that four processes accessing (two journald, cron and rpc) after killing these processes I can unmount (from /var) and check the filesystem of /dev/mapper/vg0-var. It is reported to be clean.
If I boot with an USB stick I can check the root filesystem, which is also clean.
Smartmontools report that all my disk are without any problems. But the BIOS has sometimes problems to detect my harddisk. But if the disks are detected by the BIOS, I can boot and see the problem with the /var filesystem.

If I grep 'journal -xb' for dmsetup and mount, I see the following:

rose@lynx:/home/rose/Txt/Problems(6)$ grep "dmsetup\|mount" journalctl_-xb__unmount_var_failed__impala2.txt
Mar 25 10:14:23 impala kernel: mount (530) used greatest stack depth: 14184 bytes left
Mar 25 10:14:23 impala kernel: EXT4-fs (md126): mounted filesystem with ordered data mode. Opts: (null)
Mar 25 10:14:23 impala systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
Mar 25 10:14:24 impala systemd[1]: Starting Remount Root and Kernel File Systems...
-- Subject: Unit systemd-remount-fs.service has begun start-up
-- Unit systemd-remount-fs.service has begun starting up.
Mar 25 10:14:24 impala kernel: EXT4-fs (md126): re-mounted. Opts: (null)
Mar 25 10:14:24 impala systemd[1]: Started Remount Root and Kernel File Systems.
-- Subject: Unit systemd-remount-fs.service has finished start-up
-- Unit systemd-remount-fs.service has finished starting up.
-- Subject: Unit mnt-rescue2.mount has begun start-up
-- Unit mnt-rescue2.mount has begun starting up.
-- Subject: Unit usr-local-portage-distfiles_impala.mount has begun start-up
-- Unit usr-local-portage-distfiles_impala.mount has begun starting up.
-- Subject: Unit mnt-rescue.mount has begun start-up
-- Unit mnt-rescue.mount has begun starting up.
-- Subject: Unit tmp.mount has begun start-up
-- Unit tmp.mount has begun starting up.
Mar 25 10:14:25 impala systemd[1]: usr-portage_impala.mount: Directory /usr/portage_impala to mount over is not empty, mounting anyway.
-- The directory /usr/portage_impala is specified as the mount point (second field in
-- This does not interfere with mounting, but the pre-exisiting files in
-- this directory become inaccessible. To see those over-mounted files,
-- please manually mount the underlying file system to a secondary
-- Subject: Unit usr-portage_impala.mount has begun start-up
-- Unit usr-portage_impala.mount has begun starting up.
-- Subject: Unit usr-local-portage-distfiles_impala.mount has finished start-up
-- Unit usr-local-portage-distfiles_impala.mount has finished starting up.
-- Subject: Unit mnt-rescue.mount has finished start-up
-- Unit mnt-rescue.mount has finished starting up.
-- Subject: Unit mnt-rescue2.mount has finished start-up
-- Unit mnt-rescue2.mount has finished starting up.
-- Subject: Unit data_impala.mount has begun start-up
-- Unit data_impala.mount has begun starting up.
-- Subject: Unit usr-src_impala.mount has begun start-up
-- Unit usr-src_impala.mount has begun starting up.
-- Subject: Unit home_impala.mount has begun start-up
-- Unit home_impala.mount has begun starting up.
Mar 25 10:14:25 impala kernel: EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr
-- Subject: Unit tmp.mount has finished start-up
-- Unit tmp.mount has finished starting up.
Mar 25 10:14:25 impala kernel: EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: user_xattr
Mar 25 10:14:25 impala kernel: EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr
-- Subject: Unit home_impala.mount has finished start-up
-- Unit home_impala.mount has finished starting up.
Mar 25 10:14:25 impala kernel: EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: user_xattr
-- Subject: Unit usr-src_impala.mount has finished start-up
-- Unit usr-src_impala.mount has finished starting up.
-- Subject: Unit data_impala.mount has finished start-up
-- Unit data_impala.mount has finished starting up.
-- Subject: Unit opt.mount has begun start-up
-- Unit opt.mount has begun starting up.
-- Subject: Unit home_impala-ftp.mount has begun start-up
-- Unit home_impala-ftp.mount has begun starting up.
-- Subject: Unit usr-portage_impala.mount has finished start-up
-- Unit usr-portage_impala.mount has finished starting up.
Mar 25 10:14:25 impala kernel: EXT4-fs (dm-4): mounting ext3 file system using the ext4 subsystem
Mar 25 10:14:25 impala kernel: EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null)
-- Subject: Unit opt.mount has finished start-up
-- Unit opt.mount has finished starting up.
-- Subject: Unit var.mount has begun start-up
-- Unit var.mount has begun starting up.
Mar 25 10:14:26 impala kernel: EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
Mar 25 10:14:26 impala kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr
-- Subject: Unit var.mount has finished start-up
-- Unit var.mount has finished starting up.
-- Subject: Unit var-tmp.mount has begun start-up
-- Unit var-tmp.mount has begun starting up.
Mar 25 10:14:34 impala kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr
Mar 25 10:14:34 impala kernel: XFS (dm-7): Ending clean mount
Mar 25 10:14:34 impala kernel: XFS (dm-7): Unmounting Filesystem
-- Subject: Unit var-tmp.mount has finished start-up
-- Unit var-tmp.mount has finished starting up.
-- Subject: Unit home_impala-ftp.mount has finished start-up
-- Unit home_impala-ftp.mount has finished starting up.
Mar 25 10:14:28 impala systemd-udevd[2584]: fork of '/sbin/dmsetup splitname --nameprefixes --noheadings --rows vg0-var' failed: Resource temporarily unavailable
Mar 25 10:14:28 impala systemd-udevd[2589]: fork of '/sbin/dmsetup splitname --nameprefixes --noheadings --rows vg0-vtmp' failed: Resource temporarily unavailable
Mar 25 10:14:28 impala systemd-udevd[2584]: fork of '/sbin/dmsetup splitname --nameprefixes --noheadings --rows vg0-portage' failed: Resource temporarily unavailable
Mar 25 10:14:28 impala systemd-udevd[2589]: fork of '/sbin/dmsetup splitname --nameprefixes --noheadings --rows vg0-distfiles' failed: Resource temporarily unavailable
Mar 25 10:14:34 impala umount[3342]: umount: /var: target is busy
...
Mar 25 10:14:29 impala systemd[1]: Unmounting /var...
-- Subject: Unit var.mount has begun shutting down
-- Unit var.mount has begun shutting down.
Mar 25 10:14:29 impala systemd[1]: var.mount: Mount process exited, code=exited status=32
Mar 25 10:14:29 impala systemd[1]: Failed unmounting /var.
-- Subject: Unit var.mount has finished shutting down
-- Unit var.mount has finished shutting down.
Mar 25 10:14:29 impala systemd[1]: var.mount: Unit is bound to inactive unit dev-mapper-vg0\x2dvar.device. Stopping, too.
Mar 25 10:14:29 impala systemd[1]: Unmounting /var
...
Mar 25 10:14:29 impala systemd[1]: var.mount: Unit is bound to inactive unit dev-mapper-vg0\x2dvar.device, but not stopping since we tried this too often recently.
...

root@impala:/root(31)# systemctl list-units --all --type=target| grep "basic\|graphical\|local-fs\|multi-user\|sound.target"
basic.target loaded inactive dead Basic System
graphical.target loaded inactive dead Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded inactive dead Local File Systems
multi-user.target loaded inactive dead Multi-User System
sound.target loaded inactive dead Sound Card

Comment 1 Juergen Rose 2016-03-27 11:29:10 UTC

If I do a 'mount -a' after login, all my logical volumes are mounted. And I can rather normally work with the system. I see then the following properties about my physical and logical volumes:

 root@impala:/root(32)# cat /proc/mdstat 
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4] 
md125 : active raid1 sda1[0] sdb1[1]
      131008 blocks [2/2] [UU]
      
md126 : active raid1 sda3[0] sdb3[1]
      134217664 blocks [2/2] [UU]
      
md127 : active raid1 sda6[0] sdb6[1]
      817219152 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
root@impala:/root(33)# lvscan
  ACTIVE            '/dev/vg0/var' [4.00 GiB] inherit
  ACTIVE            '/dev/vg0/vtmp' [32.00 GiB] inherit
  ACTIVE            '/dev/vg0/portage' [4.00 GiB] inherit
  ACTIVE            '/dev/vg0/distfiles' [32.00 GiB] inherit
  ACTIVE            '/dev/vg0/opt' [8.00 GiB] inherit
  ACTIVE            '/dev/vg0/home' [256.00 GiB] inherit
  ACTIVE            '/dev/vg0/tmp' [16.00 GiB] inherit
  ACTIVE            '/dev/vg0/ftp' [128.00 GiB] inherit
  ACTIVE            '/dev/vg0/src' [64.00 GiB] inherit
  ACTIVE            '/dev/vg0/data' [235.30 GiB] inherit
root@impala:/root(34)# grep /dev/vg0 /etc/fstab
root@impala:/root(35)# grep vg0 /etc/fstab
#/dev/mapper/vg0-usr                       /usr                                ext3      defaults,user_xattr    1 2
/dev/mapper/vg0-var                        /var                                ext3      noatime,user_xattr     1 2
/dev/mapper/vg0-vtmp                       /var/tmp                            ext4              noatime,user_xattr     1 2
/dev/mapper/vg0-portage                    /usr/portage_impala                 ext2              noatime,user_xattr     1 2
/dev/mapper/vg0-distfiles                  /usr/local/portage/distfiles_impala ext2      noatime,user_xattr     1 2
/dev/mapper/vg0-opt                              /opt                                ext3                defaults                                       1 2
/dev/mapper/vg0-home                             /home_impala                        ext4      noatime,user_xattr       1 2
/dev/mapper/vg0-tmp                        /tmp                                ext4              noatime,user_xattr     1 2
/dev/mapper/vg0-ftp                        /home_impala/ftp                    xfs       noatime                                        1 2
/dev/mapper/vg0-src                        /usr/src_impala                     ext4              noatime,user_xattr     1 2
/dev/mapper/vg0-data                       /data_impala                        ext4      noatime,user_xattr     1 2
root@impala:/root(36)# pvscan
  PV /dev/md127   VG vg0             lvm2 [779.36 GiB / 60.00 MiB free]
  Total: 1 [779.36 GiB] / in use: 1 [779.36 GiB] / in no VG: 0 [0   ]

Comment 2 Juergen Rose 2016-03-27 11:31:22 UTC

Created attachment 429102 [details]
The output of 'journalctl -xb'

Comment 3 Mike Gilbert gentoo-dev

2016-03-27 15:14:32 UTC

What are you using for an initramfs?

Comment 4 Juergen Rose 2016-03-27 15:28:35 UTC

(In reply to Mike Gilbert from comment #3)
> What are you using for an initramfs?

I am using the initramfs generated by genkernel.

Comment 5 Mike Gilbert gentoo-dev

2016-03-27 16:34:57 UTC

(In reply to Juergen Rose from comment #4)
> I am using the initramfs generated by genkernel.

Please try dracut; genkernel has been quite buggy with systemd.

Comment 6 Juergen Rose 2016-03-28 11:35:25 UTC

(In reply to Mike Gilbert from comment #5)
> (In reply to Juergen Rose from comment #4)
> > I am using the initramfs generated by genkernel.
> 
> Please try dracut; genkernel has been quite buggy with systemd.

Could you be a little bit more specific. I am using genkernel on about ten system with systemd. And do not see any error at least on nine systems.

Comment 7 Mike Gilbert gentoo-dev

2016-03-28 14:33:16 UTC

(In reply to Juergen Rose from comment #6)

Basically, I'm not going to waste time debugging genkernel. If it works with a different initramfs I will re-assign this bug to the genkernel team and let them figure it out.

I do not know if there is a specific problem with it, but it has exhibited strange behavior with lvm/device-mapper volumes in the past.

Comment 8 Pacho Ramos gentoo-dev

2016-04-02 12:57:34 UTC

Also, in the past genkernel-next was created as there were some problems with genkernel using mdev: 
https://bugs.gentoo.org/show_bug.cgi?id=468942

But I don't remember the details as some years have passed and I am not aware of current genkernel status :/