Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 551532 - sys-apps/systemd-220 - breaks previously working Dracut-built initramfs setup (LUKS encrypted disk + LVM)
Summary: sys-apps/systemd-220 - breaks previously working Dracut-built initramfs setup...
Status: RESOLVED OBSOLETE
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Normal major
Assignee: Gentoo systemd Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-06-09 07:48 UTC by Andrew Udvare
Modified: 2018-02-26 22:10 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Log from emergency console. (rdsosreport.txt.gz,125.51 KB, application/gzip)
2015-06-09 07:49 UTC, Andrew Udvare
Details
systemd-9999 build log (build.log,75.21 KB, text/x-log)
2015-06-09 23:01 UTC, Andrew Udvare
Details
Dracut debug log (rdsosreport-systemd-9999.txt.gz,126.05 KB, application/gzip)
2015-06-11 19:03 UTC, Andrew Udvare
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Udvare 2015-06-09 07:48:58 UTC
I upgraded to systemd 220 and it seems to break my boot up process which was previously working fine on 219. I downgraded, rebuilt the initramfs, and now everything works again.

I would hope 219 is not going to disappear anytime soon from Portage.

Reported to Dracut: https://github.com/haraldh/dracut/issues/70

Reproducible: Always

Steps to Reproduce:
1. Have a configuration like mine for Dracut:
At least one disk encrypted with LUKS
Key stored on external non-encrypted medium

Inside the LUKS encrypted disk: single LVM volume with a partition set up

Configuration for Dracut:
omit_dracutmodules+="gensplash bootchart dash plymouth btrfs mdraid multipath iscsi ndb biosdevname network fcoe-uefi ndb dmraid"
add_dracutmodules+="crypt crypt-gpg dm uefi-lib systemd crypt-loop lvm uvesafb"
add_fstab+="/usr/src/initrd-fstab"
omit_drivers+="nvidia vboxdrv vboxnetflt vboxnetadp"
hostonly="yes"

For this /usr/src/initrd-fstab file:
UUID="XXXX-XXXX" /mnt/somewhere vfat noatime 1 2

2. dracut --force
3. Re-generate GRUB, etc
4. Attempt to boot
Actual Results:  
Fails to boot. Stops at detecting disk containing key.

Expected Results:  
Should boot.

Portage 2.2.20 (python 3.4.1-final-0, default/linux/amd64/13.0/desktop/kde/systemd, gcc-4.8.4, glibc-2.20-r2, 3.18.12-gentoo-limelight x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-3.18.12-gentoo-limelight-x86_64-Intel-R-_Core-TM-_i7-5930K_CPU_@_3.50GHz-with-gentoo-2.2
KiB Mem:    32832212 total,  25820156 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sun, 07 Jun 2015 15:45:01 +0000
sh bash 4.3_p33-r2
ld GNU ld (Gentoo 2.24 p1.4) 2.24
app-shells/bash:          4.3_p33-r2::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.20.2::gentoo
dev-lang/python:          2.7.9-r1::gentoo, 3.4.1::gentoo
dev-util/cmake:           2.8.12.2-r1::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.16.4::gentoo
sys-apps/sandbox:         2.6-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo
sys-devel/automake:       1.11.6-r1::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.24-r3::gentoo
sys-devel/gcc:            4.8.4::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc:           2.20-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://tatsh/gentoo-portage
    priority: -1000

jamiel-gentoo-overlay
    location: /var/lib/layman/jamiel-gentoo-overlay
    masters: gentoo
    priority: 0

tatsh-overlay
    location: /var/lib/layman/tatsh-overlay
    masters: gentoo
    priority: 1

java
    location: /var/lib/layman/java
    masters: gentoo
    priority: 2

vortex
    location: /var/lib/layman/vortex
    masters: gentoo
    priority: 3

srcshelton
    location: /var/lib/layman/srcshelton
    masters: gentoo
    priority: 4

Installed sets: @abiword, @ansible, @bashcomp, @beanstalk, @cdr, @chrome, @clang, @compass, @cups, @dracut, @driverside, @firefox, @gimp, @git, @grub, @ibus, @icedtea, @icewm, @kde4, @kde4-devel, @kdevelop, @libimobiledevice, @media, @misc, @mupen64plus, @mysql, @php, @php-xdebug, @pidgin, @portage-dev, @portage-utilities, @postgresql, @python, @rar, @rk-vpn, @rutorrent, @skype, @ssh, @stepmania, @tatshnet, @vagrant, @virtualbox, @vlc, @wine, @x11, @x11-misc, @xirvik-vpn
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -fomit-frame-pointer -ggdb"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.0/conf"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.5/ext-active/ /etc/php/apache2-php5.6/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=native -fomit-frame-pointer -ggdb"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs candy clean-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.mirrors.pair.com/ rsync://mirrors.rit.edu/gentoo/ http://mirrors.rit.edu/gentoo/ http://gentoo.llarian.net/ rsync://gentoo.mirrors.tds.net/gentoo http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://gentoo.cites.uiuc.edu/pub/gentoo/"
LANG="en_GB.utf8"
LDFLAGS="-Wl,--as-needed"
MAKEOPTS="-j13"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi aes alsa amd64 avx avx2 bash-completion bluetooth branding bzip2 cairo cdio cjk cli cracklib crypt cups cxx dbus declarative dri dts dvd dvdr emboss exif fam ffmpeg flac fma3 fortran gdbm gif glamor gpm iconv idn ipv6 jemalloc jpeg kde kipi lcms libnotify mad mmx mmxext modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit popcnt ppds qt qt3support qt4 readline semantic-desktop session spell sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 svg syslog systemd tcpd tiff truetype udev udisks unicode upower vim-syntax vorbis wxwidgets x264 xcb xcomposite xinerama xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" DRACUT_MODULES="crypt crypt-gpg crypt-loop gensplash lvm ssh-client systemd" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="evdev joystick keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en_GB en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="nvidia v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7 3.4"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

sys-apps/systemd-219_p112::gentoo was built with the following:
USE="acl cryptsetup gudev idn introspection kmod lz4 pam (policykit) seccomp ssl -apparmor -audit -curl -doc -elfutils -gcrypt -http -importd (-kdbus) -lzma -nat -python -qrcode (-selinux) -sysv-utils -terminal -test -vanilla -xkb" ABI_X86="32 64 -x32" PYTHON_SINGLE_TARGET="python2_7 -python3_3 -python3_4" PYTHON_TARGETS="python2_7 python3_4 -python3_3"
Comment 1 Andrew Udvare 2015-06-09 07:49:26 UTC
Created attachment 404808 [details]
Log from emergency console.
Comment 2 Alexandre Rostovtsev (RETIRED) gentoo-dev 2015-06-09 20:30:30 UTC
Please give full versions with revisions, not just "220" and "219". It matters because some encryption-related bugs were present in 220 and fixed in 220-r1, for example.

Also please attach the output of "emerge --info systemd".
Comment 3 Alexandre Rostovtsev (RETIRED) gentoo-dev 2015-06-09 20:31:49 UTC
(In reply to Alexandre Rostovtsev from comment #2)
> Also please attach the output of "emerge --info systemd".

Ah, never mind, you already did that. Sorry for failing to read :/
Comment 4 Mike Gilbert gentoo-dev 2015-06-09 21:18:55 UTC
Copying dracut maintainers.
Comment 5 Mike Gilbert gentoo-dev 2015-06-09 21:27:09 UTC
(In reply to Andrew Udvare from comment #0)

If you're up for some testing, please give systemd-9999 a try. If that works, at least we know that the problem has been resolved upstream and we can look for a patch to backport.
Comment 6 Andrew Udvare 2015-06-09 21:48:47 UTC
(In reply to Mike Gilbert from comment #5)
> (In reply to Andrew Udvare from comment #0)
> 
> If you're up for some testing, please give systemd-9999 a try. If that
> works, at least we know that the problem has been resolved upstream and we
> can look for a patch to backport.

Will do in a little bit and report.
Comment 7 Andrew Udvare 2015-06-09 23:01:58 UTC
Created attachment 404856 [details]
systemd-9999 build log

I was unable to build systemd-9999 and so I am attaching the log. Is it broken at commit 3a3701f03360f9a292dcab31fda1f6e0e0629c64? Is there something else I can try?
Comment 8 Mike Gilbert gentoo-dev 2015-06-10 01:41:43 UTC
I have made some updates to the ebuild which should resolve that build failure.

+  10 Jun 2015; Mike Gilbert <floppym@gentoo.org> systemd-9999.ebuild:
+  Drop references to gudev, gtk-doc, and gobject-introspection.
Comment 9 Andrew Udvare 2015-06-10 01:46:25 UTC
(In reply to Mike Gilbert from comment #8)
> I have made some updates to the ebuild which should resolve that build
> failure.
> 
> +  10 Jun 2015; Mike Gilbert <floppym@gentoo.org> systemd-9999.ebuild:
> +  Drop references to gudev, gtk-doc, and gobject-introspection.

How can I get that newer ebuild? https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-apps/systemd/ doesn't show your commit (yet)
Comment 10 Mike Gilbert gentoo-dev 2015-06-10 02:41:21 UTC
(In reply to Andrew Udvare from comment #9)
> How can I get that newer ebuild?
> https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-apps/systemd/
> doesn't show your commit (yet)

In general, wait an hour and run emerge --sync. It takes rsync some time to catch up to CVS.

It should be updated now.
Comment 11 Andrew Udvare 2015-06-10 03:27:37 UTC
If I am updated, I am now getting this issue:

# emerge -av =sys-apps/systemd-9999

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U *] sys-apps/systemd-9999:0/2::gentoo [219_p112:0/2::gentoo] USE="acl cryptsetup idn kmod lz4 pam (policykit) seccomp ssl -apparmor -audit -curl -elfutils -gcrypt -gnuefi% -http -importd (-kdbus) -lzma -nat -python -qrcode (-selinux) -sysv-utils -terminal {-test} -vanilla -xkb (-doc%) (-gudev%*) (-introspection%*)" ABI_X86="32 (64) (-x32)" PYTHON_SINGLE_TARGET="python2_7 -python3_3 -python3_4" PYTHON_TARGETS="python2_7 python3_4 -python3_3" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

sys-apps/systemd:0

  (sys-apps/systemd-9999:0/2::gentoo, ebuild scheduled for merge) pulled in by
    =sys-apps/systemd-9999 (Argument)

  (sys-apps/systemd-219_p112:0/2::gentoo, installed) pulled in by
    >=sys-apps/systemd-212-r5:0/2[abi_x86_32(-)?,abi_x86_64(-)?,abi_x86_x32(-)?,abi_mips_n32(-)?,abi_mips_n64(-)?,abi_mips_o32(-)?,abi_ppc_32(-)?,abi_ppc_64(-)?,abi_s390_32(-)?,abi_s390_64(-)?,gudev,introspection?] required by (virtual/libgudev-215-r1:0/0::gentoo, installed)
                                                                                                                                                                                                 ^^^^^ ^^^^^^^^^^^^^^

I have masked 220 and unmasked with **.

# equery d libgudev
 * These packages depend on libgudev:
gnome-base/gvfs-1.22.3 (virtual/libgudev)
media-gfx/gimp-2.8.14 (udev ? virtual/libgudev)
sys-fs/udisks-1.0.5-r1 (virtual/libgudev)
sys-fs/udisks-2.1.4 (virtual/libgudev)
sys-power/upower-pm-utils-0.9.23-r2 (kernel_linux ? virtual/libgudev)
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-06-10 04:07:09 UTC
You need to upgrade virtual/libgudev to 215-r2 first.
Comment 13 Andrew Udvare 2015-06-10 08:31:50 UTC
(In reply to Michał Górny from comment #12)
> You need to upgrade virtual/libgudev to 215-r2 first.

I did this, but I am still getting an error about the introspection USE flag. I cannot see what is pulling this other than a lot of packages having set +introspection. I even built virtual/libgudev with -introspection but that did not help.

None of my configurations mentions adding this USE flag.

# emerge -av =sys-apps/systemd-9999

These are the packages that would be merged, in order:

Calculating dependencies                     ... done!
[ebuild     U *] sys-apps/systemd-9999:0/2::gentoo [219_p112:0/2::gentoo] USE="acl cryptsetup idn kmod lz4 pam (policykit) seccomp ssl -apparmor -audit -curl -elfutils -gcrypt -gnuefi% -http -importd (-kdbus) -lzma -nat -python -qrcode (-selinux) -sysv-utils -terminal {-test} -vanilla -xkb (-doc%) (-gudev%*) (-introspection%*)" ABI_X86="32 (64) (-x32)" PYTHON_SINGLE_TARGET="python2_7 -python3_3 -python3_4" PYTHON_TARGETS="python2_7 python3_4 -python3_3" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

sys-apps/systemd:0

  (sys-apps/systemd-9999:0/2::gentoo, ebuild scheduled for merge) pulled in by
    =sys-apps/systemd-9999 (Argument)

  (sys-apps/systemd-219_p112:0/2::gentoo, installed) pulled in by
    >=sys-apps/systemd-212-r5:0/2[abi_x86_32(-)?,abi_x86_64(-)?,abi_x86_x32(-)?,abi_mips_n32(-)?,abi_mips_n64(-)?,abi_mips_o32(-)?,abi_ppc_32(-)?,abi_ppc_64(-)?,abi_s390_32(-)?,abi_s390_64(-)?,gudev(-),introspection?] required by (virtual/libgudev-215-r2:0/0::gentoo, installed)
                                                                                                                                                                                                                  ^^^^^^^^^^^^^^
Comment 14 Andrew Udvare 2015-06-10 08:40:05 UTC
When I use @world, I get this:

WARNING: One or more updates/rebuilds have been skipped due to a dependency conflict:

sys-apps/systemd:0

  (sys-apps/systemd-9999:0/2::gentoo, ebuild scheduled for merge) conflicts with
    >=sys-apps/systemd-212-r5:0/2[abi_x86_64(-),gudev(-)] required by (virtual/libgudev-215-r2:0/0::gentoo, installed)
Comment 15 Andrew Udvare 2015-06-11 16:06:56 UTC
(In reply to Michał Górny from comment #12)
> You need to upgrade virtual/libgudev to 215-r2 first.

I am still stuck unable to build systemd-9999 with that fix among other changes in configuration.
Comment 16 Andrew Udvare 2015-06-11 16:07:43 UTC
Sorry not build. I meant to say that Portage will not let it be emerged due to some conflict I have not figured out how to resolve.
Comment 17 Mike Gilbert gentoo-dev 2015-06-11 18:20:49 UTC
As of right now, the only way I can get it to work is to explicitly install libgudev.

emerge --oneshot sys-apps/systemd dev-libs/libgudev virtual/libgudev
Comment 18 Andrew Udvare 2015-06-11 19:02:41 UTC
(In reply to Mike Gilbert from comment #17)
> As of right now, the only way I can get it to work is to explicitly install
> libgudev.
> 
> emerge --oneshot sys-apps/systemd dev-libs/libgudev virtual/libgudev

Thanks. That worked. But I had to set USE="-introspection".

Unfortunately, booting up from systemd-9999 did not work. Git commit is 4061eed4d91226f1f75e79350c5e1bdd4529d578 .

Also note that this is a new version of Dracut 041-r3 but I do not think that is to blame because it was broken with the old version.
Comment 19 Andrew Udvare 2015-06-11 19:03:20 UTC
Created attachment 404984 [details]
Dracut debug log
Comment 20 Andrew Udvare 2015-06-11 19:15:38 UTC
Thinking to do a bit of git bisect'ing. What is the commit hash for systemd-219_p112?
Comment 21 Mike Gilbert gentoo-dev 2015-06-11 21:33:21 UTC
(In reply to Andrew Udvare from comment #20)
> Thinking to do a bit of git bisect'ing. What is the commit hash for
> systemd-219_p112?

It's 85a6fabdd3e43cfab0fc6359e9f2a9e368d4a3ed from the v219-stable branch -- not master.

You would probably be better off bisecting from the v219 tag to the v220 tag.
Comment 22 Alexander Tsoy 2015-06-11 21:50:58 UTC
(In reply to Andrew Udvare from comment #0)
> add_fstab+="/usr/src/initrd-fstab"

mount-sys.sh from fstab-sys module is a pre-pivot hook that gets executed *after* root is mounted. So I have no idea how this setup worked for you.

Please generate working initramfs with an older systemd version, add "rd.break rd.debug systemd.log_level=debug" to the kernel cmdline and attach resulting rdsosreport.txt.
Comment 23 Andrew Udvare 2015-06-12 01:14:23 UTC
(In reply to Alexander Tsoy from comment #22)
> (In reply to Andrew Udvare from comment #0)
> > add_fstab+="/usr/src/initrd-fstab"
> 
> mount-sys.sh from fstab-sys module is a pre-pivot hook that gets executed
> *after* root is mounted. So I have no idea how this setup worked for you.
> 
> Please generate working initramfs with an older systemd version, add
> "rd.break rd.debug systemd.log_level=debug" to the kernel cmdline and attach
> resulting rdsosreport.txt.

That only applies if not using systemd. From mount-sys.sh:

# systemd will mount and run fsck from /etc/fstab and we don't want to
# run into a race condition.
if [ -z "$DRACUT_SYSTEMD" ]; then
    [ -f /etc/fstab ] && fstab_mount /etc/fstab
fi
Comment 24 Alexander Tsoy 2015-06-12 10:38:58 UTC
(In reply to Andrew Udvare from comment #23)
You're right. systemd-fstab-generator parses /etc/fstab even inside initrd.
Comment 25 Alexander Tsoy 2015-06-12 11:06:03 UTC
(In reply to Andrew Udvare from comment #0)
> For this /usr/src/initrd-fstab file:
> UUID="XXXX-XXXX" /mnt/somewhere vfat noatime 1 2

What if you change sixth field to "0"? After commit 4dda4e637e4c17a14db6cd265f36f5e8a5050367 all fsck units are ordered after systemd-fsck-root.service which was not generated inside initrd in <systemd-220.
Comment 26 Andrew Udvare 2015-06-12 20:28:48 UTC
(In reply to Alexander Tsoy from comment #25)
> (In reply to Andrew Udvare from comment #0)
> > For this /usr/src/initrd-fstab file:
> > UUID="XXXX-XXXX" /mnt/somewhere vfat noatime 1 2
> 
> What if you change sixth field to "0"? After commit
> 4dda4e637e4c17a14db6cd265f36f5e8a5050367 all fsck units are ordered after
> systemd-fsck-root.service which was not generated inside initrd in
> <systemd-220.

Will try this and report.

Also I will post the SOS log of the working initrd breaking before switch-root.
Comment 27 Andrew Udvare 2015-06-15 14:00:15 UTC
This bug may be related: https://bugs.freedesktop.org/show_bug.cgi?id=90913

I have not had time to try bisecting but I will soon. This is what it looks like (if anyone thinks I should start at another commit please comment):

Bisecting: 358 revisions left to test after this (roughly 9 steps)
[42f1ab5009eed71f0d4f83681b8fdbed8664fca3] man: add sd_event_{run,wait,prepare,dispatch,loop}

Bisecting: 179 revisions left to test after this (roughly 8 steps)
[9e4ded3064e9a683e004ff8f6a8ce53ac20b79d7] build-sys: generate CLEANFILES from EXTRA_DIST

Bisecting: 89 revisions left to test after this (roughly 7 steps)
[858a109f4a336be6c258ae615d31651347b9b67b] machined: fix check if host directory could be opened

Bisecting: 44 revisions left to test after this (roughly 6 steps)
[0377e373d1b4973effe14ca19e21f0c10740085d] systemd-sysv-generator test: Adjust to dropped runlevelN.target mapping

Bisecting: 21 revisions left to test after this (roughly 5 steps)
[19e887e709c31ee4366ec44a770d3963cd48cb86] systemd-fsck: always connect to systemd-fsckd

Bisecting: 10 revisions left to test after this (roughly 4 steps)
[0370612e0522191f929e3feb7d4937fff3d421e2] machined: make "machinectl copy-to" and "machinectl copy-from" server side operations

Bisecting: 5 revisions left to test after this (roughly 3 steps)
[0974a682d155a5874123ba7de9c1e314c6681e0f] bootctl: add sd-boot support

Bisecting: 2 revisions left to test after this (roughly 1 step)
[32c3d7144cf9a5c8c03761d7f198142ca0f5f7b8] journal-remote: fix client_cert memory leak

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[9c3cf9693ac5c0a332ba376f99e6adea28b1bb0d] journal-remote: fix certificate status memory leak
Comment 28 Alexander Tsoy 2015-06-15 20:24:21 UTC
(In reply to Andrew Udvare from comment #27)
> This bug may be related: https://bugs.freedesktop.org/show_bug.cgi?id=90913

No, it's a different bug.

> I have not had time to try bisecting but I will soon. This is what it looks
> like (if anyone thinks I should start at another commit please comment):

I still believe that commit 4dda4e637e4c17a14db6cd265f36f5e8a5050367 is the culprit. Have you already tried to disable fsck for partition containing the key file?
Comment 29 Andrew Udvare 2015-06-16 00:00:08 UTC
> I still believe that commit 4dda4e637e4c17a14db6cd265f36f5e8a5050367 is the
> culprit. Have you already tried to disable fsck for partition containing the
> key file?

That is in the fstab I am adding correct?

Currently mine is like so:
UUID="XXXX-XXXX" /mnt/somewhere vfat noatime 1 2
Comment 30 Alexander Tsoy 2015-06-16 02:53:11 UTC
(In reply to Andrew Udvare from comment #29)
> > I still believe that commit 4dda4e637e4c17a14db6cd265f36f5e8a5050367 is the
> > culprit. Have you already tried to disable fsck for partition containing the
> > key file?
> 
> That is in the fstab I am adding correct?
> 
> Currently mine is like so:
> UUID="XXXX-XXXX" /mnt/somewhere vfat noatime 1 2

Yes. If sixth fild != 0 then systemd-fstab-generator generates two units:
- systemd-fsck@dev-disk-by\x2duuid-XXXX\x2dXXXX.service with dependency After=systemd-fsck-root.service
- mnt-somewhere.mount with dependency After=systemd-fsck@dev-disk-by\x2duuid-XXXX\x2dXXXX.service

<systemd-220 doesn't generate systemd-fsck-root.service, so the above two units can be started before fsck on the root device.
systemd-220 generates systemd-fsck-root.service, so fsck and mount units for /mnt/somewhere are ordered after fsck unit for the root device. The latter depends on the luks device which in turn needs a key file from /mnt/somewhere. %)
Comment 31 Andrew Udvare 2015-06-20 06:13:59 UTC
(In reply to Alexander Tsoy from comment #30)
> Yes. If sixth fild != 0 then systemd-fstab-generator generates two units:
> - systemd-fsck@dev-disk-by\x2duuid-XXXX\x2dXXXX.service with dependency
> After=systemd-fsck-root.service
> - mnt-somewhere.mount with dependency
> After=systemd-fsck@dev-disk-by\x2duuid-XXXX\x2dXXXX.service
> 
> <systemd-220 doesn't generate systemd-fsck-root.service, so the above two
> units can be started before fsck on the root device.
> systemd-220 generates systemd-fsck-root.service, so fsck and mount units for
> /mnt/somewhere are ordered after fsck unit for the root device. The latter
> depends on the luks device which in turn needs a key file from
> /mnt/somewhere. %)

I think this fixed it. I set the sixth field to 0 and booted fine. No bisecting or more debugging with 9999 needed.

I extracted the fstab file, after successful boot, to confirm that the sixth field was 0 in in the initramfs image and it is.

So it appears if you have a service that depends on mounting a device before while in initramfs, you must not allow file system checks on that device. This is a bit unfortunate IMHO.

The only way around I presume is to manually modify the initramfs image after and set the dependencies in the services affected.
Comment 32 Mike Gilbert gentoo-dev 2015-07-17 15:02:45 UTC
(In reply to Andrew Udvare from comment #31)
> So it appears if you have a service that depends on mounting a device before
> while in initramfs, you must not allow file system checks on that device.
> This is a bit unfortunate IMHO.

I would call it a bug, not an intended behavior.
Comment 33 Alexander Tsoy 2015-07-21 23:12:52 UTC
So if I understand correctly we need following changes:
1. If partition is mounted outside /sysroot in initramfs, then corresponding fsck service should not have After=systemd-fsck-root.service
2. That partition should be unmounted before switching to real root (Conflicts=initrd-switch-root.target? bug 554698).
Comment 34 Andrew Udvare 2015-07-22 03:10:31 UTC
(In reply to Alexander Tsoy from comment #33)
> So if I understand correctly we need following changes:
> 1. If partition is mounted outside /sysroot in initramfs, then corresponding
> fsck service should not have After=systemd-fsck-root.service
> 2. That partition should be unmounted before switching to real root
> (Conflicts=initrd-switch-root.target? bug 554698).

#2 is the main thing (safe unmount prior to switch root). As for #1, it should be possible and safe IMHO to fsck within the chroot prior to mounting (removing my prior need for the entry in real root /etc/fstab). This does not seem to work, and as with bug 554698, real root will not allow this either. I have no idea if this is just vfat related.
Comment 35 Andrew Udvare 2015-07-22 03:10:56 UTC
(In reply to Alexander Tsoy from comment #33)
> So if I understand correctly we need following changes:
> 1. If partition is mounted outside /sysroot in initramfs, then corresponding
> fsck service should not have After=systemd-fsck-root.service
> 2. That partition should be unmounted before switching to real root
> (Conflicts=initrd-switch-root.target? bug 554698).

#2 is the main thing (safe unmount prior to switch root). As for #1, it should be possible and safe IMHO to fsck within the chroot prior to mounting (removing my prior need for the entry in real root /etc/fstab). This does not seem to work, and as with bug 554698, real root will not allow this either. I have no idea if this is just vfat related.
Comment 36 Pacho Ramos gentoo-dev 2015-09-07 18:50:51 UTC
What is the status of this with systemd-225?
Comment 37 Andrew Udvare 2015-09-07 19:13:34 UTC
I have not tried reverting back to the old setting without nofail in fstab (in the initramfs fstab). Basically, machine boots, finds USB partition with key, mounts it, decrypts, then the root switches without an unmount before. Yes this leaves the partition in dirty state.