When using a btrfs root filesystem, dracut, and systemd, btrfs will do a full filesystem check on each boot. On machines with large filesystems, this can take a while (2+ hours on my 17 TB root filesystem). When using systemd and dracut, this check will time out after 1.5 minutes, and you must boot manually from the emergency shell. In my case, the filesystem check is always clean when I run a manual check (btrfs check /dev/myroot), and my fstab is set to not check the root partition on boot. This issue appears to be similar to this one: https://bugs.launchpad.net/ubuntu/+source/btrfs-tools/+bug/750209 One suggested workaround is to symlink /sbin/fsck.btrfs to /bin/true in the initrd (I have not tried this yet, but I'm fairly certain this would allow me to boot without manual intervention). As requested, my emerge --info: gorgon ~ # emerge --info Portage 2.2.14 (python 2.7.9-final-0, default/linux/amd64/13.0/desktop/kde/systemd, gcc-4.8.3, glibc-2.19-r1, 3.17.7-gentoo x86_64) ================================================================= System uname: Linux-3.17.7-gentoo-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.2 KiB Mem: 16067936 total, 105040 free KiB Swap: 16356348 total, 16355840 free Timestamp of tree: Wed, 18 Feb 2015 05:00:01 +0000 ld GNU ld (Gentoo 2.24 p1.4) 2.24 app-shells/bash: 4.2_p53 dev-java/java-config: 2.2.0 dev-lang/perl: 5.18.2-r2 dev-lang/python: 2.7.9-r1, 3.3.5-r1, 3.4.1 dev-util/cmake: 2.8.12.2-r1 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.13.9 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.11.6-r1, 1.12.6, 1.13.4 sys-devel/binutils: 2.24-r3 sys-devel/gcc: 4.7.3-r1, 4.8.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.4 sys-devel/make: 4.0-r1 sys-kernel/linux-headers: 3.16 (virtual/os-headers) sys-libs/glibc: 2.19-r1 Repositories: gentoo ixit local ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=core-avx2" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/bind /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=core-avx2" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/ixit /usr/local/portage" USE="X a52 aac acl acpi amd64 avx avx2 bash-completion berkdb branding bzip2 cairo caps cdda cli cracklib crypt cryptsetup cscope cups cxx dbus declarative dri dts emboss encode exif fam ffmpeg firefox flac fortran gdbm gif glamor gles gles1 gles2 gnutls gpm gssapi gtk iconv infiniband ipv6 java jpeg kde kerberos kipi lcms ldap libav libedit libnotify mad mmx mmxext mng modules mp3 mp4 mpeg multilib mysql ncurses nls nptl ogg opengl openmp openvg pam pango pch pcre pdf perl phonon plasma plymouth png policykit ppds python qt3support qt4 readline sasl sdl semantic-desktop session spell sse sse2 sse3 sse4_1 ssl ssse3 startup-notification subversion svg syslog systemd tcpd threads tiff truetype udev udisks unicode upower urandom usb vaapi vdpau vim-syntax vorbis wxwidgets x264 xcb xcomposite xinerama xinetd xml xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_us" NGINX_MODULES_HTTP="access auth_basic autoindex browser charset dav empty_gif fancyindex fastcgi geo gzip limit_conn limit_req map memcached perl proxy referer rewrite scgi split_clients ssi upload upload_progress upstream_ip_hash userid uwsgi" NGINX_MODULES_MAIL="imap smtp" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5 php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON
/sbin/fsck.btrfs does not exist in early versions of btrfs-progs. In the latest versions, it is a shell script that does nothing. Have you done something strange on your system? Like linking it to btrfsck?
(In reply to Mike Gilbert from comment #1) > /sbin/fsck.btrfs does not exist in early versions of btrfs-progs. In the > latest versions, it is a shell script that does nothing. > > Have you done something strange on your system? Like linking it to btrfsck? You're right - btrfs.fsck is a shell script that does nothing. The "workaround" I posted above might have worked for Ubuntu, but does not work in Gentoo. Reading the comments in the shell script, apparently by default dracut initrd calls a filesystem consistency check on every boot: --- # fsck.btrfs is a type of utility that should exist for any filesystem and is # called during system setup when the corresponding /etc/fstab entries contain # non-zero value for fs_passno. (See fstab(5) for more.) # # Traditional filesystems need to run their respective fsck utility in case the # filesystem was not unmounted cleanly and the log needs to be replayed before # mount. This is not needed for BTRFS. You should set fs_passno to 0. # # If you wish to check the consistency of a BTRFS filesystem or repair a # damaged filesystem, see btrfs(8) subcommand 'check'. By default the # filesystem consistency is checked, the repair mode is enabled via --repair # option (use with care!). --- The image dracut generates to boot with has no fstab, so there is no fs_passno to set (or detect). I'm guessing the default value is 1 for anything not defined.
(In reply to Richard Ostrow from comment #2) I am using btrfs root + dracut + systemd, and I do not experience this problem. In the initramfs image, /sbin/fsck.btrfs is the same dummy shell script that exists on my real root filesystem. So, even though systemd may try to invoke it, it does noting and returns immediately. What is /sbin/fsck.btrfs in your generated initramfs image? If it is a real fsck binary, something has gone horribly wrong.
(In reply to Richard Ostrow from comment #2) > You're right - btrfs.fsck is a shell script that does nothing. The > "workaround" I posted above might have worked for Ubuntu, but does not work > in Gentoo. Reading the comments in the shell script, apparently by default > dracut initrd calls a filesystem consistency check on every boot: dracut does not fsck btrfs filesystems. This has been in place since dracut-021. https://github.com/haraldh/dracut/commit/1594d0bf9cf19b45fdb5574e141e0cae163546f5
(In reply to Mike Gilbert from comment #3) > > I am using btrfs root + dracut + systemd, and I do not experience this > problem. > > In the initramfs image, /sbin/fsck.btrfs is the same dummy shell script that > exists on my real root filesystem. So, even though systemd may try to invoke > it, it does noting and returns immediately. > > What is /sbin/fsck.btrfs in your generated initramfs image? If it is a real > fsck binary, something has gone horribly wrong. As I posted in comment # 2 (that's where the commented out lines were pulled from), it's a shell script that generally does nothing. Despite that, somehow a filesystem check is being invoked, and the process takes 2+ hours (which systemd is intelligent enough to abort after a 1.5 minute timeout and drop me to the dracut emergency shell). Reading the comments in the shell script, it appears that a consistency check will be invoked (if not through the script (as it appears to do nothing), then through some other route) whenever the system determines that the fs_passno is set to a non-zero value. In my case, I have no fstab in the dracut image (none of my machines do), so the only thing I can think of is that the fs_passno defaults to non-zero if no entry is found in the fstab. When I generate my dracut image, I run the following script: --- #!/bin/bash DRACUT_GENSPLASH_THEME=natural_gentoo DRACUT_GENSPLASH_RES=1024x768 dracut -M -v --force --xz /boot/initramfs-$1$2.img $1-gentoo$2 && ln -sf initramfs-$1$2.img /boot/initramfs-$1-gentoo$2.img --- For the kernel in question, it is run as: "~/gendracut.sh 3.17.8 -r1" (so 3.17.8 is $1, and -r1 is $2) My dracut.conf is attached.
Created attachment 397060 [details] dracut.conf
(In reply to Mike Gilbert from comment #4) > (In reply to Richard Ostrow from comment #2) > > You're right - btrfs.fsck is a shell script that does nothing. The > > "workaround" I posted above might have worked for Ubuntu, but does not work > > in Gentoo. Reading the comments in the shell script, apparently by default > > dracut initrd calls a filesystem consistency check on every boot: > > dracut does not fsck btrfs filesystems. This has been in place since > dracut-021. > > https://github.com/haraldh/dracut/commit/ > 1594d0bf9cf19b45fdb5574e141e0cae163546f5 ... I don't know what to think on seeing that. I'm running dracut-041, btrfs root, and systemd-216-r3, and despite the code snippet you show (from which I'd agree, it *shouldn't* be doing a consistency check), it's doing a filesystem consistency check. On a long-shot, it could be different because my root is on a luks dm-crypt root, on which btrfs is mounted? For reference, here are my kernel parameters: BOOT_IMAGE=/vmlinuz-3.16.5-gentoo root=/dev/mapper/luks-2b9ae983-8b10-4f4e-a5a9-c5f604c77943 ro rootflags=subvol=root-2014-12-30 rd.luks.uuid=luks-2b9ae983-8b10-4f4e-a5a9-c5f604c77943 rootflags=rw,compress=zlib,nobarrier rootfstype=btrfs init=/usr/lib64/systemd/systemd I fell back to 3.16.5 because when I generated the image for 3.17.8-r1 and attempted to change out /sbin/fsck.btrfs with a symlink to /bin/true, I got a kernel panic with no emergency shell and haven't gone back to adjust yet. I note that the 3.16.5 initrd also claims to do a consistency check, but for some reason it somehow finished in under the 1.5 minute time limit imposed by systemd.
(In reply to Richard Ostrow from comment #5) > Reading the comments in the shell > script, it appears that a consistency check will be invoked (if not through > the script (as it appears to do nothing), then through some other route) > whenever the system determines that the fs_passno is set to a non-zero > value. In my case, I have no fstab in the dracut image (none of my machines > do), so the only thing I can think of is that the fs_passno defaults to > non-zero if no entry is found in the fstab. You are misunderstanding the comment. It is describing what happens for a typical filesystem with a real fsck binary. Since fsck.btrfs is a dummy shell script, the fs_passno in fstab is irrelevant; /sbin/fsck -a looks at that, but ultimately ends up running /sbin/fsck.<fstype> to do the actual work.
Could you upload your initramfs image somewhere so we can take a look at it?
Also it would be interesting to see rdsosreport.txt generated by dracut. Please add "log_buf_len=1M rd.debug systemd.log_level=debug" to the kernel cmdline. When you will be dropped to an emergency shell, save the file "/run/initramfs/rdsosreport.txt" on persistent storage (e.g. real root or /boot). And attach it here.
