Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 410605 - sys-apps/util-linux-2.20-r1: fsck complains about checking file systems mounted ro
Summary: sys-apps/util-linux-2.20-r1: fsck complains about checking file systems mount...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Normal normal with 3 votes (vote)
Assignee: Gentoo's Team for Core System packages
URL:
Whiteboard:
Keywords:
: 443018 (view as bug list)
Depends on:
Blocks: 415249
  Show dependency tree
 
Reported: 2012-04-02 21:30 UTC by Matthias Maier
Modified: 2020-05-24 09:37 UTC (History)
17 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info (emerge.info,4.84 KB, text/plain)
2012-04-02 21:30 UTC, Matthias Maier
Details
/etc/fstab (fstab,1.11 KB, text/plain)
2012-04-02 21:31 UTC, Matthias Maier
Details
rc-fsck.log (rc-fsck.log,877 bytes, text/plain)
2012-04-12 15:41 UTC, William Hubbs
Details
rc.log showing /proc/mounts prior to fsck (rc-withmounts.log,7.78 KB, text/plain)
2012-04-13 08:21 UTC, Matthias Maier
Details
patch to enable fsck of /usr when mounted ro by initramfs (e2fsprogs-1.42.6-unix.patch,1.67 KB, patch)
2013-01-20 19:47 UTC, BobbyK
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Maier gentoo-dev 2012-04-02 21:30:06 UTC
For modern udev it is necessary to mount /usr (if it is on a separate partition) prior to system startup.

sys-kernel/genkernel-3.4.27 does this just fine.

However, now the fsck service fails on system startup:

 * Checking local filesystems  ...
/dev/mapper/ssd-root: clean, 6772/32768 files, 74767/131072 blocks
/dev/mapper/ssd-usr is mounted.  e2fsck: Cannot continue, aborting.
 
/dev/mapper/ssd-var: clean, 98464/655360 files, 499701/2621440 blocks
/dev/mapper/volume-home: clean, 136144/131072000 files, 288593473/524288000 blocks
/dev/mapper/ssd-datengrab: clean, 74789/7208960 files, 5000359/28835840 blocks
/dev/mapper/ssd-opt: clean, 2515/327680 files, 76151/1310720 blocks
/dev/mapper/ssd-usr--portage: clean, 206279/655360 files, 1765162/2621440 blocks (check after next mount)
/dev/mapper/ssd-usr--portage--packages: clean, 1521/983040 files, 2177635/3932160 blocks
 * Operational error [ !! ]
 * Remounting root filesystem read/write ... [ ok ]
 * Remounting filesystems ... [ ok ]

This happens for passno set to 1 or 2 for /usr in /etc/fstab.

Reproducible: Always
Comment 1 Matthias Maier gentoo-dev 2012-04-02 21:30:27 UTC
Created attachment 307551 [details]
emerge --info
Comment 2 Matthias Maier gentoo-dev 2012-04-02 21:31:17 UTC
Created attachment 307553 [details]
/etc/fstab
Comment 3 William Hubbs gentoo-dev 2012-04-12 04:02:44 UTC
Hi,

(In reply to comment #0)
> For modern udev it is necessary to mount /usr (if it is on a separate
> partition) prior to system startup.
> 
> sys-kernel/genkernel-3.4.27 does this just fine.
> 
> However, now the fsck service fails on system startup:

Actually, We warn you that there was an operational error, but the fsck service doesn't treat this as a failure.

>  * Checking local filesystems  ...
> /dev/mapper/ssd-root: clean, 6772/32768 files, 74767/131072 blocks
> /dev/mapper/ssd-usr is mounted.  e2fsck: Cannot continue, aborting.
>  
> /dev/mapper/ssd-var: clean, 98464/655360 files, 499701/2621440 blocks
> /dev/mapper/volume-home: clean, 136144/131072000 files, 288593473/524288000
> blocks
> /dev/mapper/ssd-datengrab: clean, 74789/7208960 files, 5000359/28835840
> blocks
> /dev/mapper/ssd-opt: clean, 2515/327680 files, 76151/1310720 blocks
> /dev/mapper/ssd-usr--portage: clean, 206279/655360 files, 1765162/2621440
> blocks (check after next mount)
> /dev/mapper/ssd-usr--portage--packages: clean, 1521/983040 files,
> 2177635/3932160 blocks
>  * Operational error [ !! ]

This output shows that all of your file systems, except /usr,  were checked. 
We let you know that there is an operational error (which there is, from fsck's pov anyway, because we tried to run fsck on /usr which is already mounted), but fsck -a checks all other file systems and the fsck service script exits successfully in this situation and does not treat this as a critical error.

I would like to keep the warning, because I dont know all other situations that could cause fsck to exit with exit code 8.
How would you suggest keeping the message but not making it appear as a failure message?

Ideally, I would tel fsck -a to skip /usr isf it is mounted rw and not report this as a failure.
Comment 4 Matthias Maier gentoo-dev 2012-04-12 09:25:30 UTC
(In reply to comment #3)
> Actually, We warn you that there was an operational error, but the fsck
> service doesn't treat this as a failure.
> 
> This output shows that all of your file systems, except /usr,  were checked. 
> We let you know that there is an operational error (which there is, from
> fsck's pov anyway, because we tried to run fsck on /usr which is already
> mounted), but fsck -a checks all other file systems and the fsck service
> script exits successfully in this situation and does not treat this as a
> critical error.

Jup. This is the way I understood the "operational error". I'm sorry to have confused you with "fsck fails" in my bug description.

> I would like to keep the warning, because I dont know all other situations
> that could cause fsck to exit with exit code 8.
> How would you suggest keeping the message but not making it appear as a
> failure message?

I would leave the error message as it is.

The point is, that the genkernel init script mounts /usr ro - at least it is telling this (I haven't checked manually.)

So ideally, I would like to have fsck -a just succeed in checking /usr as it succeeds in checking the already (ro) mounted /.

Does fsck -a have a problem with any other partition (beside /) already mounted? rw obviously, but ro?

>
> Ideally, I would tel fsck -a to skip /usr isf it is mounted rw and not
> report this as a failure.

I wouldn't do this. It is always possible to skip a rw mounted partition / or any other special partition with a passno of 0 in fstab.
Comment 5 William Hubbs gentoo-dev 2012-04-12 15:34:12 UTC
(In reply to comment #4)
> The point is, that the genkernel init script mounts /usr ro - at least it is
> telling this (I haven't checked manually.)
> 
> So ideally, I would like to have fsck -a just succeed in checking /usr as it
> succeeds in checking the already (ro) mounted /.
> 
> Does fsck -a have a problem with any other partition (beside /) already
> mounted? rw obviously, but ro?

Apparently it does.

I checked my rc.log and have the same issue.

I also attempted to manually fsck a partition that was mounted ro and fsck complained very loudly about how if I do this I will cause serious file system damage.

This is not a bug in OpenRc but an issue with fsck. I will send this bug to base-system.
Comment 6 William Hubbs gentoo-dev 2012-04-12 15:40:17 UTC
All,

it appears that fsck complains loudly when we ask it to check a
read-only filesystem. This causes confusing results when fsck is run in
the latest openrc because we report an operational error but the fsck
service does not fail. I will attach the last log of my boot runlevel
showing this.

Also,it is possible to reproduce the complaint by mounting a filesystem
ro then trying to run fsck on that file system. If you do this you will
get a big warning about how you will cause file system damage if you
continue.
Comment 7 William Hubbs gentoo-dev 2012-04-12 15:41:55 UTC
Created attachment 308663 [details]
rc-fsck.log

This is the fsck portion of my rc.log the last time I booted my system.

Note that fsck complains about /dev/sda6. This is my /usr file system
which is mounted ro by genkernel in the initramfs.
Comment 8 SpanKY gentoo-dev 2012-04-12 22:31:39 UTC
and you're sure it's mounted ro ?  compare /proc/mounts and /etc/mtab and see what you get for that mount point.
Comment 9 William Hubbs gentoo-dev 2012-04-12 23:54:50 UTC
@vapier:
On my system, /etc/mtab is a file, and not a symbolic link. If I mount
something ro, it shows up as ro in /proc/mounts but not in /etc/mtab .

The mtab init script in openrc allows /etc/mtab to existas a symbolic
link, but if it exists as a file, it gets updated with a snapshot of
/proc/mounts, without tmpfs entries, as part of the boot process.

The mtab script runs *after* fsck and after all local file systems are
mounted, so, it looks like we do have a bug in openrc. If mtab is a
file, it will show /usr mounted rw while /proc/mounts shows it mounted
ro.

I see two ways out of this: I can require everyone to link
/etc/mtab->/proc/mounts, or I can filter the /usr file system out of
/etc/mtab.

What are your thoughts?
Comment 10 SpanKY gentoo-dev 2012-04-13 00:42:00 UTC
(In reply to comment #9)

you cannot require /etc/mtab be a symlink (yet).  there's still information in there that userland mount stores that is not mentioned in /proc/mounts.

before the boot runlevel finishes processing, you can do whatever you like to /etc/mtab.  but once everything has finished booting, all real mounts must be reflected in /etc/mtab (including /usr).
Comment 11 William Hubbs gentoo-dev 2012-04-13 01:00:37 UTC
(In reply to comment #10)
> before the boot runlevel finishes processing, you can do whatever you like
> to /etc/mtab.  but once everything has finished booting, all real mounts
> must be reflected in /etc/mtab (including /usr).

The issue is that /etc/mtab needs to be manipulated before fsck runs to make sure that the settings for /usr match those in /proc/mounts.

Root is still ro at that point, so the only way I could fis this would be to remount root rw, do the fix, then remount root ro.

That seems not to be a good idea to me. What are your thoughts?
Comment 12 Matthias Maier gentoo-dev 2012-04-13 08:06:50 UTC
(In reply to comment #11)

> The issue is that /etc/mtab needs to be manipulated before fsck runs to make
> sure that the settings for /usr match those in /proc/mounts.

Is this really the issue?

On my system /etc/mtab is already a symbolic link pointing to /proc/self/mounts.


> Root is still ro at that point, so the only way I could fis this would be to
> remount root rw, do the fix, then remount root ro.
Comment 13 Matthias Maier gentoo-dev 2012-04-13 08:19:15 UTC
(In reply to comment #8)
> and you're sure it's mounted ro ?  compare /proc/mounts and /etc/mtab and
> see what you get for that mount point.

I've just tested this. /usr is indeed mounted ro at the time fsck starts.
See the next attachment.
Comment 14 Matthias Maier gentoo-dev 2012-04-13 08:21:18 UTC
Created attachment 308745 [details]
rc.log showing /proc/mounts prior to fsck

rc.log with a hook showing /proc/mounts and /etc/mtab (which is a symlink to /proc/self/mounts, hence the same output as /proc/mounts) prior to fsck
Comment 15 William Hubbs gentoo-dev 2012-04-13 14:07:13 UTC
(In reply to comment #13)
> (In reply to comment #8)
> > and you're sure it's mounted ro ?  compare /proc/mounts and /etc/mtab and
> > see what you get for that mount point.
> 
> I've just tested this. /usr is indeed mounted ro at the time fsck starts.
> See the next attachment.

In that case, sorry about all of the extra noise on this bug. This is definitely an fsck issue; there isn't much I can do in OpenRC. fsck runs, -a runs, but it is reporting errors when checking ro mounted file systems.
Comment 16 Marc Schiffbauer gentoo-dev 2012-04-18 10:36:13 UTC
If fsck can handle ro-/ a patch should be trivial to also let it accept ro-/usr the same way.

I consider this a severe issue for all sperate-/usr users because /usr will never be checked even after a crash anymore a users will work with broken /usr filesystems which is a very bad thing.

Lets hope a patch will be available soonish.
Comment 17 SpanKY gentoo-dev 2012-04-18 14:43:11 UTC
the trouble is that patching `fsck` won't help i don't think.  that's just a multiplexing program that turns around and runs `fsck.<fs>`, and that fs-specific fsck is what does the mount check.

i'll have to query on the util-linux mailing list to see if they have any ideas.
Comment 18 Matthias Maier gentoo-dev 2012-04-18 15:20:41 UTC
(In reply to comment #17)

Hi,

I don't quite understand why you changed the bug summary to "
fsck complains about checking file systems mounted ro when /etc/mtab says rw"

The behaviour I observed was more that at system startup fsck complains about any filesystem mounted ro to any other location than /.

I don't think that this has to do anything with /etc/mtab at all. The reasons:

1.) In my case /etc/mtab is a symlink to /proc/self/mounts - so it should not contain an invalid entry.

2.) The root file system is remounted rw _after_ fsck. Before that there is no way to update /etc/mtab. And /etc/mtab must be assumed to be invalid (e.g. a hard crash). fsck usually handles this cases quite fine.
Comment 19 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2012-04-18 17:22:14 UTC
vapier:
I was considering the problem, and having some better way of telling which subset of mountpoints  fsck -a should run on, would greatly improve matters.
For a given boot, being able to tell it "I've already checked A,B,C" would help cover it, as well as better control over RO-check behavior.
Comment 20 William Hubbs gentoo-dev 2012-10-18 04:36:08 UTC
(In reply to comment #17)
> the trouble is that patching `fsck` won't help i don't think.  that's just a
> multiplexing program that turns around and runs `fsck.<fs>`, and that
> fs-specific fsck is what does the mount check.
> 
> i'll have to query on the util-linux mailing list to see if they have any
> ideas.

Do you have any updates on this? It seems like fsck* could be modified to check any read-only file system like it already checks /.
Comment 21 William Hubbs gentoo-dev 2012-12-20 22:09:56 UTC
*** Bug 443018 has been marked as a duplicate of this bug. ***
Comment 22 BobbyK 2013-01-14 17:20:11 UTC
Seeing this as well.  I have an initramfs built by genkernel (v 3.4.45) and /usr is listed in /etc/initramfs.mounts (along with /var and /opt - there are fsck complaints about these as well as /usr).

Is there any news on a fix?
Comment 23 BobbyK 2013-01-15 00:57:28 UTC
Did some digging and noticed the following in e2fsck/unix.c from e2fsck-1.42 (which is installed on my system):

      if ((!(ctx->mount_flags & EXT2_MF_MOUNTED)) ||
          ((ctx->mount_flags & EXT2_MF_ISROOT) &&
           (ctx->mount_flags & EXT2_MF_READONLY) &&
           !(ctx->options & E2F_OPT_WRITECHECK)))
              return;

      if ((ctx->options & E2F_OPT_READONLY) &&
          !(ctx->options & E2F_OPT_WRITECHECK)) {
              printf(_("Warning!  %s is mounted.\n"), ctx->filesystem_name);
              return;
      }

Note that for / ctx->mount_flags is checked for read only, whereas for other filesystems ctx->options is used.

I modified the code to:

        if ((!(ctx->mount_flags & EXT2_MF_MOUNTED)) ||
            ((ctx->mount_flags & EXT2_MF_READONLY) &&
             !(ctx->options & E2F_OPT_WRITECHECK))) {
                if (!(ctx->mount_flags & EXT2_MF_ISROOT))
                        printf(_("Warning!  %s is mounted.\n"), ctx->filesystem_name);
                return;
        }

and did not get a "WARNING!!!  The filesystem is mounted.   If you continue you ***WILL*** cause ***SEVERE*** filesystem damage." message, instead it returned the somewhat less alarming warning in the printf statement.

When the filesystem in question is mounted with "ro" option I see:
ctx->mount_flags & EXT2_MF_READONLY is TRUE.
ctx->options & E2F_OPT_READONLY is FALSE.

when mounted regularly (i.e. "rw") I see:
ctx->mount_flags & EXT2_MF_READONLY is FALSE.
ctx->options & E2F_OPT_READONLY is FALSE.

I've not tried patching e2fsck yet (in my /sbin, the files e2fsck, fsck.ext2 fsck.ext3, fsck.ext4 and fsck.ext4dev are identical), though I'm wondering if this might have something to do with the complaint about /usr on boot.
Comment 24 BobbyK 2013-01-15 01:30:51 UTC
More digging, looks like e2fsprogs-1.42.6 might do something similar:

        if ((!(ctx->mount_flags & (EXT2_MF_MOUNTED | EXT2_MF_BUSY))) ||
            ((ctx->mount_flags & EXT2_MF_ISROOT) &&
             (ctx->mount_flags & EXT2_MF_READONLY) &&
             !(ctx->options & E2F_OPT_WRITECHECK)))
                return;

        if (((ctx->options & E2F_OPT_READONLY) ||
             ((ctx->options & E2F_OPT_FORCE) &&
              (ctx->mount_flags & EXT2_MF_READONLY))) &&
            !(ctx->options & E2F_OPT_WRITECHECK)) {
                log_out(ctx, _("Warning!  %s is %s.\n"),
                        ctx->filesystem_name,
                        ctx->mount_flags & EXT2_MF_MOUNTED ?
                                "mounted" : "in use");
                return;
        }

though it looks like it requires a force option somewhere ...
Comment 25 BobbyK 2013-01-17 00:18:16 UTC
I'll need to reboot to see whether fsck is forced in the genkernel initramfs:

# date ; mount -o "ro,noauto,noatime" /dev/sdg4 /next ; date ; fsck /dev/sdg4 ; date ; fsck -nf /dev/sdg4 ; date ; umount /next
Wed Jan 16 19:08:49 EST 2013
Wed Jan 16 19:08:49 EST 2013
fsck from util-linux 2.21.2
e2fsck 1.42.6 (21-Sep-2012)
/dev/sdg4 is mounted.
e2fsck: Cannot continue, aborting.


Wed Jan 16 19:08:49 EST 2013
fsck from util-linux 2.21.2
e2fsck 1.42.6 (21-Sep-2012)
Warning!  /dev/sdg4 is mounted.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdg4: 6270/1155072 files (0.0% non-contiguous), 166536/4612663 blocks
Wed Jan 16 19:08:51 EST 2013

Have to wonder why it's acceptable to fsck the root filesystem when it's mounted ro and not other filesystems when they are mounted ro.
Comment 26 BobbyK 2013-01-20 19:47:43 UTC
Created attachment 336242 [details, diff]
patch to enable fsck of /usr when mounted ro by initramfs

Patch attached for e2fsprogs-1.42.6.  After applying this, fsck runs with a warning on /usr during boot.  No guarantees that it does something else to fsck and allows it to run in situations where it shouldn't.  Also started a thread on the e2fsprogs sourceforge Help forum (https://sourceforge.net/p/e2fsprogs/discussion/7053/thread/138e160a/) to discuss the patch.
Comment 27 Roland 2013-02-04 18:49:53 UTC
Hello,

any solutions yet? I must check the system manually
Comment 28 Matthias Maier gentoo-dev 2013-02-05 14:36:29 UTC
Theodore Ts'o's comment at the disussion started by BobbyK is quite insightful on why enabling fsck for read-only mounted file systems is not a good solution [1].

At the moment I only see one proper fix for that:

- Let the initramfs check every file system before it is mounted

- Later in openrc, silently skip all already mounted file systems in fsck.

See bug 415249 for a discussion on that.
Comment 29 Matthias Maier gentoo-dev 2013-02-05 14:37:22 UTC
*sigh* (I'll learn it someday)
[1] http://sourceforge.net/p/e2fsprogs/discussion/7053/thread/138e160a/#35ed
Comment 30 Roland 2013-03-20 17:16:05 UTC
Any progress to solve the Problem?
Comment 31 Roland 2013-04-29 06:12:38 UTC
Hello,

is there any Solutions in Portage available?

Regards
Comment 32 Ulrich Müller gentoo-dev 2013-12-04 14:35:20 UTC
(In reply to Matthias Maier from comment #28)
> Theodore Ts'o's comment at the disussion started by BobbyK is quite
> insightful on why enabling fsck for read-only mounted file systems is not
> a good solution [1].

Sorry, I don't find his comment particularly insightful. He raises two valid points, which however are self-evident:
  - Trying to repair an rw mounted fs is not a good idea.
  - After an ro mounted fs has been repaired, the system must be rebooted.

The rest of his argumentation isn't compelling at all: If the init system is buggy and fails to reboot the system (in the second case above), then of course there will be problems. But I fail to see why fscking ro-mounted root and separate /usr would result in "a rickety system set up using mucilage and bailing wire" whereas fscking a large root (including /usr) wouldn't.