Bug 42929 - After Running Prelink the shutdown script cannot remount read-only the root filesystem
Bug#: 42929 Product:  Gentoo Linux Version: 1.4 Platform: x86
OS/Version: Linux Status: RESOLVED Severity: normal Priority: P2
Resolution: FIXED Assigned To: cretin@gentoo.org Reported By: maria_libera2000@yahoo.it
Component: Core system
URL: 
Summary: After Running Prelink the shutdown script cannot remount read-only the root filesystem
Keywords:  
Status Whiteboard: 
Opened: 2004-02-25 14:25 0000
Description:   Opened: 2004-02-25 14:25 0000 
If I try to run "prelink -amfR" (I've also tried only "prelink" ) of "prelink
-au" during the system uptime when I have to Reboot or Shutdown the system ,
the shutdown process gives an error at the "Remounting Filesystems read-only 
[!!]" , then is printed : "Give root password for maintenance:" , after few
seconds the system shutdowns or reboots (according if you have gave the halt or
reboot command ) . The next time the system is booted the ROOT filesystem is
marked as UnClean , and in the dmesg you can find some errors about the
filesystem ide0(3,3):Removing [5181 5779 0x0 SD]..done
ide0(3,3):Removing [5112 5560 0x0 SD]..done
ide0(3,3):Removing [5112 5161 0x0 SD]..done
ide0(3,3):There were 3 uncompleted unlinks/truncates. Completed

Now the system returns perfectly normal , and you can reboot/shutdown without
problems until the next time you run the prelink executable .

I've tried to Remount manually the root filesystem before and after running
prelink with "umount -r /" , and if I run this before running prelink the
result is : OK , after the result is "Device busy" 

Reproducible: Always
Steps to Reproduce:
1.Load the system
2.Run the prelink executable (ex: prelink -amfR )
3.Reboot the system and see the error messages explained above

Actual Results:  
The ROOT filesystem during the halt/reboot steps is not correctly unmounted and
during the next boot it needs to be repaired by checking its Journal

Expected Results:  
The ROOT filesystem should unmount correctly

bash-2.05b# emerge info
Portage 2.0.50-r1 (default-x86-1.4, gcc-3.3.2, glibc-2.3.2-r9,
2.4.22-gentoo-r7)
=================================================================
System uname: 2.4.22-gentoo-r7 i686 AMD Athlon(tm) XP 2400+
Gentoo Base System version 1.4.3.13
Autoconf: sys-devel/autoconf-2.58-r1
Automake: sys-devel/automake-1.7.7
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-O3 -mcpu=athlon-xp -march=athlon-xp -fomit-frame-pointer -ftracer
-fprefetch-loop-arrays -fforce-addr -mfpmath=sse -momit-leaf-frame-pointer
-mmmx
-m3dnow -msse -pipe"
CHOST="i686-pc-linux-gnu"
COMPILER="gcc3"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3/share/config /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d"
CXXFLAGS="-O3 -mcpu=athlon-xp -march=athlon-xp -fomit-frame-pointer -ftracer
-fprefetch-loop-arrays -fforce-addr -mfpmath=sse -momit-leaf-frame-pointer
-mmmx
-m3dnow -msse -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache sandbox"
GENTOO_MIRRORS="http://gentoo.oregonstate.edu
http://distro.ibiblio.org/pub/Linux/distributions/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/p-portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow X acpi alsa apm avi berkdb crypt cups dga encode fam foomaticdb gdbm
gif gpm gtk gtk2 imlib jpeg libg++ libwww mad mikmod mmx motif mozilla mpeg
ncurses nls oggvorbis opengl oss pam pdflib perl png python quicktime readline
sdl slang spell sse ssl svga tcpd threads truetype x86 xml2 xmms xv zlib"


-------------------------------------------------------------------------------

I've tried various combination of CFLAGS (also disabled) in order to get rid of
the problem but without success

------- Comment #1 From Stefan Jones (RETIRED) 2004-02-27 09:14:05 0000 ------- 
This will be difficult to debug.

Could you run prelink and then try to remount root file system read-only after it has finished.

You will first have to switch to the single user mode by running:
"init 1"
and then kill all processes except bash (your command line).

Then please try:
"mount -o remount,ro /" (remount root as read only)

Please report your finding

------- Comment #2 From Heiko Helmle 2004-03-15 03:46:01 0000 ------- 
I have the same problem and tried that, but remounting root is impossible
(device busy), although there are only bash, init and the kernel processes
left.

------- Comment #3 From Stefan Jones (RETIRED) 2004-03-16 07:10:54 0000 ------- 
Can you install lsof (emerge lsof) and run it when you only have bash and init
running.

This should display the open file descripers and find what is keeping the disk
busy.

Many thanks for your efforts.

------- Comment #4 From Tarragon M. Allen 2004-04-06 20:55:01 0000 ------- 
I can confirm this problem. After prelinking the entire system I get the same
results on reboot or halt. Also of note is an error when it tries to unmount my
NFS mount. I can supply exact errors and do any further tests if required.

------- Comment #5 From Tarragon M. Allen 2004-04-06 20:57:09 0000 ------- 
I can confirm this problem. After prelinking the entire system I get the same
results on reboot or halt. Also of note is an error when it tries to unmount my
NFS mount. I can supply exact errors and do any further tests if required.

------- Comment #6 From Stefan Jones (RETIRED) 2004-04-07 02:11:04 0000 ------- 
Can you do the following:

0) prelink to trigger the error
1) install lsof (emerge lsof)
2) run "init 1"
3) run "ps -e" (report results)
4) run "lsof" (report results)
5) Try to mount the root filesystem read only
6) umount all non-root filesysems

Can you also upgrade to latest version of prelink, sys-devel/prelink-20040317
and see if it fixes things

------- Comment #7 From Tarragon M. Allen 2004-04-09 20:24:38 0000 ------- 
> Can you also upgrade to latest version of prelink, sys-devel/prelink-20040317

This is the version I'm using.

> 0) prelink to trigger the error

Done.

> 1) install lsof (emerge lsof)

Already had it installed. :)

> 2) run "init 1"

I forgot to do this, but it shouldn't matter. I mounted /usr (so I could get to lsof) and /tmp (so I had somewhere to put the output) before i ran the following tests.

> 3) run "ps -e" (report results)

  PID TTY          TIME CMD
    1 ?        00:00:06 init
    2 ?        00:00:00 ksoftirqd/0
    3 ?        00:00:00 events/0
    4 ?        00:00:00 kblockd/0
    7 ?        00:00:00 pdflush
    6 ?        00:00:00 kapmd
   10 ?        00:00:00 aio/0
    9 ?        00:00:00 kswapd0
   12 ?        00:00:00 kseriod
   15 ?        00:00:00 kjournald
11070 ?        00:00:00 pdflush
13365 ?        00:00:00 rc
17879 tty1     00:00:00 bash
17893 ?        00:00:00 reiserfs/0
17897 ?        00:00:00 kjournald
17901 tty1     00:00:00 ps

> 4) run "lsof" (report results)

COMMAND     PID USER   FD   TYPE DEVICE    SIZE       NODE NAME
init          1 root  cwd    DIR   33,5    1024          2 /
init          1 root  rtd    DIR   33,5    1024          2 /
init          1 root  txt    REG   33,5   31600      36298 /sbin/init
init          1 root  mem    REG   33,5   89752      34969 /lib/ld-2.3.3.so
init          1 root  mem    REG   33,5 1285308      34970 /lib/libc-2.3.3.so
init          1 root   10u  FIFO   0,10               2779 /dev/initctl
ksoftirqd     2 root  cwd    DIR   33,5    1024          2 /
ksoftirqd     2 root  rtd    DIR   33,5    1024          2 /
events/0      3 root  cwd    DIR   33,5    1024          2 /
events/0      3 root  rtd    DIR   33,5    1024          2 /
kblockd/0     4 root  cwd    DIR   33,5    1024          2 /
kblockd/0     4 root  rtd    DIR   33,5    1024          2 /
kapmd         6 root  cwd    DIR   33,5    1024          2 /
kapmd         6 root  rtd    DIR   33,5    1024          2 /
pdflush       7 root  cwd    DIR   33,5    1024          2 /
pdflush       7 root  rtd    DIR   33,5    1024          2 /
kswapd0       9 root  cwd    DIR   33,5    1024          2 /
kswapd0       9 root  rtd    DIR   33,5    1024          2 /
aio/0        10 root  cwd    DIR   33,5    1024          2 /
aio/0        10 root  rtd    DIR   33,5    1024          2 /
kseriod      12 root  cwd    DIR   33,5    1024          2 /
kseriod      12 root  rtd    DIR   33,5    1024          2 /
kjournald    15 root  cwd    DIR   33,5    1024          2 /
kjournald    15 root  rtd    DIR   33,5    1024          2 /
pdflush   11070 root  cwd    DIR   33,5    1024          2 /
pdflush   11070 root  rtd    DIR   33,5    1024          2 /
rc        13365 root  cwd    DIR   0,10       0        848 /dev
rc        13365 root  rtd    DIR   33,5    1024          2 /
rc        13365 root  txt    REG   33,5  641548      24151 /bin/bash
rc        13365 root  mem    REG   33,5   27720      34243 /lib/libnss_compat-2.3.3.so
rc        13365 root  mem    REG   33,5   33168      34191 /lib/libnss_nis-2.3.3.so
rc        13365 root  mem    REG   33,5   35584      34179 /lib/libnss_files-2.3.3.so
rc        13365 root  mem    REG   33,5   89752      34969 /lib/ld-2.3.3.so
rc        13365 root  mem    REG   33,5 1285308      34970 /lib/libc-2.3.3.so
rc        13365 root  mem    REG   33,5   11880      34973 /lib/libdl-2.3.3.so
rc        13365 root  mem    REG   33,5   77584      34975 /lib/libnsl-2.3.3.so
rc        13365 root    0u   CHR    5,1               1688 /dev/console
rc        13365 root    1u   CHR    5,1               1688 /dev/console
rc        13365 root    2u   CHR    5,1               1688 /dev/console
rc        13365 root  255r   REG   33,5   20238      36199 /sbin/rc
bash      17879 root  cwd    DIR   33,5    1024      42172 /root
bash      17879 root  rtd    DIR   33,5    1024          2 /
bash      17879 root  txt    REG   33,5  641548      24151 /bin/bash
bash      17879 root  mem    REG   33,5   27720      34243 /lib/libnss_compat-2.3.3.so
bash      17879 root  mem    REG   33,5   33168      34191 /lib/libnss_nis-2.3.3.so
bash      17879 root  mem    REG   33,5   35584      34179 /lib/libnss_files-2.3.3.so
bash      17879 root  mem    REG   33,5   89752      34969 /lib/ld-2.3.3.so
bash      17879 root  mem    REG   33,5 1285308      34970 /lib/libc-2.3.3.so
bash      17879 root  mem    REG   33,5   11880      34973 /lib/libdl-2.3.3.so
bash      17879 root  mem    REG   33,5   77584      34975 /lib/libnsl-2.3.3.so
bash      17879 root    0u   CHR    5,1               1688 /dev/console
bash      17879 root    1u   CHR    5,1               1688 /dev/console
bash      17879 root    2u   CHR    5,1               1688 /dev/console
bash      17879 root  255u   CHR    5,1               1688 /dev/console
reiserfs/ 17893 root  cwd    DIR   33,5    1024          2 /
reiserfs/ 17893 root  rtd    DIR   33,5    1024          2 /
kjournald 17897 root  cwd    DIR   33,5    1024          2 /
kjournald 17897 root  rtd    DIR   33,5    1024          2 /
lsof      17899 root  cwd    DIR   33,5    1024      42172 /root
lsof      17899 root  rtd    DIR   33,5    1024          2 /
lsof      17899 root  txt    REG   33,7   98768        106 /usr/sbin/lsof
lsof      17899 root  mem    REG   33,5   89752      34969 /lib/ld-2.3.3.so
lsof      17899 root  mem    REG   33,5 1285308      34970 /lib/libc-2.3.3.so
lsof      17899 root    0u   CHR    5,1               1688 /dev/console
lsof      17899 root    1w   REG   33,8       0         12 /tmp/lsof
lsof      17899 root    2u   CHR    5,1               1688 /dev/console
lsof      17899 root    3r   DIR    0,3       0          1 /proc
lsof      17899 root    4r   DIR    0,3       0 1173028873 /proc/17899/fd
lsof      17899 root    5w  FIFO    0,7              47743 pipe
lsof      17899 root    6r  FIFO    0,7              47744 pipe
lsof      17900 root  cwd    DIR   33,5    1024      42172 /root
lsof      17900 root  rtd    DIR   33,5    1024          2 /
lsof      17900 root  txt    REG   33,7   98768        106 /usr/sbin/lsof
lsof      17900 root  mem    REG   33,5   89752      34969 /lib/ld-2.3.3.so
lsof      17900 root  mem    REG   33,5 1285308      34970 /lib/libc-2.3.3.so
lsof      17900 root    4r  FIFO    0,7              47743 pipe
lsof      17900 root    7w  FIFO    0,7              47744 pipe

> 5) Try to mount the root filesystem read only

# mount -oremount,rw /
umount: /: device is busy

> 6) umount all non-root filesysems

Why? :)

Btw, for reference, here's the output of 'mount':

/dev/hde5 on / type ext3 (rw,noatime)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev type ramfs (rw)
none on /dev/pts type devpts (rw)
/dev/hde7 on /usr type reiserfs (rw,noatime)
/dev/hde8 on /tmp type ext3 (rw,noatime)

I run the gentoodev-source 2.6.5 kernel with udev 024-r1, if that's relevant.

t

------- Comment #8 From Stefan Jones (RETIRED) 2004-04-11 01:39:15 0000 ------- 
I have reproduced the problem on my machine also.

I get the same results for lsof and ps

My guess it is either a kernel bug with mmaped file reference counts or
some strangeness with shared libraries (mmaped files) being used which are no longer on the disk (as they have been modified with prelink).

I have tried "init u", which restarts init incase that is locking a shared library, but it did not help. All other processes were also restarted (except kernel ones ofcourse)

Will invesigate further.

------- Comment #9 From Stefan Jones (RETIRED) 2004-04-12 03:43:22 0000 ------- 
Found the fix but I am not sure how to intergrate it.

After you run prelink -amfR run (also as root) "telinit u"

After that you can poweroff the computer with no trouble.

Now as users are fond of running the prelink executable, how to do it so that this is done automatically?

I think some prelink wrapper script is needed, which could optionally be added as a cron job or added into portage. Will think about it.

------- Comment #10 From SpanKY 2004-06-11 16:52:46 0000 ------- 
not related to Bug 33271 ?

------- Comment #11 From Stefan Jones (RETIRED) 2004-06-11 17:18:24 0000 ------- 
Probably, need to get the people do do the same diagnositcs as happened here.

Emerging baselayout would change the init executable, so "init u" would need to be run.

But it looks like it does:
/sbin/init U &>/dev/null
unless the code round it is broken ...

------- Comment #12 From Stefan Jones (RETIRED) 2004-09-09 11:15:13 0000 ------- 
Done the hack, see prelink-20040707.ebuild ( you have to change the KEYWORDS as
it is masked)

It reruns init if it is changed, this should fix the problem. Please test.

Stefan

------- Comment #13 From Stefan Jones (RETIRED) 2004-09-10 09:40:20 0000 ------- 
Added prelink-20040707.ebuild which autodetects if this could happen and runs
"init u" as required. Please test.

------- Comment #14 From Heinrich Wendel (RETIRED) 2004-10-23 16:15:36 0000 ------- 
seems to work fine, what about marking it stable?

------- Comment #15 From Stefan Jones (RETIRED) 2004-10-23 17:41:03 0000 ------- 
Ok, marked it as stable on x86.

As always other archs are up to there respective maintainers.