If I try to run "prelink -amfR" (I've also tried only "prelink" ) of "prelink -au" during the system uptime when I have to Reboot or Shutdown the system , the shutdown process gives an error at the "Remounting Filesystems read-only [!!]" , then is printed : "Give root password for maintenance:" , after few seconds the system shutdowns or reboots (according if you have gave the halt or reboot command ) . The next time the system is booted the ROOT filesystem is marked as UnClean , and in the dmesg you can find some errors about the filesystem ide0(3,3):Removing [5181 5779 0x0 SD]..done ide0(3,3):Removing [5112 5560 0x0 SD]..done ide0(3,3):Removing [5112 5161 0x0 SD]..done ide0(3,3):There were 3 uncompleted unlinks/truncates. Completed Now the system returns perfectly normal , and you can reboot/shutdown without problems until the next time you run the prelink executable . I've tried to Remount manually the root filesystem before and after running prelink with "umount -r /" , and if I run this before running prelink the result is : OK , after the result is "Device busy" Reproducible: Always Steps to Reproduce: 1.Load the system 2.Run the prelink executable (ex: prelink -amfR ) 3.Reboot the system and see the error messages explained above Actual Results: The ROOT filesystem during the halt/reboot steps is not correctly unmounted and during the next boot it needs to be repaired by checking its Journal Expected Results: The ROOT filesystem should unmount correctly bash-2.05b# emerge info Portage 2.0.50-r1 (default-x86-1.4, gcc-3.3.2, glibc-2.3.2-r9, 2.4.22-gentoo-r7) ================================================================= System uname: 2.4.22-gentoo-r7 i686 AMD Athlon(tm) XP 2400+ Gentoo Base System version 1.4.3.13 Autoconf: sys-devel/autoconf-2.58-r1 Automake: sys-devel/automake-1.7.7 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O3 -mcpu=athlon-xp -march=athlon-xp -fomit-frame-pointer -ftracer -fprefetch-loop-arrays -fforce-addr -mfpmath=sse -momit-leaf-frame-pointer -mmmx -m3dnow -msse -pipe" CHOST="i686-pc-linux-gnu" COMPILER="gcc3" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d" CXXFLAGS="-O3 -mcpu=athlon-xp -march=athlon-xp -fomit-frame-pointer -ftracer -fprefetch-loop-arrays -fforce-addr -mfpmath=sse -momit-leaf-frame-pointer -mmmx -m3dnow -msse -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="http://gentoo.oregonstate.edu http://distro.ibiblio.org/pub/Linux/distributions/gentoo" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/p-portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow X acpi alsa apm avi berkdb crypt cups dga encode fam foomaticdb gdbm gif gpm gtk gtk2 imlib jpeg libg++ libwww mad mikmod mmx motif mozilla mpeg ncurses nls oggvorbis opengl oss pam pdflib perl png python quicktime readline sdl slang spell sse ssl svga tcpd threads truetype x86 xml2 xmms xv zlib" ------------------------------------------------------------------------------- I've tried various combination of CFLAGS (also disabled) in order to get rid of the problem but without success
This will be difficult to debug. Could you run prelink and then try to remount root file system read-only after it has finished. You will first have to switch to the single user mode by running: "init 1" and then kill all processes except bash (your command line). Then please try: "mount -o remount,ro /" (remount root as read only) Please report your finding
I have the same problem and tried that, but remounting root is impossible (device busy), although there are only bash, init and the kernel processes left.
Can you install lsof (emerge lsof) and run it when you only have bash and init running. This should display the open file descripers and find what is keeping the disk busy. Many thanks for your efforts.
I can confirm this problem. After prelinking the entire system I get the same results on reboot or halt. Also of note is an error when it tries to unmount my NFS mount. I can supply exact errors and do any further tests if required.
Can you do the following: 0) prelink to trigger the error 1) install lsof (emerge lsof) 2) run "init 1" 3) run "ps -e" (report results) 4) run "lsof" (report results) 5) Try to mount the root filesystem read only 6) umount all non-root filesysems Can you also upgrade to latest version of prelink, sys-devel/prelink-20040317 and see if it fixes things
> Can you also upgrade to latest version of prelink, sys-devel/prelink-20040317 This is the version I'm using. > 0) prelink to trigger the error Done. > 1) install lsof (emerge lsof) Already had it installed. :) > 2) run "init 1" I forgot to do this, but it shouldn't matter. I mounted /usr (so I could get to lsof) and /tmp (so I had somewhere to put the output) before i ran the following tests. > 3) run "ps -e" (report results) PID TTY TIME CMD 1 ? 00:00:06 init 2 ? 00:00:00 ksoftirqd/0 3 ? 00:00:00 events/0 4 ? 00:00:00 kblockd/0 7 ? 00:00:00 pdflush 6 ? 00:00:00 kapmd 10 ? 00:00:00 aio/0 9 ? 00:00:00 kswapd0 12 ? 00:00:00 kseriod 15 ? 00:00:00 kjournald 11070 ? 00:00:00 pdflush 13365 ? 00:00:00 rc 17879 tty1 00:00:00 bash 17893 ? 00:00:00 reiserfs/0 17897 ? 00:00:00 kjournald 17901 tty1 00:00:00 ps > 4) run "lsof" (report results) COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME init 1 root cwd DIR 33,5 1024 2 / init 1 root rtd DIR 33,5 1024 2 / init 1 root txt REG 33,5 31600 36298 /sbin/init init 1 root mem REG 33,5 89752 34969 /lib/ld-2.3.3.so init 1 root mem REG 33,5 1285308 34970 /lib/libc-2.3.3.so init 1 root 10u FIFO 0,10 2779 /dev/initctl ksoftirqd 2 root cwd DIR 33,5 1024 2 / ksoftirqd 2 root rtd DIR 33,5 1024 2 / events/0 3 root cwd DIR 33,5 1024 2 / events/0 3 root rtd DIR 33,5 1024 2 / kblockd/0 4 root cwd DIR 33,5 1024 2 / kblockd/0 4 root rtd DIR 33,5 1024 2 / kapmd 6 root cwd DIR 33,5 1024 2 / kapmd 6 root rtd DIR 33,5 1024 2 / pdflush 7 root cwd DIR 33,5 1024 2 / pdflush 7 root rtd DIR 33,5 1024 2 / kswapd0 9 root cwd DIR 33,5 1024 2 / kswapd0 9 root rtd DIR 33,5 1024 2 / aio/0 10 root cwd DIR 33,5 1024 2 / aio/0 10 root rtd DIR 33,5 1024 2 / kseriod 12 root cwd DIR 33,5 1024 2 / kseriod 12 root rtd DIR 33,5 1024 2 / kjournald 15 root cwd DIR 33,5 1024 2 / kjournald 15 root rtd DIR 33,5 1024 2 / pdflush 11070 root cwd DIR 33,5 1024 2 / pdflush 11070 root rtd DIR 33,5 1024 2 / rc 13365 root cwd DIR 0,10 0 848 /dev rc 13365 root rtd DIR 33,5 1024 2 / rc 13365 root txt REG 33,5 641548 24151 /bin/bash rc 13365 root mem REG 33,5 27720 34243 /lib/libnss_compat-2.3.3.so rc 13365 root mem REG 33,5 33168 34191 /lib/libnss_nis-2.3.3.so rc 13365 root mem REG 33,5 35584 34179 /lib/libnss_files-2.3.3.so rc 13365 root mem REG 33,5 89752 34969 /lib/ld-2.3.3.so rc 13365 root mem REG 33,5 1285308 34970 /lib/libc-2.3.3.so rc 13365 root mem REG 33,5 11880 34973 /lib/libdl-2.3.3.so rc 13365 root mem REG 33,5 77584 34975 /lib/libnsl-2.3.3.so rc 13365 root 0u CHR 5,1 1688 /dev/console rc 13365 root 1u CHR 5,1 1688 /dev/console rc 13365 root 2u CHR 5,1 1688 /dev/console rc 13365 root 255r REG 33,5 20238 36199 /sbin/rc bash 17879 root cwd DIR 33,5 1024 42172 /root bash 17879 root rtd DIR 33,5 1024 2 / bash 17879 root txt REG 33,5 641548 24151 /bin/bash bash 17879 root mem REG 33,5 27720 34243 /lib/libnss_compat-2.3.3.so bash 17879 root mem REG 33,5 33168 34191 /lib/libnss_nis-2.3.3.so bash 17879 root mem REG 33,5 35584 34179 /lib/libnss_files-2.3.3.so bash 17879 root mem REG 33,5 89752 34969 /lib/ld-2.3.3.so bash 17879 root mem REG 33,5 1285308 34970 /lib/libc-2.3.3.so bash 17879 root mem REG 33,5 11880 34973 /lib/libdl-2.3.3.so bash 17879 root mem REG 33,5 77584 34975 /lib/libnsl-2.3.3.so bash 17879 root 0u CHR 5,1 1688 /dev/console bash 17879 root 1u CHR 5,1 1688 /dev/console bash 17879 root 2u CHR 5,1 1688 /dev/console bash 17879 root 255u CHR 5,1 1688 /dev/console reiserfs/ 17893 root cwd DIR 33,5 1024 2 / reiserfs/ 17893 root rtd DIR 33,5 1024 2 / kjournald 17897 root cwd DIR 33,5 1024 2 / kjournald 17897 root rtd DIR 33,5 1024 2 / lsof 17899 root cwd DIR 33,5 1024 42172 /root lsof 17899 root rtd DIR 33,5 1024 2 / lsof 17899 root txt REG 33,7 98768 106 /usr/sbin/lsof lsof 17899 root mem REG 33,5 89752 34969 /lib/ld-2.3.3.so lsof 17899 root mem REG 33,5 1285308 34970 /lib/libc-2.3.3.so lsof 17899 root 0u CHR 5,1 1688 /dev/console lsof 17899 root 1w REG 33,8 0 12 /tmp/lsof lsof 17899 root 2u CHR 5,1 1688 /dev/console lsof 17899 root 3r DIR 0,3 0 1 /proc lsof 17899 root 4r DIR 0,3 0 1173028873 /proc/17899/fd lsof 17899 root 5w FIFO 0,7 47743 pipe lsof 17899 root 6r FIFO 0,7 47744 pipe lsof 17900 root cwd DIR 33,5 1024 42172 /root lsof 17900 root rtd DIR 33,5 1024 2 / lsof 17900 root txt REG 33,7 98768 106 /usr/sbin/lsof lsof 17900 root mem REG 33,5 89752 34969 /lib/ld-2.3.3.so lsof 17900 root mem REG 33,5 1285308 34970 /lib/libc-2.3.3.so lsof 17900 root 4r FIFO 0,7 47743 pipe lsof 17900 root 7w FIFO 0,7 47744 pipe > 5) Try to mount the root filesystem read only # mount -oremount,rw / umount: /: device is busy > 6) umount all non-root filesysems Why? :) Btw, for reference, here's the output of 'mount': /dev/hde5 on / type ext3 (rw,noatime) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev type ramfs (rw) none on /dev/pts type devpts (rw) /dev/hde7 on /usr type reiserfs (rw,noatime) /dev/hde8 on /tmp type ext3 (rw,noatime) I run the gentoodev-source 2.6.5 kernel with udev 024-r1, if that's relevant. t
I have reproduced the problem on my machine also. I get the same results for lsof and ps My guess it is either a kernel bug with mmaped file reference counts or some strangeness with shared libraries (mmaped files) being used which are no longer on the disk (as they have been modified with prelink). I have tried "init u", which restarts init incase that is locking a shared library, but it did not help. All other processes were also restarted (except kernel ones ofcourse) Will invesigate further.
Found the fix but I am not sure how to intergrate it. After you run prelink -amfR run (also as root) "telinit u" After that you can poweroff the computer with no trouble. Now as users are fond of running the prelink executable, how to do it so that this is done automatically? I think some prelink wrapper script is needed, which could optionally be added as a cron job or added into portage. Will think about it.
not related to Bug 33271 ?
Probably, need to get the people do do the same diagnositcs as happened here. Emerging baselayout would change the init executable, so "init u" would need to be run. But it looks like it does: /sbin/init U &>/dev/null unless the code round it is broken ...
Done the hack, see prelink-20040707.ebuild ( you have to change the KEYWORDS as it is masked) It reruns init if it is changed, this should fix the problem. Please test. Stefan
Added prelink-20040707.ebuild which autodetects if this could happen and runs "init u" as required. Please test.
seems to work fine, what about marking it stable?
Ok, marked it as stable on x86. As always other archs are up to there respective maintainers.
