Bugzilla Bug 27064

AFAIK all binaries in root filesystem shold work even if /usr partition is not
availible.
It depends on particular setup but at least following ebuilds are affected:

sys-apps/coreutils-5.0-r1 (old fileutils also) with acl support (see bug 18151)
because of sys-apps/attr-2.4.7
sys-apps/acl-2.2.13 - same reason
net-libs/nss_ldap-210 - complex dependencies, move to /usr/lib and symlink (?)
sys-apps/utils-linux-2.12 (cramfs) - sys-libs/zlib: move zlib.so to /lib
net-fs/samba-3.0.0_beta3-r1 - move mount.smbfs to /usr/sbin
net-fs/nfs-utils-1.0.5-r1 - sys-apps/tcp-wrappers: move to /lib (statd is needed
for mounting nfs)
sys-apps/xfsdump-2.2.4-r1 - sys-apps/attr

The most important is of course attr/acl stuff which currently makes fully
acl-enabled system unbootable if /usr is on different partition.

Reproducible: Sometimes
Steps to Reproduce:
for i in /bin/* /lib/* /sbin/*; do echo $i `ldd $i 2>/dev/null | grep /usr/lib`;
done | grep /usr/lib
Actual Results:  
/bin/cp libattr.so.1 => /usr/lib/libattr.so.1 (0x40027000)
/bin/dir libattr.so.1 => /usr/lib/libattr.so.1 (0x40164000)
/bin/groups libcrack.so.2 => /usr/lib/libcrack.so.2 (0x4004e000)
/bin/install libattr.so.1 => /usr/lib/libattr.so.1 (0x40027000)
/bin/ls libattr.so.1 => /usr/lib/libattr.so.1 (0x40164000)
/bin/mv libattr.so.1 => /usr/lib/libattr.so.1 (0x40027000)
/bin/su libcrack.so.2 => /usr/lib/libcrack.so.2 (0x4004e000)
/bin/vdir libattr.so.1 => /usr/lib/libattr.so.1 (0x40164000)
/lib/libacl.so libattr.so.1 => /usr/lib/libattr.so.1 (0x40011000)
/lib/libacl.so.1 libattr.so.1 => /usr/lib/libattr.so.1 (0x40011000)
/lib/libacl.so.1.1.0 libattr.so.1 => /usr/lib/libattr.so.1 (0x40011000)
/lib/libnss_ldap-2.3.2.so libldap.so.2 => /usr/lib/libldap.so.2 (0x40020000)
liblber.so.2 => /usr/lib/liblber.so.2 (0x40048000) libsasl2.so.2 =>
/usr/lib/libsasl2.so.2 (0x401a2000) libkrb5.so.3 => /usr/lib/libkrb5.so.3
(0x401b3000) libssl.so.0.9.7 => /usr/lib/libssl.so.0.9.7 (0x4020a000)
libcrypto.so.0.9.7 => /usr/lib/libcrypto.so.0.9.7 (0x40234000) libk5crypto.so.3
=> /usr/lib/libk5crypto.so.3 (0x40306000) libcom_err.so.3 =>
/usr/lib/libcom_err.so.3 (0x40325000)
/lib/libnss_ldap.so.2 libldap.so.2 => /usr/lib/libldap.so.2 (0x40020000)
liblber.so.2 => /usr/lib/liblber.so.2 (0x40048000) libsasl2.so.2 =>
/usr/lib/libsasl2.so.2 (0x401a2000) libkrb5.so.3 => /usr/lib/libkrb5.so.3
(0x401b3000) libssl.so.0.9.7 => /usr/lib/libssl.so.0.9.7 (0x4020a000)
libcrypto.so.0.9.7 => /usr/lib/libcrypto.so.0.9.7 (0x40234000) libk5crypto.so.3
=> /usr/lib/libk5crypto.so.3 (0x40306000) libcom_err.so.3 =>
/usr/lib/libcom_err.so.3 (0x40325000)
/sbin/fsck.cramfs libz.so.1 => /usr/lib/libz.so.1 (0x40021000)
/sbin/mkfs.cramfs libz.so.1 => /usr/lib/libz.so.1 (0x40021000)
/sbin/mount.smbfs libcrypt.so.1 => /usr/lib/libcrypt.so.1 (0x40017000)
libresolv.so.2 => /usr/lib/libresolv.so.2 (0x40044000) libnsl.so.1 =>
/usr/lib/libnsl.so.1 (0x40055000) libdl.so.2 => /usr/lib/libdl.so.2 (0x40069000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x4006c000) libcom_err.so.3 =>
/usr/lib/libcom_err.so.3 (0x4008c000) libkrb5.so.3 => /usr/lib/libkrb5.so.3
(0x4008e000) libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x400e2000)
liblber.so.2 => /usr/lib/liblber.so.2 (0x400f2000) libldap.so.2 =>
/usr/lib/libldap.so.2 (0x400fc000) libsasl2.so.2 => /usr/lib/libsasl2.so.2
(0x40256000) libssl.so.0.9.7 => /usr/lib/libssl.so.0.9.7 (0x4026a000)
libcrypto.so.0.9.7 => /usr/lib/libcrypto.so.0.9.7 (0x40293000)
/sbin/rpc.statd libwrap.so.0 => /usr/lib/libwrap.so.0 (0x40021000)
/sbin/xfsdump libattr.so.1 => /usr/lib/libattr.so.1 (0x40024000)
/sbin/xfsrestore libattr.so.1 => /usr/lib/libattr.so.1 (0x40024000)


Expected Results:  
Empty output

Portage 2.0.48-r7 (default-x86-1.4, gcc-3.3.1, glibc-2.3.2-r3)
=================================================================
System uname: 2.6.0-test3-mm1 i686 AMD Duron(tm)
GENTOO_MIRRORS="http://gentoo.oregonstate.edu
http://distro.ibiblio.org/pub/Linux/distributions/gentoo"
CONFIG_PROTECT="/etc /var/qmail/control /usr/kde/2/share/config
/usr/kde/3/share/config /var/bind /usr/X11R6/lib/X11/xkb
/usr/kde/3.1/share/config /usr/share/config"
CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d"
PORTDIR="/usr/portage"
DISTDIR="/mnt/gentoo-os/distfiles"
PKGDIR="/mnt/gentoo-os/packages/athlon-xp-gcc3.3"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR_OVERLAY="/mnt/gentoo-os/portage-local"
USE="x86 oss 3dnow apm avi crypt cups encode foomaticdb gif jpeg kde libg++ mad
mikmod mmx mpeg ncurses nls pdflib png quicktime spell truetype xml2 xmms xv
zlib gdbm berkdb slang readline arts java mysql X sdl gpm tcpd pam libwww perl
python imlib oggvorbis gnome gtk qt motif opengl acl gtk2 ipv6 kerberos ldap ssl
acpi alsa apache2 mozilla sse nptl esd gstreamer imap mozp3p mozsvg mozxmlterm
samba sasl slp xfs -svga odbc"
COMPILER="gcc3"
CHOST="i686-pc-linux-gnu"
CFLAGS="-march=athlon-xp -Os -pipe"
CXXFLAGS="-march=athlon-xp -Os -pipe"
ACCEPT_KEYWORDS="x86 ~x86"
MAKEOPTS="-j2"
AUTOCLEAN="yes"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
FEATURES="sandbox ccache buildpkg"

mmm nice analysis :) 
 
i agree with all the proposed package changes ... now to get the ppl who maintain then 
to look them over ... 

# A search of my system with only returns stuff for cramfs, I'm mostly running
~x86
----------------------------
for d in /bin /sbin /lib; do
        cd $d
        file * | grep ELF | cut -d : -f 1 | while read fname ; do
                libs=$(ldd $fname | awk '{print $3}')
                for lib in $libs ; do
                        if [ "$lib" != "dynamic" ]; then
                                if [ ! -f "$lib" ]; then
                                        echo "$fname has a broken link to $lib"
                                else
                                        base_lib=$(basename $lib)
                                        [ ! -f /lib/${base_lib} ] && echo
"$fname is linked to $lib : /lib/${base_lib}"
                                fi
                        fi
                done
        done
done

solar, you will need acl patches for coreutils (like theese at the bottom of
the mentioned bug 18151) and USE="acl" to have ls,cp etc. linked against libacl
and libattr.

I was just looking at debian packages, it seems they have all (most?) that
stuff in proper location.

This do not belong to me - attr,acl,nss_ldap,etc is not under my supervision.

It seems I was wrong about mount.smbfs, it has to be in /sbin otherwise mount
-t smfs will not work.

Another one for the list:
app-shells/zsh -> dev-libs/libpcre
/bin/zsh libpcre.so.0 => /usr/lib/libpcre.so.0 (0x4001c000)
/bin/zsh-4.1.1 libpcre.so.0 => /usr/lib/libpcre.so.0 (0x4001c000)

For a moment, I want to disect the nss_ldap dependancies:
ldd /lib/libnss_ldap-2.3.2.so | grep -v ' /lib':
(from openldap)
libldap.so.2 => /usr/lib/libldap.so.2 (0x4001d000)
liblber.so.2 => /usr/lib/liblber.so.2 (0x40056000)
(from db)
libdb-4.1.so => /usr/lib/libdb-4.1.so (0x40063000)
(from cyrus-sasl)
libsasl2.so.2 => /usr/lib/libsasl2.so.2 (0x40274000)
(from kerberos)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x40289000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x402ec000)
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x402ff000)
(from openssl)
libssl.so.0.9.6 => /usr/lib/libssl.so.0.9.6 (0x40302000)
libcrypto.so.0.9.6 => /usr/lib/libcrypto.so.0.9.6 (0x40332000)

nss_ldap-210:
net-nds/openldap
kerberos? ( app-crypt/mit-krb5 )
sasl? ( dev-libs/cyrus-sasl )
dev-libs/openssl
>=sys-libs/db-4.1*

as such, while having all binaries in the root filesystem usable without /usr is a laudable goal, there is absoltely no way it is practical under all circumstances.

A good number of the things in /sbin /bin /lib are there because other things require them to be there (/lib/security for example, contains many unresolved symbols even, but this isn't an error, it's meant that way since PAM dlopens the files there as needed).

To get around this, I propose one additional constraint. everything crucial to the system (eg the system profile) and anything needed for booting a basic system should work without /usr being mounted.

Looking at your list here, I am willing to implement this set of changes:
zlib : move to root filesystem
attr : move to root filesystem
acl : move to root filesystem

What about tcp-wrappers, having them in /lib is important for people mounting
/usr over nfs.

acl, attr done.

Todo still, with sizes of stuff that would need to exist on the root partition:
sys-libs/cracklib - 2Mb
sys-apps/tcp-wrappers - 40kb
dev-libs/libpcre - 87kb
sys-libs/zlib - 64kb

Due to this, i'm strongly inclined to _not_ move cracklib at this time, as I don't want to bloat root partitions.

libpcre and zlib i'll move tonight probably, tcp-wrappers in a few days.

I'm not sure moving libpcre to /lib is such a good idea. Granted, 
87K isn't that much, but it looks like zsh is the only program 
in /bin that currently uses it; and I'd guess a large number of 
users don't even use zsh. Also, zsh is in no way essential for 
boot-up or system maintenance and isn't even usable to its fullest
without all the plugins that still go into /usr. Don't get me wrong, 
I like zsh, but still don't consider it important enough. Another 
thing to consider is that some programs go into /bin for 
historical reasons and still won't work until /usr is mounted, so 
it is probably ok if zsh is in /usr/bin with a compatibility 
symlink in /bin.

(another example is /bin/login; doesn't directly depend on stuff 
in /usr, but indirectly via pam: for one cracklib as mentioned above, 
another one would be /usr/lib/glib.so via pam_console; but, not 
needed until /usr is mounted so no problem here...)

Just discovered another one: /bin/mail!

uses: 
  /usr/lib/mail.help
  /usr/lib/mail.tildehelp

Again, not needed for bootup and other distributions put it in /usr/bin.

Cracklib is only used by /bin/groups (not counting /lib/security stuff which is
dlopen'ed):
ldd /bin/groups
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x40021000)
        libcrack.so.2 => /usr/lib/libcrack.so.2 (0x4004e000)
        libc.so.6 => /lib/libc.so.6 (0x40058000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
'groups' however doesn't need neither libcrack nor libcrypt (there is a related
bug 27071).

Along the lines of nss-ldap; nss-mysql and libnss-mysql also have a dependency
in /usr (particularly: libmysqlclient.so.12 => /usr/lib/libmysqlclient.so.12).
 However, I would say that people using alternative nss libraries should
still have system accounts in "files" so this isn't that big of a problem.

For /bin/mail ... it is not the same as /usr/bin/mail ... /bin/mail is a
simple mail client, where /usr/bin/mail historically (??) comes from sendmail
and do not have the same functionality.  I may have this wrong though, and
I may be even more wrong, but I think even FHS have something about this
... 

I notice that tcp-wrappers (namely libwrap) has not been
moved to /lib yet.
This is causing me problems because I want to mount
/usr over NFS.
Is this still on the todo list?

libz seems not to have been moved yet as well...

zlib, tcpwrappers moved now.
any further reason for this bug to remain open, given that libpcre/cracklib have little reason to move?

closing, no response for a month and seems to be resolved.

I just wanted to note this here, for posterity.  I am reverting the change in
zsh due to http://bugs.gentoo.org/show_bug.cgi?id=182534#c17 which shows that
it is safe.  It also allows me to resolve that bug.