The problems are within udev rules. Many packages now install their own udev rules. Those rules sometimes depend on programs in /usr and /opt. This can be a problem on systems where /usr and /opt are partitioned separately from /. Particularly at boot time. This is what happens: 1. mount / 2. start udev 3. mount /usr and /opt When udev starts, it invokes udev rules. When udev rules try to run programs that are on /usr or /opt, they will fail because /usr and /opt have not been mounted yet. I have already reported this as a bug in virtualbox (bug #368215) and as a bug in alsa-utils (bug #369699). But after reading the following article, I decided that the problem is possibly more fundamental. http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken The installation handbook uses this partition layout as an example. $ df -h Filesystem Type Size Used Avail Use% Mounted on /dev/sda5 ext3 509M 132M 351M 28% / /dev/sda2 ext3 5.0G 3.0G 1.8G 63% /home /dev/sda7 ext3 7.9G 6.2G 1.3G 83% /usr /dev/sda8 ext3 1011M 483M 477M 51% /opt /dev/sda9 ext3 2.0G 607M 1.3G 32% /var /dev/sda1 ext2 51M 17M 31M 36% /boot /dev/sda6 swap 516M 12M 504M 2% <not mounted> It was once a very good example. I've had a similar layout for years. But now I think /, /usr and /opt should to be merged. And this needs to suggested to new Gentoo users. Otherwise they will experience problems. Reproducible: Always
That's just one sample layout. If you actually read through the instructions, you'll see that the examples used throughout the installation process are just three partitions: /boot, /, and swap. The solution is to either fix udev (especially where it installs/reads its files from), or not blindly follow examples, instead tweaking for your own system's needs.
I understand that they are just examples, but new users will often follow the examples and split /, /usr and /opt. Also, two other installation documents suggest separating /usr and /opt. I had intended to include them in the description, but I was tired and I forgot. http://www.gentoo.org/doc/en/lvm2.xml http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml (In reply to comment #1) > The solution is to either fix udev (especially where it installs/reads its > files from) That was my first thought. That's why I had been reporting bugs in the udev rules of alsa-utils and virtualbox (and there are lots of other apps that install similarly problematic udev rules). But after doing some research, I learned that developers are increasingly marking such bug reports as WONTFIX. Their preferred solution, it seems, is for users to simply not have separate partitions for /usr and /opt. I'm reporting this bug here in the documentation because people have asked me to. For their sake, please could this bug remain open to provide a place to discuss this issue. I think it is an issue that needs to be discussed. Not a major one, but still an issue.
Rather than change the example in the handbook, perhaps it would be better to add a note informing users that separating /usr and /opt from / causes problems with udev rules.
Bugzilla is not a place for discussion. That belongs on the mailing lists and with the ebuild maintainers for alsa/udev/etc. The documentation will not be modified.
Ok, but I think that bug reports relating to this udev situation will be more frequent in the near future when certain packages are marked stable. Something will eventually need to be done. I genuinely believe a brief note in the docs would prevent problems for many people. But, of course, it's your decision. I won't be pushing this bug any further.
Josh, the issue discussed here is very real. Other distributions have already deprecated separate /usr for this reason. I do agree with you that having udev break when no /usr is available is unfortunate, but we have to work with certain constraints, and fixing udev is not really an option for the GDP. I suggest to either completely remove the discussion of a separate /usr, or change the text to clearly read that having random mount points will lead to trouble in future.
(In reply to comment #6) > Josh, the issue discussed here is very real. Other distributions have already > deprecated separate /usr for this reason. I do agree with you that having udev > break when no /usr is available is unfortunate, but we have to work with > certain constraints, and fixing udev is not really an option for the GDP. > > I suggest to either completely remove the discussion of a separate /usr, or > change the text to clearly read that having random mount points will lead to > trouble in future. Two solutions: 1. Reassign/retitle this bug; give it to the udev maintainers. IIRC gregkh, one of our Gentoo developers, is also udev upstream. Fundamentally, udev and/or all those apps that ask to be called from /usr (or /opt)...that's all broken, so it should be fixed. 2. Remove the bit about separate /usr partitions in the examples. You're welcome to tackle all those handbook files if y'really want to, but I'm not going to do it.The handbooks don't have instructions for users creating separate /usr directories, anyway; it's only that single sample config.
(In reply to comment #7) > Two solutions: > > 1. Reassign/retitle this bug; give it to the udev maintainers. IIRC gregkh, one > of our Gentoo developers, is also udev upstream. Fundamentally, udev and/or all > those apps that ask to be called from /usr (or /opt)...that's all broken, so it > should be fixed. There's nothing in udev itself that needs fixing. The problem is with other packages that install udev rules. And the only way of fixing those is to have them copy or move files from /usr. I've been trying that and it's very inelegant. Upstream developers don't like it either. They won't fix their packages, so any such fixes would have to be maintained by ebuild maintainers. Which would also be quite inelegant. > 2. Remove the bit about separate /usr partitions in the examples. You're > welcome to tackle all those handbook files if y'really want to, but I'm not > going to do it.The handbooks don't have instructions for users creating > separate /usr directories, anyway; it's only that single sample config. For the handbooks that use the 20GB laptop sample config, we just need a new sample config without a separate /usr. Then the text "/usr is rather full (83% used) here, but once all software is installed, /usr doesn't tend to grow that much." can be removed. If "(Unpartitioned space for future usage: 2 GB)" can also be removed, that will mean no translation to other languages will be necessary. These two quick install documents will also need to be changed. Their instructions include the creation of a separate /usr. http://www.gentoo.org/doc/en/lvm2.xml http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml I can help with this, if you want. I have checked out the xml files from cvs and can provide patches.
(In reply to comment #8) > If "(Unpartitioned space for future usage: 2 GB)" can also be removed, > that will mean no translation to other languages will be necessary. Come to think of it, that doesn't need to be removed. It's already been translated. Sorry about that. Brain fart.
Isn't this the point of the udev-postmount init script?
(In reply to comment #10) > Isn't this the point of the udev-postmount init script? Yes, udev-postmount will re-execute run lists that failed when they were first triggered. They might fail the first time because / is read-only, or a certain resource isn't available yet, or whatever. It does work around the issue being discussed here, but is it good enough as a permanent solution? I see it as more of a workaround. I've just had a look at udev's git tree, and the "--type=failed" option, which udev-postmount depends on, has been removed from the udevadm man page since udev-167. The fact that they no longer document it suggests that it might not be there forever. http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;h=864fde8a087c0edbc0ee3aca83f9289fc32cfcee
to be clear, unified / and /usr is not a general requirement in Gentoo and i dont foresee it changing any time soon. the opposite is generally true.
(In reply to comment #12) > to be clear, unified / and /usr is not a general requirement in Gentoo and i > dont foresee it changing any time soon. the opposite is generally true. It would be a shame to make it a requirement. I don't want that. I still want the option of a separate /usr. Personally, I've always preferred having a separate /usr. It's been that way on my system for years. But times they are a-changin. It's now considered ok for udev rules to make use of resources on /usr. Lots of apps do this now. And I don't think it's going to stop. To be honest, I don't really mind. I actually quite like having /usr and / unified. I didn't really need to keep them separate, and it means udev is less limited at boot time. Anyway, users that separate /usr from / are going to increasingly experience problems. Fewer distros are doing it. It's becoming the path less trodden. So let's not encourage new users to do it. Experienced users, that's fine. They can fix stuff, report bugs, send patches. But new users reading the handbook need to be shown good examples of configurations that will work.
sorry if i wasnt clear, but my point was that *neither* setup is a requirement. both should work -- split or unified.
(In reply to comment #14) > sorry if i wasnt clear, but my point was that *neither* setup is a requirement. > both should work -- split or unified. I did understand that. I know there's no chance of separate /usr being unsupported. I just think it would be a good idea to encourage people installing new systems to not separate /usr from /. It will cause problems in the future, mostly with ~arch systems, but stable systems won't be unaffected. I'm not so impassioned about this issue now that I no longer have problems on my system. And changing the documentation isn't going to solve the problems, although it might help new users avoid experiencing them. I'm going to close this bug now, but if someone else thinks this is a good idea they're welcome to reopen it. I'm not going to reopen bug #368215 or bug #369699, but someone else may want to. One final thought, ebuild maintainers might need some convincing that this is a problem, or I expect that some bug reports will be marked WONTFIX.
My thought is that we have a /bin and /sbin specifically for executeables that are needed when only / is mounted. Applications that install udev rules should therefore place the required files in /bin or /sbin rather than /usr/bin or /usr/sbin.
(In reply to comment #16) > My thought is that we have a /bin and /sbin specifically for executeables that > are needed when only / is mounted. Applications that install udev rules should > therefore place the required files in /bin or /sbin rather than /usr/bin or > /usr/sbin. I agree with that. But apps _are_ placing required files in /usr/bin, /usr/sbin, and then they often depend on libraries in /usr/lib and read files from /usr/share. Everyone here seems to agree that the way forward is to fix the broken packages. But that isn't really happening. We need more bug reports. And I should probably reopen bug #368215 and bug #369699.
i'm not sure the apps are broken. if i have a camera package that installs udev rules, i dont expect all those camera packages (and the libraries it uses) to be in my root. i expect those to get run after /usr is available (i.e. udev-postmount).
(In reply to comment #18) > i'm not sure the apps are broken. if i have a camera package that installs > udev rules, i dont expect all those camera packages (and the libraries it uses) > to be in my root. i expect those to get run after /usr is available (i.e. > udev-postmount). That is a good point. I just don't like warning messages. And I don't really like udev-postmount. But maybe that's my problem.
(In reply to comment #18) > i'm not sure the apps are broken. if i have a camera package that installs > udev rules, i dont expect all those camera packages (and the libraries it uses) > to be in my root. i expect those to get run after /usr is available (i.e. > udev-postmount). Either the apps are broken or udev should handle the resulting 'errors' gracefully and run the failed udev rules after all local partitions are mounted without complaint (or warning).
(In reply to comment #20) > Either the apps are broken or udev should handle the resulting 'errors' > gracefully and run the failed udev rules after all local partitions are mounted > without complaint (or warning). I don't know if there is a practical way that udev could be made to handle those errors gracefully; and udev-postmount actually does re-run the stuff that failed. I find udev-postmount a bit ugly, but it works. The error messages from the initial fail bother me somewhat. I guess they could be sent to /dev/null, but they'd still be "there". I've been thinking about this a lot, and there doesn't seem to be an elegant solution. Perhaps there is nothing to be done at this time. I am a little bit uneasy about udev-postmount, not just because it's not pretty, but because it works by running 'udevadm trigger --type=failed' and udev upstream may possibly remove the --type=failed functionality at some point. I don't know that for sure. I should ask someone from udev. But from reading the git logs it seems they don't like that feature very much.
(In reply to comment #18) > i'm not sure the apps are broken. if i have a camera package that installs > udev rules, i dont expect all those camera packages (and the libraries it uses) > to be in my root. i expect those to get run after /usr is available (i.e. > udev-postmount). Maybe "broken" wasn't the right word. Technically things are working. But the way it currently works is a bit laboured: 1. package "foo" installs a udev rule, /lib/udev/rules.d/99-foo.rules 2. same package installs /usr/bin/foo 3. at boot time, 99-foo.rules invokes /usr/bin/foo which doesn't exist yet 4. udev records that it failed to run /usr/bin/foo 5. boot process continues, /usr is mounted 6. udev-postmount tells udev to re-run anything that previously failed I want to make udev-postmount unnecessary. Specifically the "udevadm trigger --failed" part. I think it is possible, but probably not without changing udev itself. This is not related to the documentation anymore, so we (mostly I) are (am) spamming the documentation team. Sorry, Josh. This discussion, if it continues, will be on the mailing lists.
