Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 437320 - stage3-amd64-20120917 is missing /dev/console, boot hangs unless kernel has DEVTMPFS_MOUNT=y
Summary: stage3-amd64-20120917 is missing /dev/console, boot hangs unless kernel has D...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Release Media
Classification: Unclassified
Component: Stages (show other bugs)
Hardware: AMD64 Linux
: Normal major (vote)
Assignee: Gentoo Release Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-05 16:22 UTC by Ben Kohler
Modified: 2012-11-01 17:47 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ben Kohler gentoo-dev 2012-10-05 16:22:51 UTC
stage3-amd64-20120917.tar.bz2 is missing /dev/console, this causes boots to hang just before init starts (the typical "Freeing unused kernel memory..." being the last line shown).  This can be worked around by enabling CONFIG_DEVTMPFS_MOUNT but this isn't what the handbook advises so many new users are getting hung up here.

# tar xpf stage3-amd64-20120917.tar.bz2 -C /mnt/gentoo
# ls -l /mnt/gentoo/dev
-rw-r--r-- 1 root root 0 Sep 17 03:38 null
#
Comment 1 Rafał Mużyło 2012-10-05 16:34:07 UTC
That's a pretty bad description of the problem.

If I'm reading this correctly, the real problem is that the docs don't stress that CONFIG_DEVTMPFS is mandatory with recent udev (though I don't think CONFIG_DEVTMPFS_MOUNT is).
Comment 2 Ben Kohler gentoo-dev 2012-10-05 16:35:27 UTC
This happens with CONFIG_DEVTMPFS=y enabled, AND on old udev.  This is an issue with /dev/console being required for boot (init), and it being included on all previous stage3 tarballs, but not this one.
Comment 3 kfm 2012-10-06 14:33:20 UTC
This is reminiscent of bug 368597. For whatever reason, the usual complement of static device nodes provided by sys-apps/makedev with USE="build" in effect is not making it into the stage tarballs.
Comment 4 kfm 2012-10-10 13:15:43 UTC
Just to add that the reason it's affecting so many is that DEVTMPFS_MOUNT is only a default for architectures that are relatively obscure:

# find arch -name '*defconfig' -exec grep -l DEVTMPFS_MOUNT=y {} \;
arch/arm/configs/at91sam9g45_defconfig
arch/arm/configs/imote2_defconfig
arch/arm/configs/imx_v4_v5_defconfig
arch/arm/configs/imx_v6_v7_defconfig
arch/arm/configs/marzen_defconfig
arch/mips/configs/nlm_xlp_defconfig
arch/mips/configs/nlm_xlr_defconfig
arch/openrisc/configs/or1ksim_defconfig
arch/powerpc/configs/44x/currituck_defconfig
arch/powerpc/configs/mpc512x_defconfig
arch/tile/configs/tilegx_defconfig
arch/tile/configs/tilepro_defconfig
arch/um/defconfig

In any case, this should not be a mandatory option in Gentoo. I'm taking the liberty of bumping the priority because it's broadly harming the new user experience.

Another way to work around this problem (for most) would be to have genpatches start modifying defconfig for x86 and to fast-track a stable bump of gentoo-sources. Mike, do you think this is worth doing?
Comment 5 Alon Bar-Lev 2012-10-10 13:24:17 UTC
I just created /dev/console at the filesystem, it solved this issue.

It is embarrassing as I am trying to push gentoo to people, and this last install-cd and stages made me look like a joke. I never had so many issues during installation.
Comment 6 Mike Pagano gentoo-dev 2012-10-10 20:14:48 UTC
Thanks, Kerin for CC'ing me.  What I would like to do is work on patches that could eventually make it upstream.

Just patching Kconfig to build in DEVTMPFS_MOUNT and DEVTMPFS will work, but it's something we would have to carry forever and never see accepted by upstream.

But, Linus has detailed how he would like to see distro configs make it into the kernel.

I will work on this patchset, and initially carry it in gentoo-sources, with the long term goal of seeing it go upstream.

Two questions:

What kernel versions are we looking to see this in?
Do we need both DEVTMPFS_MOUNT and DEVTMPFS?
Comment 7 kfm 2012-10-10 20:33:02 UTC
Hi Mike. Yes, it would be better for the defconfig to be changed upstream so as to be more broadly useful. It would be better yet for the stage tarball to be fixed. I'm floating the idea of patching in DEVTMPFS_MOUNT purely as an interim measure so that new users don't get burned.

If you don't mind doing that then, at the very least, one would need to bump current stable (3.4.9) - perhaps the current ~arch version also. CONFIG_DEVTMPFS_MOUNT implies - and requires - CONFIG_DEVTMPFS so both would need to be present. Covering x86 is of the most importance. If you'd prefer for a more comprehensive approach then let me know as I have a script to generate a suitable patch.
Comment 8 Alon Bar-Lev 2012-10-10 20:34:08 UTC
(In reply to comment #6)
> Just patching Kconfig to build in DEVTMPFS_MOUNT and DEVTMPFS will work, but
> it's something we would have to carry forever and never see accepted by
> upstream.

Why not just create /dev/console?
Comment 9 kfm 2012-10-10 20:46:21 UTC
(In reply to comment #8)
> (In reply to comment #6)
> > Just patching Kconfig to build in DEVTMPFS_MOUNT and DEVTMPFS will work, but
> > it's something we would have to carry forever and never see accepted by
> > upstream.
> 
> Why not just create /dev/console?

A better question would be: how long is it going to take releng to address the regressions that are adversely affecting the stage generation process? I sense that it could take a while. In the meantime, forcing DEVTMPFS_MOUNT in gentoo-sources would help to ensure that new users are unlikely to encounter the bug during installation. I'm not suggesting that it would bring this bug to a state of closure.
Comment 10 Alon Bar-Lev 2012-10-10 20:58:32 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #6)
> > > Just patching Kconfig to build in DEVTMPFS_MOUNT and DEVTMPFS will work, but
> > > it's something we would have to carry forever and never see accepted by
> > > upstream.
> > 
> > Why not just create /dev/console?
> 
> A better question would be: how long is it going to take releng to address
> the regressions that are adversely affecting the stage generation process? I
> sense that it could take a while. In the meantime, forcing DEVTMPFS_MOUNT in
> gentoo-sources would help to ensure that new users are unlikely to encounter
> the bug during installation. I'm not suggesting that it would bring this bug
> to a state of closure.

This is not the right approach. As people build custom kernels and may disable this or use the vanila.

The solution should be to fix the staging, not break what is not broken.
Comment 11 kfm 2012-10-10 21:12:38 UTC
(In reply to comment #10)
> This is not the right approach. As people build custom kernels and may
> disable this or use the vanila.

Yes, they might. That does not render such a maneuver worthless.

> 
> The solution should be to fix the staging, not break what is not broken.

Nobody is claiming otherwise. Note also that Mike is not a member of the releng herd. Indeed, nobody from releng has commented yet. Whether you consider the approach proper or not, the fact is that it would prevent a majority of new users from running into this problem right *now*. Why you would object to this is mystifying to me, especially as you clearly have strong feelings about the importance of the new user experience (as indicated in Comment 5).

Furthermore, although /dev/console missing is to blame, there should actually be a fully fledged set of static device nodes (as per Comment 3). Who knows how long it will take for the cause to be established and for well-formed stage tarballs to hit the mirrors? Just look at prior bugs of a similar ilk.
Comment 12 William Hubbs gentoo-dev 2012-10-10 21:28:50 UTC
All,

We are populating the /dev directory in the stages correctly from the
makedev ebuild if stages are being built.

Here is more information that jmbsvicetto gave me on IRC:

<@jmbsvicetto> WilliamH: I suspect that issue might be transient and
related to the new box. If anyone wants to confirm, just check the latest
stage on my server
<@jmbsvicetto> http://www.jmbsvicetto.name/releases/amd64/
  <@jmbsvicetto> If the stages on my server have /dev/console, then it's most
likely an issue with skimmer that I hope to fix tomorrow
 <@jmbsvicetto> Before someone complains about it, we're currently in a state
 of flux as I'm trying to move away from the old box (poseidon) to the new
 box (skimmer).  The new box already builds stages, although it's
 missing the cron entries, we still have to take care of the mirroring
 and, I suspect, it may have a few issues related to the build of stages
 on a hardened box
Comment 13 William Hubbs gentoo-dev 2012-10-10 21:33:31 UTC
Mike,

I would recommend holding off on patching this into the kernel. Let's
see what releng comes up with; this should be taken care of quickly by
them.
Comment 14 kfm 2012-10-10 21:39:27 UTC
(In reply to comment #12)

[snip]

>  and, I suspect, it may have a few issues related to the build of stages
>  on a hardened box

GRKERNSEC_CHROOT_MKNOD and GRKERNSEC_CHROOT_CHMOD are possible offenders. They can be disabled through the sysctl interface, provided that grsec_lock hasn't been activated.
Comment 15 Mike Pagano gentoo-dev 2012-10-10 23:09:16 UTC
(In reply to comment #13)
> Mike,
> 
> I would recommend holding off on patching this into the kernel. Let's
> see what releng comes up with; this should be taken care of quickly by
> them.

No problem. I'll hold off. Just let me know if I can be of any help.
Comment 16 Jorge Manuel B. S. Vicetto (RETIRED) Gentoo Infrastructure gentoo-dev 2012-10-14 01:38:10 UTC
I've confirmed this was caused by a grsec issue in the new build box.

tar tjvf buildroot/amd64-dev/builds/default/stage3-amd64-20121013.tar.bz2  | grep /dev/console
crw------- root/root          5,1 2012-10-13 14:32 ./dev/console

As soon as the stages finish to build, I'm going to push them to the mirrors and drop the 20120917 stages.
Comment 17 kfm 2012-10-14 02:24:22 UTC
Thanks for the update. As a matter of curiosity, may I ask what precisely was the issue with grsec?
Comment 18 Jorge Manuel B. S. Vicetto (RETIRED) Gentoo Infrastructure gentoo-dev 2012-10-14 05:35:08 UTC
(In reply to comment #17)
> Thanks for the update. As a matter of curiosity, may I ask what precisely
> was the issue with grsec?

To use catalyst on a grsec enabled kernel, one needs to disable some restrictions:

echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_chmod
echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mknod
echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_chroot
echo 0 > /proc/sys/kernel/grsecurity/linking_restrictions
echo 0 > /proc/sys/kernel/grsecurity/chroot_caps

The issue here was that while setting up the box, there was some miscomunication between me and the infra member and so the restrictions weren't lifted - causing the stage build to fail to create the devices.
Comment 19 Ben Kohler gentoo-dev 2012-10-14 17:28:52 UTC
I have just finished a fresh install this morning with stage3-amd64-20121013.tar.bz2 and the issue appears to be resolved.
Comment 20 Jorge Manuel B. S. Vicetto (RETIRED) Gentoo Infrastructure gentoo-dev 2012-11-01 17:47:37 UTC
I'm closing this bug as the grsec restrictions on skimmer were lifted and the latest stage on mirrors has all the /dev nodes.