After an emerge -DuN @world this morning that included OpenRC, busybox, udev-init-scripts, kmod and a new gentoo-sources-3.14.29 kernel I built the kernel as usual and attempted to boot into it. It failed to boot Reproducible: Always Steps to Reproduce: 1. emerge -DuN @world 2. build & install kernel 3. reboot Actual Results: When it fails the kernel is booting but then drops into busybox telling me something like Line 16: mount not found Line 17: mount not found Line 18: mount not found (I'll get the exact wording if needed.) As best I can tell these new updates are not accepting my initramfs setup that I've used for the last few years. 3.14.28 boots fine. I build initramfs into the kernel. Expected Results: Should boot. I will attach the emerge logs & initramfs & kernel config files after posting this report. c2RAID6 ~ # emerge --info Portage 2.2.14 (python 2.7.9-final-0, default/linux/amd64/13.0/desktop/kde, gcc-4.8.3, glibc-2.19-r1, 3.14.28-gentoo x86_64) ================================================================= System uname: Linux-3.14.28-gentoo-x86_64-Intel-R-_Core-TM-_i7_CPU_X_980_@_3.33GHz-with-gentoo-2.2 KiB Mem: 24685632 total, 22102732 free KiB Swap: 12582904 total, 12582904 free Timestamp of tree: Sun, 18 Jan 2015 13:45:01 +0000 ld GNU ld (Gentoo 2.24 p1.4) 2.24 app-shells/bash: 4.2_p53 dev-java/java-config: 2.2.0 dev-lang/perl: 5.18.2-r2 dev-lang/python: 2.7.9-r1, 3.3.5-r1 dev-util/cmake: 2.8.12.2-r1 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.13.7 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.11.6-r1, 1.13.4 sys-devel/binutils: 2.24-r3 sys-devel/gcc: 4.7.3-r1, 4.8.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2-r1 sys-devel/make: 4.0-r1 sys-kernel/linux-headers: 3.16 (virtual/os-headers) sys-libs/glibc: 2.19-r1 Repositories: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA dlj-1.1 PUEL AdobeFlash-10.3 skype-eula google-chrome skype-4.0.0.7-copyright google-talkplugin Google-TOS" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=native -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=native -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --jobs=5" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://gentoo.osuosl.org/ " LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j13 -l8" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa amd64 berkdb branding bzip2 cairo cdda cdr cleartype cli consolekit corefonts cracklib crypt cxx dbus declarative dri dts dvd dvdr emboss encode exif fam firefox flac fortran gdbm gif glamor gpm gstreamer gtk iconv java jpeg jpeg2k kde kipi lcms ldap libnotify mad mmx mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds qt3support qt4 readline sdl semantic-desktop session spell sse sse2 ssl ssse3 startup-notification svg tcpd tiff truetype type1 udev udisks unicode upower usb vdpau vorbis wxwidgets x264 xcb xcomposite xinerama xml xscreensaver xv xvid xvmc zlib" ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USE_PYTHON="2.7 3.3" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS c2RAID6 ~ #
Created attachment 394276 [details] initramfs config 1
Created attachment 394278 [details] initramfs_init_new.sh I believe the line 16/17/18 messages come from this file.
Created attachment 394280 [details] mdadm config
Created attachment 394282 [details] 3.14.29 kernel config
Created attachment 394284 [details] Partial emerge log showing updates this morning
The exact language at boot fail time is: /init: line 16: mount: not found /init: line 17: mount: not found /init: line 18: mount: not found which I believe correspond to lines 16-18 in initramfs_init_new.sh: mount -t proc none /proc mount -t sysfs none /sys mount -t devtmpfs none /dev
It's really hard to tell what is going on here. Please use our forums, IRC channels or mailing lists to get support. When you find an actual bug, please report it on this bug tracker.
Please do not mark this report as 'Invalid'. It's unconfirmed. As requested I'll look for support elsewhere and post back here when I find an answer. However the machine doesn't boot and this busybox update appears to be the cause. If it's determined not to be the cause I'll gladly mark it as invalid. Unless the report is here there's no chance the busybox folks will even know this is going on.
There is no bug here. If it were currently settled what is as yet unconfirmed, what is it that we have that would ultimately be confirmed?
Or, to express the issue in a more practical consideration: what if five or a dozen people jumped on board saying they have *the very same* but *vague* issue that resembles "sys-apps/busybox-1.23.0 fails to assemble RAID6", and you ended up with 200 comments figuring out which multiple issues you're trying to squeeze into a single bug report?
I masked busybox-1.23.0 and went back to 1.21.0 which I was running until yesterday. gentoo-sources-3.14.29 now builds and boots fine. That means the other 4 updates yesterday are apparently uninvolved with this issue. I am not a developer and I don't know how to identify it more specifically at this point other than to get a developer/package maintainer involved. The problem here is busybox-1.23.0 breaks assembly of my RAID6 at boot time. The following shows 3.14.29 has now boot after masking the new busybox and going back to busybox-1.21.0: mark@c2RAID6 ~ $ uname -a Linux c2RAID6 3.14.29-gentoo #2 SMP PREEMPT Mon Jan 19 11:32:27 PST 2015 x86_64 Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz GenuineIntel GNU/Linux mark@c2RAID6 ~ $ eix -I busybox [I] sys-apps/busybox Available versions: 1.20.2^t 1.21.0^t ~1.21.1^t ~1.22.0^t ~1.22.1^t ~1.22.1-r1^t [m]1.23.0^t [m]**9999^t {debug ipv6 livecd make-symlinks math mdev -pam savedconfig selinux sep-usr +static syslog systemd} Installed versions: 1.21.0^t(11:18:37 01/19/15)(static -ipv6 -livecd -make-symlinks -math -mdev -pam -savedconfig -selinux -sep-usr -syslog -systemd) Homepage: http://www.busybox.net/ Description: Utilities for rescue and embedded systems mark@c2RAID6 ~ $ This is NOT an unspecific, unclear bug report. There *MAY* be multiple things involved here but it's specific, repeatable on my machine and limited to only the busybox update. If there's something about my initramfs setup that's no longer supported then there should be info about that which so far I cannot find. I'm not saying I won't do updates to my setup to move to 1.23.0, only that someone who develops this stuff needs to tell us users what to do. I've marked it again as unconfirmed.
(In reply to Mark Knecht from comment #0) > After an emerge -DuN @world this morning that included OpenRC, busybox, > udev-init-scripts, kmod and a new gentoo-sources-3.14.29 kernel I built the > kernel as usual and attempted to boot into it. It failed to boot > > Reproducible: Always > > Steps to Reproduce: > 1. emerge -DuN @world > 2. build & install kernel > 3. reboot > Actual Results: > When it fails the kernel is booting but then drops into busybox telling me > something like > > Line 16: mount not found > Line 17: mount not found > Line 18: mount not found > When busybox installs, it sets up sym links to the programs it provides. Can you see the contents of the initramfs root before it ts packaged up inside the kernel? If so, let's have an listing of whats in /bin, /sbin, /usr/bin and /usr/sbin
I'll have to look for some instructions on how to do that, and then to do it for both 1.21.0 (works) & 1.23.0 (fails) to see the difference. I don't know how to see what's actually in those directories when the initramfs is built by the kernel, and I've never done it any other way so it may take awhile/ There is another individual on the gentoo-amd64 list who has now seen the same problem. He said he fixed it by doing a "/bin/busybox --install -s" just before the mount commands. I suspect that will likely fix it for me also but need to do it when I can take the machine down. Prior to his report I was considering modifying the main initram config setup: dir /bin 755 0 0 file /bin/busybox /bin/busybox 755 0 0 slink /bin/sh busybox 777 0 0 dir /realroot 755 0 0 and adding slink /bin/mount busybox 755 0 0 to duplicate the sh command. I think that would handle the mount issue but maybe I'd still need mdadm to assemble the RAID. Anyway, pass back some instructions on how to get the data you want and I'll gladly get it ASAP. Thanks!
(In reply to Mark Knecht from comment #13) > There is another individual on the gentoo-amd64 list who has now seen the > same problem. He said he fixed it by doing a "/bin/busybox --install -s" That's how the applets are created. Just add that immediately before the failing mount commands. jer is right though. This is not a bug. There's nothing for the maintainers to do.
I mostly read all the comments here, but still I want to tell few words ... I see these issues on my system too after upgrade, but maybe I'm doing something wrong, and did it wrong for a long time. I use simple and custom initramfs [1], this setup worked fine for me on multiple systems. 'init' (script) is just a shell script which starts with '#!/bin/busybox sh' and it always worked fine, I never did any symlinks, just installed my extra tools (which arent part of busybox) and used them + busybox ('mount' mostly, mdev) in the script, worked good (without symlink). Not sure if it was a magic (probably something I don't understand how it works/worked), but as I said it worked fine, and for a long time without problems. So my initramfs was directory structure with /bin/busybox + /bin|/sbin/... (extra tools I need) and /init. (no symlinks (don't know how, but it worked)). Now I just added '/bin/busybox install -s' as a "workaround" and it does what I need + some extra "error messages" which I need to fix. I'm not a developer and as you can see ^^^ I don't know much about this, especially busybox, but on the busybox webiste it says "23 December 2014 -- BusyBox 1.23.0 (unstable)", I don't want to tell you what to do, but it is strange that if upstream marked its release as unstable, to be stable in gentoo? [1] - https://wiki.gentoo.org/wiki/Custom_Initramfs This is just my comment, I don't want a control of other people, or tell them what to do ... or insult someone, English is not my main language (I learned it while reading ... (not very well)), so if something sounds rude ^^^, sorry it is not how I wanted to say it!
This bug also broke my system (AMD64). After upgrading Busybox from 1.21.0 to (officially unstable) 1.23.0, the initramfs init script failed because busybox commands like mount are no longer being found. The most "official" Gentoo information on manually creating an initramfs I could find is this wiki article: http://wiki.gentoo.org/wiki/Custom_Initramfs#Init The sample script is clearly no longer working with 1.23, as it references busybox commands directly, without prefixing "/bin/busybox" or calling "/bin/busybox --install -s". Moreover, "--install -s" is only mentioned as being required to start a rescue shell, and even that won't work because it calls "busybox --install -s" instead of "/bin/busybox --install -s". So either the available Gentoo documentation is wrong (despite working fine for many years), or the current (officially unstable) Busybox is broken. I adjusted my init script by prefixing "/bin/busybox" to several commands, as I don't really understand what "--install" and "-s" actually do. Busybox documentation seems pretty poor in that regard.
I Have exactly the same Problem after updating from 1.21.0 to 1.23.0. After updating does my custom init script not work, even after "pre-fixing" all commands with "/bin/" it still does not work. I am not 100% sure but I think I can see a "/dev/tty not found" scroll past in the very beginning of the init script (it scrolls past so fast that I am not 100% sure (and "scrollback" does not work on this system)).
(In reply to Stefan Kiesler from comment #16) > So either the available Gentoo documentation is wrong (despite working fine > for many years), or the current (officially unstable) Busybox is broken. > I don't understand why that documentation worked up till now. I have always had to populate my chroot for my initramfs using busybox --install -s. > I adjusted my init script by prefixing "/bin/busybox" to several commands, > as I don't really understand what "--install" and "-s" actually do. Busybox > documentation seems pretty poor in that regard. Do this: 1) USE=static emerge buysbox 2) mkdir -p /tmp/chroot/{bin,sbin} 3) cd /tmp/chroot 4) cp /bin/busybox bin At this point look at what's in your chroot 5) chroot . /bin/busybox --install -s Look again! Afaik busybox as *always* worked this way, the documentation is missing this step, and some lucky happenstance has make that documentation work up till now.
(In reply to Dan Johansson from comment #17) > I Have exactly the same Problem after updating from 1.21.0 to 1.23.0. > After updating does my custom init script not work, even after "pre-fixing" > all commands with "/bin/" it still does not work. > > I am not 100% sure but I think I can see a "/dev/tty not found" scroll past > in the very beginning of the init script (it scrolls past so fast that I am > not 100% sure (and "scrollback" does not work on this system)). This sounds like a different issue.
(In reply to bitlord from comment #15) > > I'm not a developer and as you can see ^^^ I don't know much about this, > especially busybox, but on the busybox webiste it says > "23 December 2014 -- BusyBox 1.23.0 (unstable)", I don't want to tell you > what to do, but it is strange that if upstream marked its release as > unstable, to be stable in gentoo? > This discussion should happen in the forums not in a bug report. I don't see any bug here. This version of buysbox is behaving as it is documented to behave. We needed 1.23 for its updates to mdev for openrc-0.13 which we needed for a whole other chain of dependancies. That plus the fact that I've been using 1.23 since the day it was pushed out and found no problems. It is unlikely that the behavior your seeing in 1.23 will go away with later releases. More likely its an oversight in the documentation that would have come up sooner or later. I haven't tried (and probably won't try) to reproduce the steps of that documentatoin to find out why it worked on 1.22. The correct approach is to run `busybox --install -s` at some point in the initramfs's root to create those sym links so mount and friends works.
After 13 years using Gentoo I can say I've never had much luck with the forums. Maybe I should try again someday. I just got it working, but your suggested command busybox --install -s didn't work for me. Doing it that way I got a message about busybox not being an absolute path. Changing it to /bin/busybox --install -s just before the mount commands did work. What I do find a bit strange is that one experiment I did managed to fall into my error block where your suggested command without the path is executed. In that case the shell commands like ls & mount worked fine. However until I added /bin in front as above nothing worked. It seems that maybe there's something a bit different about the way busybox is now handling paths. Anyway, whether something is broken now or whether something was luckily working before I don't have the skill set to figure out. Hopefully this will help others. I will attach my modified initramfs_init_new.sh file for any others that come along later. Cheers
Created attachment 394554 [details] Modified initramfs_init_new.sh file
It's because `allyesconfig` was changed to `defconfig` in the ebuilds. Changing it back to `allyesconfig` resolves the issue. related: https://bugs.gentoo.org/show_bug.cgi?id=526008
Good catch Andreas. So, why would the package maintainer, after all this time with busybox working correctly, make this change? There appears to be one new use flag (debug) and a requirement for a newer OpenRC.
Classic Gentoo, making each and every update a thrilling experience. :-| Why such a critical change is introduced (within a new ebuild release of the same application version!) without issuing a big fat warning during emerge (or even a portage news item) that this might BREAK your setup, is really beyond me. Please add a warning to the ebuild, a stable Gentoo package shouldn't break user systems without even giving a clue about the cause.
(In reply to Andreas Klauer from comment #23) > It's because `allyesconfig` was changed to `defconfig` in the ebuilds. > Changing it back to `allyesconfig` resolves the issue. > > related: https://bugs.gentoo.org/show_bug.cgi?id=526008 Thanks that explains it! I always do an savedconfig so I didn't hit that. So the problem isn't in busybox itself, its in the ebuild. Changing behaviors like that is annoying because it messes up expectations and documentation. (In reply to Stefan Kiesler from comment #25) > Classic Gentoo, making each and every update a thrilling experience. :-| > Why such a critical change is introduced (within a new ebuild release of the > same application version!) without issuing a big fat warning during emerge > (or even a portage news item) that this might BREAK your setup, is really > beyond me. > > Please add a warning to the ebuild, a stable Gentoo package shouldn't break > user systems without even giving a clue about the cause. I'm not the busybox mainatainer, but I did give the go ahead to stabilize 1.23.0. I know its marked unstable upstream etc, but I have been using it and was confident the codebase was okay and we needed mdev to move ahead with openrc. What I didn't anticipate (since I don't take care of the ebuild) is that the *ebuild* had changed from 1.21.1 to 1.23.0: diff -Naur busybox-1.21.0.ebuild busybox-1.23.0.ebuild # setup the config file - emake -j1 allyesconfig > /dev/null + emake -j1 -s defconfig >/dev/null # nommu forces a bunch of things off which we want on #387555 busybox_config_option n NOMMU sed -i '/^#/d' .config I personally like allyes since its safe. It guarantees busybox will have everything although you do pay the price of bloat.
(In reply to Anthony Basile from comment #26) > Changing behaviors like that is annoying because it messes up > expectations and documentation. It messed up some people's boot process. It would've messed up mine, if I hadn't been lazy about updating lately. I already fixed the "Custom Initramfs" article since whatever is decided here, should not affect users looking for help in the wiki. > I personally like allyes since its safe. It guarantees busybox will have > everything although you do pay the price of bloat. It's not safe, actually. But the ebuild already took care of that. It disabled the options known to cause trouble. So allyes was merely the base to build on, not used verbatim. A very good approach since it gets the most out of the software. It's like cherry picking. Not much work to maintain either; occasionally someone might stumble over a forgotten debug flag, easy to disable with none being the wiser. It's not like busybox ever gets swamped with hundreds of new flags, development is slow at around one release a year with few changes in the dangerous department. With defconfig a lot of finetuning code would have to be added, if one wanted to reach the previous level of completeness. Enabling everything that's useful is quite a lot more work than disabling dangerous things, simply because there are so many more useful options than dangerous ones. It won't be done.
(In reply to Andreas Klauer from comment #27) > (In reply to Anthony Basile from comment #26) > > Changing behaviors like that is annoying because it messes up > > expectations and documentation. > > It messed up some people's boot process. It would've messed up mine, if I > hadn't been lazy about updating lately. > > I already fixed the "Custom Initramfs" article since whatever is decided > here, should not affect users looking for help in the wiki. We have gone back to allyes. > > > I personally like allyes since its safe. It guarantees busybox will have > > everything although you do pay the price of bloat. > > It's not safe, actually. But the ebuild already took care of that. It > disabled the options known to cause trouble. So allyes was merely the base > to build on, not used verbatim. A very good approach since it gets the most > out of the software. It's like cherry picking. I meant in the context of the ebuild for the reason you state. When I said "everything" I meant applets, not things like WERROR which are undesireable in a production env. So I think we're in agreement here. > > Not much work to maintain either; occasionally someone might stumble over a > forgotten debug flag, easy to disable with none being the wiser. It's not > like busybox ever gets swamped with hundreds of new flags, development is > slow at around one release a year with few changes in the dangerous > department. > > With defconfig a lot of finetuning code would have to be added, if one > wanted to reach the previous level of completeness. Enabling everything > that's useful is quite a lot more work than disabling dangerous things, > simply because there are so many more useful options than dangerous ones. It > won't be done. defconfig is not appropriate. We don't know how users will use busybox so we want to make sure they get as much as possible without crossing the dagner line. Again, I think we're in agreement. It don't know why the maintainer addressed bug #525586 by switching to defconfig. I feel partially at fault for busybox-1.23 because I gave the green light to stabilize. I was only thinking of the busybox codebase which is pretty solid with 1.23.0 despite being "unstable" upstream. I didn't think the ebuild would have changed its default behavior this way.
Anthony - So can this be marked Resolved/Fixed?
(In reply to Mark Knecht from comment #29) > Anthony - So can this be marked Resolved/Fixed? Yep. Thanks for the report. This is in that grey area between configuration which is the users responsibity and a bug in the code which is ours. But busybox is something of an exception. Here we do control the configuration in the ebuild. However, you could configure busybox yourself and also fix this "bug", soo .... is it a bug? meh. I don't want to argue semantics. What happened here is we threw you a curve ball and we fixed that. So RESOLVED FIXED.
Thanks Anthony. I completely agree with your assessment and I appreciate the help.
