Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 392995 - Cannot boot system after recent openrc upgrade, udev can be started, "pam error: Module is unknown"
Summary: Cannot boot system after recent openrc upgrade, udev can be started, "pam err...
Status: RESOLVED NEEDINFO
Alias: None
Product: Gentoo Hosted Projects
Classification: Unclassified
Component: OpenRC (show other bugs)
Hardware: All Linux
: Normal blocker (vote)
Assignee: OpenRC Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-12-03 08:25 UTC by Paul Osmialowski
Modified: 2012-08-13 08:24 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
serial console bootup log (bootup.log,45.97 KB, text/plain)
2011-12-04 12:54 UTC, Paul Osmialowski
Details
/tmp/emerge-info.txt (emerge-info.txt,4.29 KB, text/plain)
2011-12-04 13:45 UTC, William Hubbs
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Osmialowski 2011-12-03 08:25:43 UTC
I'm upgrading gentoo on my machines quite frequently. One of them is turned off for the night. After yesterday openrc update it never woke up again.
start-stop-daemon complains that it can't start udevd, so nothing else works. As a reason, "pam error: Module is unknown" was thrown. After third reboot I was called for root password for mainenance, it bringed me root shell, I went to /etc/pam.d and checked all files to be there. Unfortunately, I don't know how udev pam and openrc are related, so my PC is dead until you give me some help. Now I'm afraid to reboot all my other machines :(

Reproducible: Always

Steps to Reproduce:
1. update gentoo
2. reboot
3.
Actual Results:  
udev can't start due to "pam error: Module is unknown" caused by start-stop-daemon; as a result whole system can't start

Expected Results:  
system fully booted
Comment 1 Paul Osmialowski 2011-12-03 08:39:36 UTC
I was so busy yesterday that on each my Gentoo machine I answered yes for each change prompted by etc-update (there were whole bunch of them - it is typical situation when openrc is upgraded). Therefore I don't know which files were touched, as I recall there was something about PAM among them...
Comment 2 Paul Osmialowski 2011-12-03 09:16:05 UTC
There must be a race condition in startup scripts and I've found a very nasty hack to win this race. First I must put it here that I forgot to mention that I entered single mode in grub and that's why I was able to access root shell (after being asked for a password).
When I was in, I remounted / with remount,rw options then I edited one file: /etc/conf.d/udev and set:

udev_debug="YES"

then I remounted / with remount,ro and rebooted machine.

This change will keep udev busy so other things will have chance to happend. This may not work on faster machines. My machine now is up and running.

Note also that afrer recent openrc upgrade all rc.init scripts are throwing warnings:

"Use of the opts variable is deprecated (...)"

Shouldn't this deprecation be covered by etc-udpate?!
Comment 3 Rafał Mużyło 2011-12-03 13:04:58 UTC
See bug 391945 for problems with rc_parallel, if that was what you've meant by "race condition".

As for deprecation warnings, it should not matter: either updated scripts will reach you once the relevant packages get into stable or they're already fixed in the tree, but the maintainer didn't see it a good enough reason for a revision bump.
Comment 4 William Hubbs gentoo-dev 2011-12-03 22:19:21 UTC
Hi Paul,

(In reply to comment #2)
> There must be a race condition in startup scripts and I've found a very nasty
> hack to win this race. First I must put it here that I forgot to mention that I
> entered single mode in grub and that's why I was able to access root shell
> (after being asked for a password).
> When I was in, I remounted / with remount,rw options then I edited one file:
> /etc/conf.d/udev and set:
> 
> udev_debug="YES"
> 
> then I remounted / with remount,ro and rebooted machine.
> 
> This change will keep udev busy so other things will have chance to happend.
> This may not work on faster machines. My machine now is up and running.

Try the following:

1) reverse the change you made to UDEV_DEBUG.
2) Remove RC_PARALLEL from /etc/rc.conf, so that it is not set.

3) reboot and let me know if there is still an issue.

> Note also that afrer recent openrc upgrade all rc.init scripts are throwing
> warnings:
> 
> "Use of the opts variable is deprecated (...)"
> 
> Shouldn't this deprecation be covered by etc-udpate?!

The init scripts will work fine with this warning. It is just a message that tells the maintainers they need to update their init scripts.

This warning will go away as soon as new versions of those packages are stabilized that have that fix.
Comment 5 Christian Ruppert (idl0r) gentoo-dev 2011-12-03 23:29:16 UTC
Please attach your emerge --info including your udev version.
Comment 6 Paul Osmialowski 2011-12-04 00:34:25 UTC
[ebuild   R    ] sys-fs/udev-164-r2  USE="extras -build (-selinux) -test" 0 kB

Portage 2.1.10.11 (default/linux/x86/10.0, gcc-4.5.3, glibc-2.12.2-r0, 2.6.39-gentoo-r3 i686)
=================================================================
System uname: Linux-2.6.39-gentoo-r3-i686-Intel-R-_Pentium-R-_4_CPU_3.00GHz-with-gentoo-2.0.3
Timestamp of tree: Fri, 02 Dec 2011 09:00:01 +0000
app-shells/bash:          4.1_p9
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.4.6, 2.6.6-r2, 2.7.2-r3, 3.1.4-r3
dev-util/cmake:           2.8.6-r4
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.4_p6-r1, 1.5, 1.6.3, 1.7.9-r2, 1.8.5-r3, 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.4.5, 4.5.3-r1
sys-devel/gcc-config:     1.4.1-r1
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)
sys-libs/glibc:           2.12.2
Repositories: gentoo armagetron
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="*"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=i686 -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /opt/openjms/config /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb /var/spool/torque"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"                                                                                                                                                                               
CXXFLAGS="-O2 -march=i686 -pipe"                                                                                                                                                        
DISTDIR="/usr/portage/distfiles"                                                                                                                                                        
FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"                                                                                                                                                                       
FFLAGS=""                                                                                                                                                                               
GENTOO_MIRRORS="http://distfiles.gentoo.org"                                                                                                                                            
LC_ALL="pl_PL.UTF-8"                                                                                                                                                                    
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="pl"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/layman/armagetron"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="16bit-indices 32bit 7zip R X Xaw3d a52 aac aalib abiword accessibility acl acpi adns adplug ads aesicm akode akonadi alias alisp allegro alsa amr amrnb amrwb angelscript animgif ao apbs apng applet archive arpack artworkextra asf asn aspell ass assistant asyncns athena attica attrib audacious audio audiofile auth automount autoreplace autotrace avalon-framework avalon-logkit avantgo background badval bash-completion bazaar bcmath berkdb bidi bittorrent bl blas blender-game blksha1 bluetooth boehm-gc bonjour bonobo bookmarks boost branding bs2b bzip2 cairo calendar caps capslib catalogs cblas cdda cddax cdf cdio cdparanoia cdr cdrtools cegui celt cg cgraph chardet chdir chicken chm clamd clamdtop cli client clipboard clisp clucene cluster clutter collab common-lisp community compat connectionstatus consolekit contactnotes context contrast contrib corefonts coverart cracklib cramfs crashreporter cron crypt cscope css ctype ctypes-python cuda cue cups curl cursors cvs cxx d darcs data dbase dbus dc1394 debugger declarative default-font deprecated designer designer-plugin desktopglobe detex devfs-compat devhelp device-mapper devil dht dia digitalradio dirac discard-path discouraged djbfft djvu dmi dnd docbook dot double-precision dri drmaa dselect dssi dtmf dts dv dvb dvd dvdnav dvdr dvi dvi2tty dvipdfm dynamic eap-tls ebook edit editor eds egl eigen elf elisp emacs embedded emf emoticons-manager emovix enca enchant encode enscript eolconv epiphany escreen esd eselect ethernet excel exceptions exif expat expoblending extensions extra extra-cardsets extra-tools extrafilters extramodules extras faac faad facedetect fam fame fat fax festival ffamber ffmpeg fftw fileinfo filter findbugs finger firefox firefox3 fits flac floppy fltk fluidsynth fontconfig fontforge foomaticdb fortran fox fping fpx frei0r ftp fts3 fuse gadu gallium games gcdmaster gconf gcrypt gd gdal gdbm gdm gdu gecko gedit geoip geolocation geos ggi gif gimp ginac git gjdoc glade glibc-omitfp glitz gloox glut gmath gme gml gmp gmplayer gnet gnokii gnome gnome-keyring gnomecd gnuplot gnutls gopher gpg gphoto2 gpm gpu grammar graphics graphite graphviz groupwise grp grub gs gsl gsm gssapi gstreamer gtk gtk3 gtkhtml gtkstyle guile gunit gzip-el h224 h281 h323 hal handbook hardcoded-tables hash hdf hdf5 health heterogeneous highlight hipe hires-icons history hlapi hog hotpixels hpcups hpijs html http humanities hunspell iax ical icecast icons iconv icu id3 id3tag idn ieee1394 ifp image image-cache imagemagick imlib inherit-graph inifile inkjar inotify intl iodbc ipc iplayer ipod ipv6 irc irrlicht isag it3tag itk ivr ixj jabber jack jadetex jai java java5 java6 javacomm javamail javascript jbig jit jmf jms joystick jpeg jpeg2k json kate kde kdm kdrive kig-scripting kipi kontact kpathsea ladspa lame lapack largefile lash lasi lastfm lastfmradio latex latex3 lcdfilter lcms ldap ldap-sasl ldb leim less levels libass libburn libcaca libcanberra libclamav libdsk libextractor libffi libgda libmms libmpeg2 libnotify libs libsamplerate libssh2 libsysfs libtiger libv4l libv4l2 libvisual libwww libyaml lid lightning linuxkeys listen-queue live llvm lm_sensors log4j logitech-mouse logrotate lpsol lqr lua luatex lvm1 lxde lyx lzma lzo m17n-lib mad mail maps math mathml matroska mbox mcve md5sum mdns-bundled memmap mercurial metis metric mew mhash midi mikmod mime mixer mjpeg mmap mms mmx mng modplug modules mono moonlight motif moviepack moviesounds mozdevelop mozdom mozilla mozsha1 mp2 mp3 mp3tunes mp4 mpeg mpg123 mpi mplayer mpx msn mta mtp mudflap mule multimedia musepack music musicbrainz mvl mzscheme nautilus ncurses neXt nelma net netapi netbeans netcdf netlink netpbm network networking new-clx nextaw nfsv3 nfsv4 nis njb nls nntp no-old-linux nodrm nogyroscopic nokia normalize nowlistening npp nptl nptlonly nsplugin nss ntfs ntlm ntlm_unsupported_patch ntp numpy nut nuv objc objc++ objc-gc ocaml ocamlopt ocr octave odbc ode offensive ofx ogdi ogg ogg123 ogm okular omega opal openal opencl opencore-amr openexr opengl openldap openmp openxml orc oscar osdmenu osgapps oss otp otr ots overlays pager pam pango pari parport password pasteafter pbs pcap pcntl pcre pcsc-lite pda pdf pdfannotextractor pdo perl perl-geoipupdate pg-intdatetime phar phonon php physfs pike pipe pixmaps pkinit plasma player playlist plotutils plugins pnat png pnm policykit poppler-data portaudio posix postgres postproc postscript povray powerkadu ppds ppp pppd preview-latex private-headers profiler proj projectm pstricks pth publishers pvr python qhull qimageblitz qos qpak qscintilla qt-bundled qt-faststart qt-static qt3 qt3support qt4 qualculate quicktime quotes qwt radio raptor rar raw rcs readline recursion-limit redeyes redland reflection regex remote remoteosd resid rle romio rpm rrdtool rss rtc rtf rtmp rtsp ruby rups samba sametime sasl sbc scale0tilt scanner schroedinger science script sctp sdk sdl sdl-image sdl-sound sdlgfx semantic-desktop sensord serial servletapi session sge shaders shared-glapi sharedext sharedmem shine shm shmvideo shout sid sift sigsegv simplexml sip sipim skins skype slang smbclient smbkrb5passwd smbtav2 smi smp sms sndfile snmp soap sockets solver sound soundex sounds soundtouch soup sourceview sox sparse speech speex spell spl splash sql sqlite sqlite3 srtp sse sse2 ssl startup-notification statistics stats stk stream stun subject-rewrite subtitles subversion suhosin supernodal svg svm swat swig sysfs syslog sysvipc szip t1lib taglib tcl tcpd tdb tdbtest telepathy teletext telnet test-framework tex4ht texteffect tga themedesigner themes theora thesaurus thin-splines threads threadsafe thumbnail thumbnails tidy tiff tilepath timezone timidity tk tokenizer toolame toolbar toolkit-scroll-bars tools totem track-src-odirect traits translator tremor truetype tta twisted twolame type3 udev udev-acl udis86 umfpack unicode unlock-notify usb utempter utils uudeview uuid v4l v4l2 vaapi vala valgrind vamp vapigen vcd vcdinfo vcdx vdpau vdr verse vhook video vidix vim-pager vim-with-x virtuoso visualization vlm vmd vnc voice vorbis vpx vst vxml wad watchdog wav wavpack wddx weather webdav webdav-neon webdav-serf webkit webm webp widescreen win32codecs winbind wininst winpopup wma wma-fixed wmf wordperfect wxwidgets wxwindows x264 x86 xanim xbase xcap xcb xchatdccserver xcomposite xemacs xembed xerces-c xetex xext xface xforms xforward xfs xft xgetdefault xindy xine xinerama xinetd xml xmldoclet xmlpatterns xmlreader xmlrpc xmlwriter xmp xmpp xnest xorg xosd xpdf-headers xpm xrandr xrender xscreensaver xsl xslt xterm xulrunner xv xvfb xvid xvmc yahoo yaz youtube yp yv12 zip zlib zvbi" ALSA_CARDS="emu10k1 emu10k1x" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="canon spca50x template ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev keyboard mouse joystick" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="pl" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="dummy nv nvidia mach64 r128 v4l intel vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 7 Paul Osmialowski 2011-12-04 00:37:57 UTC
@William

Following steps:

1) reverse the change you made to UDEV_DEBUG.
2) Remove RC_PARALLEL from /etc/rc.conf, so that it is not set.

3) reboot and let me know if there is still an issue.

brrought back the problem again. Seems like it is not the RC_PARALLEL issue.

Actually, RC_PARALLEL was not even set (rc_parallel="NO" was remarked in my rc.conf, so I removed the remarking hash and rebooted with rc_parallel set explicitely to "NO" - this did not help either).
Comment 8 Christian Ruppert (idl0r) gentoo-dev 2011-12-04 00:59:01 UTC
Do you have a serial console or something equivalent to get a boot log?
What pam version do you use?

Also try:
# for foo in $(qfile -C /etc/pam.d|awk '{ print $1 }'); do qcheck $foo; done
Comment 9 Napoleon 2011-12-04 04:50:26 UTC
I have had a hell of a week after the update to openrc-0.9.4. I'm running a production server with a hardened-amd64-no-multilib profile and after updating files with etc-update and rebooting, the system began to hang at "Starting udevd...". I could press Ctrl+C and make the boot sequence resume but then I would get other errors like Myslq failing to start (even though it starts).

In a thread, on the Gentoo forums, someone mentionned that the update to the /etc/pam.d/start-top-daemon file changed from this:

auth            required        pam_permit.so
account         required        pam_permit.so
password        required        pam_deny.so
session         optional        pam_limits.so

to this:

account required pam_permit.so
session include system-services

After modifying the file back to the older version, my server booted correctly.
Comment 10 Paul Osmialowski 2011-12-04 11:12:33 UTC
[ebuild   R    ] sys-libs/pam-1.1.5  USE="berkdb cracklib nis nls -audit -debug (-selinux) -test -vim-syntax" 0 kB

As for serial console, I need to prepare null-modem cable myself, so it will take some time.
Comment 11 Paul Osmialowski 2011-12-04 11:16:13 UTC
# for foo in $(qfile -C /etc/pam.d|awk '{ print $1 }'); do qcheck $foo; done
Checking sys-apps/openrc-0.9.4 ...
 AFK: /lib/rc/init.d/.keep_sys-apps_openrc-0
 MD5-DIGEST: /etc/conf.d/hwclock
 MD5-DIGEST: /etc/conf.d/keymaps
 MD5-DIGEST: /etc/rc.conf
  * 292 out of 296 files are good
Checking sys-apps/shadow-4.1.4.3 ...
  * 480 out of 480 files are good
Checking sys-process/vixie-cron-4.1-r11 ...
  * 36 out of 36 files are good
Checking sys-auth/pambase-20101024 ...
  * 8 out of 8 files are good
Checking sys-auth/polkit-qt-0.99.0 ...
 MTIME: /usr/lib/pkgconfig/polkit-qt-1.pc
 MTIME: /usr/lib/pkgconfig/polkit-qt-core-1.pc
 MTIME: /usr/lib/pkgconfig/polkit-qt-gui-1.pc
 MTIME: /usr/lib/pkgconfig/polkit-qt-agent-1.pc
  * 52 out of 56 files are good
Checking sys-auth/polkit-kde-agent-0.99.0 ...
  * 11 out of 11 files are good
Checking sys-auth/polkit-0.102 ...
 MTIME: /usr/lib/pkgconfig/polkit-gobject-1.pc
 MTIME: /usr/lib/pkgconfig/polkit-backend-1.pc
 MTIME: /usr/lib/pkgconfig/polkit-agent-1.pc
 AFK: /var/run/polkit-1/.keep_sys-auth_polkit-0
  * 111 out of 115 files are good
Checking app-admin/sudo-1.8.1_p2 ...
  * 54 out of 54 files are good
Checking app-misc/screen-4.0.3-r4 ...
 AFK: /var/run/screen/.keep_app-misc_screen-0
  * 58 out of 59 files are good
Checking net-misc/netkit-rsh-0.17-r9 ...
  * 39 out of 39 files are good
Checking net-misc/openssh-5.8_p1-r1 ...
 MD5-DIGEST: /etc/ssh/sshd_config
 MTIME: /etc/conf.d/sshd
  * 65 out of 67 files are good
Checking dev-libs/cyrus-sasl-2.1.23-r4 ...
 MTIME: /usr/lib/sasl2/libsasldb.la
 MTIME: /usr/lib/sasl2/libcrammd5.la                                                                                                                                                    
 MTIME: /usr/lib/sasl2/libdigestmd5.la                                                                                                                                                  
 MTIME: /usr/lib/sasl2/libplain.la                                                                                                                                                      
 MTIME: /usr/lib/sasl2/libanonymous.la                                                                                                                                                  
 MTIME: /usr/lib/sasl2/liblogin.la                                                                                                                                                      
 MTIME: /usr/lib/sasl2/libntlm.la                                                                                                                                                       
 MTIME: /usr/lib/sasl2/libldapdb.la                                                                                                                                                     
 MTIME: /usr/lib/libsasl2.la                                                                                                                                                            
 MTIME: /usr/lib/libjavasasl.la                                                                                                                                                         
 MTIME: /etc/conf.d/saslauthd
  * 181 out of 192 files are good
Checking net-mail/mailbase-1 ...
  * 18 out of 18 files are good
Checking x11-misc/xlockmore-5.31 ...
  * 29 out of 29 files are good
Checking x11-misc/xscreensaver-5.14 ...
  * 707 out of 707 files are good
Checking x11-apps/xdm-1.1.10-r2 ...
  * 46 out of 46 files are good
Checking dev-db/postgresql-server-9.1.1 ...
 MD5-DIGEST: /etc/conf.d/postgresql-9.1
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-9.1
  * 326 out of 328 files are good
Checking dev-db/postgresql-server-9.0.5 ...
 MD5-DIGEST: /etc/conf.d/postgresql-9.0
 MTIME: /etc/pam.d/postgresql
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-9.0
  * 262 out of 265 files are good
Checking dev-db/postgresql-server-8.4.9 ...
 MD5-DIGEST: /etc/conf.d/postgresql-8.4
 MTIME: /etc/pam.d/postgresql
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-8.4
  * 251 out of 254 files are good
Checking dev-db/postgresql-server-9.1.1 ...
 MD5-DIGEST: /etc/conf.d/postgresql-9.1
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-9.1
  * 326 out of 328 files are good
Checking dev-db/postgresql-server-9.0.5 ...
 MD5-DIGEST: /etc/conf.d/postgresql-9.0
 MTIME: /etc/pam.d/postgresql
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-9.0
  * 262 out of 265 files are good
Checking dev-db/postgresql-server-8.4.9 ...
 MD5-DIGEST: /etc/conf.d/postgresql-8.4
 MTIME: /etc/pam.d/postgresql
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-8.4
  * 251 out of 254 files are good
Checking dev-db/postgresql-server-9.1.1 ...
 MD5-DIGEST: /etc/conf.d/postgresql-9.1
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-9.1
  * 326 out of 328 files are good
Checking dev-db/postgresql-server-9.0.5 ...
 MD5-DIGEST: /etc/conf.d/postgresql-9.0
 MTIME: /etc/pam.d/postgresql
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-9.0
  * 262 out of 265 files are good
Checking dev-db/postgresql-server-8.4.9 ...
 MD5-DIGEST: /etc/conf.d/postgresql-8.4
 MTIME: /etc/pam.d/postgresql
 AFK: /var/run/postgresql/.keep_dev-db_postgresql-server-8.4
  * 251 out of 254 files are good
Checking gnome-base/gdm-2.20.11-r1 ...
 MD5-DIGEST: /etc/X11/gdm/custom.conf
  * 206 out of 207 files are good
Checking gnome-extra/gnome-screensaver-2.30.2-r1 ...
 MTIME: /usr/lib/pkgconfig/gnome-screensaver.pc
  * 80 out of 81 files are good
Checking net-print/cups-1.4.8-r1 ...
 MTIME: /usr/lib/perl5/vendor_perl/5.12.4/i686-linux/CUPS.pm
 AFK: /var/run/cups/certs/.keep_net-print_cups-0
 AFK: /var/spool/cups/tmp/.keep_net-print_cups-0
  * 579 out of 582 files are good
Checking net-fs/samba-3.5.11 ...
  * 507 out of 507 files are good
Checking kde-base/kdebase-pam-7 ...
  * 4 out of 4 files are good
Checking net-dialup/ppp-2.4.5-r1 ...
 MTIME: /etc/ppp/options
  * 129 out of 130 files are good
Checking dev-vcs/cvs-1.12.12-r6 ...
  * 72 out of 72 files are good
Checking dev-vcs/cvsgraph-1.6.1 ...
  * 25 out of 25 files are good
Checking dev-vcs/cvsps-2.1 ...
  * 11 out of 11 files are good
Comment 12 Paul Osmialowski 2011-12-04 12:54:47 UTC
Created attachment 294709 [details]
serial console bootup log

Finally I managed to prepare null-modem cable (fortunately, I had all required parts @ home!)
Comment 13 Rafał Mużyło 2011-12-04 13:18:52 UTC
(In reply to comment #12)
> Created attachment 294709 [details]
> serial console bootup log
> 
> Finally I managed to prepare null-modem cable (fortunately, I had all required
> parts @ home!)

Well, first of all, pam error is present in this log.
Second '/dev/hda9' is just asking for trouble.
Comment 14 William Hubbs gentoo-dev 2011-12-04 13:44:04 UTC
I just downgraded my box to match this configuration and booted
successfully. I will include my /etc/pam.d/start-stop-daemon and emerge
--info for comparison:

# /etc/pam.d/start-stop-daemon
account required pam_permit.so
session include system-services
Comment 15 William Hubbs gentoo-dev 2011-12-04 13:45:30 UTC
Created attachment 294711 [details]
/tmp/emerge-info.txt

My emerge --info
Comment 16 Paul Osmialowski 2011-12-04 15:59:40 UTC
Seems like I got the same:

# cat /etc/pam.d/start-stop-daemon 
account required pam_permit.so
session include system-services
Comment 17 Paul Osmialowski 2011-12-04 16:11:11 UTC
Now I have followed Napoleonb advices: I have reverted udev_debug="NO" and changed this:

account required pam_permit.so
session include system-services

to this:

auth            required        pam_permit.so
account         required        pam_permit.so
password        required        pam_deny.so
session         optional        pam_limits.so

in /etc/pam.d/start-top-daemon file. As a result I'm having fully booted system.
Something must be wrong with new pam settings.
Comment 18 Rafał Mużyło 2011-12-04 18:00:26 UTC
(In reply to comment #13)
> 
> Second '/dev/hda9' is just asking for trouble.

If you don't know what my comment is about, just search the forum for "libata migration" topics.
Comment 19 William Hubbs gentoo-dev 2011-12-05 04:11:01 UTC
Hi Paul,

(In reply to comment #17)
> Now I have followed Napoleonb advices: I have reverted udev_debug="NO" and
> changed this:
> 
> account required pam_permit.so
> session include system-services
> 
> to this:
> 
> auth            required        pam_permit.so
> account         required        pam_permit.so
> password        required        pam_deny.so
> session         optional        pam_limits.so
> 
> in /etc/pam.d/start-top-daemon file. As a result I'm having fully booted
> system.
> Something must be wrong with new pam settings.

No, afaik the pam settings are correct. If they were wrong, no daemons at all would start.

If you are referring to your disk drives as /dev/hd*, I recommend that you follow the above advice about libata migration.
Comment 20 Paul Osmialowski 2011-12-05 08:12:01 UTC
Even if the right solution is the libata migration, it is still unclear for me what these start-stop-daemon pam settings suggested by Napoleon B. (that actually solved problem for me) have in common with device naming scheme!
Comment 21 William Hubbs gentoo-dev 2011-12-05 15:13:28 UTC
(In reply to comment #20)
> Even if the right solution is the libata migration, it is still unclear for me
> what these start-stop-daemon pam settings suggested by Napoleon B. (that
> actually solved problem for me) have in common with device naming scheme!

They don't, and if they did, many others would have the same issue.

I found another bug that gave me an idea.

Can you please attach your rc.conf?
Comment 22 Napoleon 2011-12-05 16:21:14 UTC
(In reply to comment #21)
> (In reply to comment #20)
> > Even if the right solution is the libata migration, it is still unclear for me
> > what these start-stop-daemon pam settings suggested by Napoleon B. (that
> > actually solved problem for me) have in common with device naming scheme!
> 
> They don't, and if they did, many others would have the same issue.
> 
> I found another bug that gave me an idea.
> 
> Can you please attach your rc.conf?

I had all sorts of problems after updating to openrc-0.9.4. Boot sequence would stop at "Starting udevd..." and I could only make it continue by pressing "Ctrl+C". Then MySQL would give me an error on starting even though it would actually start and Samba would not accept connections from Windows clients.

I really didn't get into how pam works but using the 4 lines fixed absolutely all the problems I had (Samba is now accepting back connections from Windows clients).

And, for the record, we're not alone:

http://forums.gentoo.org/viewtopic-t-892872-start-0.html

Problem seem to have started when openrc-0.9.X hit the ~amd64. But then, the usual condescending attitude using ~arch began to flow. I'm using stable and I paid for this.
Comment 23 Christian Ruppert (idl0r) gentoo-dev 2011-12-05 18:42:55 UTC
Diego, any ideas if it might be really a PAM issue or not?
I doubt its PAM but I just want your opinion on it.
Comment 24 Paul Osmialowski 2011-12-05 19:15:45 UTC
my rc.conf:

# Global OpenRC configuration settings

# Set to "YES" if you want the rc system to try and start services
# in parallel for a slight speed improvement. When running in parallel we
# prefix the service output with its name as the output will get
# jumbled up.
# WARNING: whilst we have improved parallel, it can still potentially lock
# the boot process. Don't file bugs about this unless you can supply
# patches that fix it without breaking other things!
rc_parallel="NO"

# Set rc_interactive to "YES" and you'll be able to press the I key during
# boot so you can choose to start specific services. Set to "NO" to disable
# this feature. This feature is automatically disabled if rc_parallel is
# set to YES.
#rc_interactive="YES"

# If we need to drop to a shell, you can specify it here.
# If not specified we use $SHELL, otherwise the one specified in /etc/passwd,
# otherwise /bin/sh
# Linux users could specify /sbin/sulogin
rc_shell=/sbin/sulogin

# Do we allow any started service in the runlevel to satisfy the dependency
# or do we want all of them regardless of state? For example, if net.eth0
# and net.eth1 are in the default runlevel then with rc_depend_strict="NO"
# both will be started, but services that depend on 'net' will work if either
# one comes up. With rc_depend_strict="YES" we would require them both to
# come up.
#rc_depend_strict="YES"

# rc_hotplug is a list of services that we allow to be hotplugged.
# By default we do not allow hotplugging.
# A hotplugged service is one started by a dynamic dev manager when a matching
# hardware device is found.
# This service is intrinsically included in the boot runlevel.
# To disable services, prefix with a !                                                                                                                                                  
# Example - rc_hotplug="net.wlan !net.*"                                                                                                                                                
# This allows net.wlan and any service not matching net.* to be plugged.                                                                                                                
# Example - rc_hotplug="*"                                                                                                                                                              
# This allows all services to be hotplugged                                                                                                                                             
#rc_hotplug="*"                                                                                                                                                                         
                                                                                                                                                                                        
# rc_logger launches a logging daemon to log the entire rc process to                                                                                                                   
# /var/log/rc.log                                                                                                                                                                       
# NOTE: Linux systems require the devfs service to be started before
# logging can take place and as such cannot log the sysinit runlevel.
#rc_logger="YES"

# Through rc_log_path you can specify a custom log file.
# The default value is: /var/log/rc.log
#rc_log_path="/var/log/rc.log"

# By default we filter the environment for our running scripts. To allow other
# variables through, add them here. Use a * to allow all variables through.
#rc_env_allow="VAR1 VAR2"

# By default we assume that all daemons will start correctly.
# However, some do not - a classic example is that they fork and return 0 AND
# then child barfs on a configuration error. Or the daemon has a bug and the
# child crashes. You can set the number of milliseconds start-stop-daemon
# waits to check that the daemon is still running after starting here.
# The default is 0 - no checking.
#rc_start_wait=100

# rc_nostop is a list of services which will not stop when changing runlevels.
# This still allows the service itself to be stopped when called directly.
#rc_nostop=""

# rc will attempt to start crashed services by default.
# However, it will not stop them by default as that could bring down other
# critical services.
#rc_crashed_stop=NO
#rc_crashed_start=YES

##############################################################################
# MISC CONFIGURATION VARIABLES
# There variables are shared between many init scripts

# Set unicode to YES to turn on unicode support for keyboards and screens.
unicode="YES"

# Below is the default list of network fstypes.
#
# afs cifs coda davfs fuse fuse.sshfs gfs glusterfs lustre ncpfs
# nfs nfs4 ocfs2 shfs smbfs
#
# If you would like to add to this list, you can do so by adding your
# own fstypes to the following variable.
#extra_net_fs_list=""

##############################################################################
# SERVICE CONFIGURATION VARIABLES
# These variables are documented here, but should be configured in
# /etc/conf.d/foo for service foo and NOT enabled here unless you
# really want them to work on a global basis.

# Some daemons are started and stopped via start-stop-daemon.
# We can set some things on a per service basis, like the nicelevel.
#export SSD_NICELEVEL="-19"

# Pass ulimit parameters
#rc_ulimit="-u 30"

# It's possible to define extra dependencies for services like so
#rc_config="/etc/foo"
#rc_need="openvpn"
#rc_use="net.eth0"
#rc_after="clock"
#rc_before="local"
#rc_provide="!net"

# You can also enable the above commands here for each service. Below is an
# example for service foo.
#rc_foo_config="/etc/foo"
#rc_foo_need="openvpn"
#rc_foo_after="clock"

# You can also remove dependencies.
# This is mainly used for saying which servies do NOT provide net.
#rc_net_tap0_provide="!net"

##############################################################################
# LINUX SPECIFIC OPTIONS

# This is the subsystem type. Valid options on Linux:
# ""        - nothing special
# "lxc"     - Linux Containers
# "openvz"  - Linux OpenVZ
# "prefix"  - Prefix
# "uml"     - Usermode Linux
# "vserver" - Linux vserver
# "xen0"    - Xen0 Domain
# "xenU"    - XenU Domain
# If this is commented out, automatic detection will be attempted.
# Note that autodetection will not work in a prefix environment or in a
# linux container.
#
# This should be set to the value representing the environment this file is
# PRESENTLY in, not the virtualization the environment is capable of.
#rc_sys=""

# This is the number of tty's used in most of the rc-scripts (like
# consolefont, numlock, etc ...)
rc_tty_number=12
Comment 25 William Hubbs gentoo-dev 2011-12-05 21:39:54 UTC
If this is a pam issue, it is related to bug #365149. The fix would be
to possibly revert the change made for that bug, but, I'm not convinced
either that this is a pam issue.
Comment 26 Diego Elio Pettenò (RETIRED) gentoo-dev 2011-12-13 20:45:03 UTC
Has anyone here hand-edited the system-auth service file? If so can you please post it here? Thanks.
Comment 27 Matthew Stapleton 2011-12-14 05:03:48 UTC
On an older system that I upgraded to openrc-0.9.4 I got this problem due to having sys-libs/pam-0.78-r3 which doesn't support /etc/pam.d/system-services .  openrc-0.9.4 should depend on a newer version of pam that does support the system-services file.
Comment 28 Diego Elio Pettenò (RETIRED) gentoo-dev 2011-12-14 13:35:54 UTC
William please add a dependency over pambase (which is the provider of said file) rather than one on PAM.

And I seriously won't care much about anyone running pam 0.78, that's absurdly old and buggy :/
Comment 29 William Hubbs gentoo-dev 2011-12-14 17:36:08 UTC
(In reply to comment #28)
> William please add a dependency over pambase (which is the provider of said
> file) rather than one on PAM.

All, this dependency has been adjusted as requested.

@Paul, @Napoleon:

After about 18:00 UTC today, Please resync your portage trees, go back to the default pam and udev settings, make sure that your systems are up to date and report back to this bug whether or not that fixed the issue.
Comment 30 William Hubbs gentoo-dev 2011-12-16 15:41:51 UTC
I am moving this bug to test request status per my previous comment.
Please re-open if you still have the issue.
Comment 31 Napoleon 2011-12-19 22:40:35 UTC
(In reply to comment #29)
> (In reply to comment #28)
> > William please add a dependency over pambase (which is the provider of said
> > file) rather than one on PAM.
> 
> All, this dependency has been adjusted as requested.
> 
> @Paul, @Napoleon:
> 
> After about 18:00 UTC today, Please resync your portage trees, go back to the
> default pam and udev settings, make sure that your systems are up to date and
> report back to this bug whether or not that fixed the issue.

Changing /etc/pam.d/start-stop-daemon to the 2 liners, still makes my system hang at "Starting udevd ..." (I have to press Ctrl+C to continue), I still get "MySQL NOT started (0)" message (even though mysql daemon is indeed started) and Samba doesn't accept connections from my Windows clients.

My system is fully up to date as I write theese lines.

I have pam-1.1.5, udev-164-r2, pambase-20101024 and openrc-0.9.4 installed if that matters.
Comment 32 Diego Elio Pettenò (RETIRED) gentoo-dev 2011-12-20 01:31:07 UTC
Which two liners? I really _really_ want to see your stacks, guys.
Comment 33 Paul Osmialowski 2011-12-20 08:08:10 UTC
I would be more than happy to see version bump of any of these:

[ebuild   R    ] sys-libs/pam-1.1.5  USE="berkdb cracklib nis nls -audit -debug (-selinux) -test -vim-syntax" 0 kB
[ebuild   R    ] sys-apps/openrc-0.9.4  USE="ncurses pam unicode -debug (-selinux)" 0 kB
[ebuild   R    ] sys-fs/udev-164-r2  USE="extras -build (-selinux) -test" 0 kB

I'm syncing twice a week regularry (last week even more to help with this bug) but it is still in the same version.
1. pam-1.1.5 updated 1st of November
2. openrc-0.9.4 updated 2nd of December
3. udev-164-r2 updated 23rd of June

So what change should I observe if nothing has changed?

Anyway, two liner contains just two lines:

# cat /etc/pam.d/start-stop-daemon 
account required pam_permit.so
session include system-services
Comment 34 Diego Elio Pettenò (RETIRED) gentoo-dev 2011-12-20 10:35:46 UTC
system-auth please
Comment 35 Napoleon 2011-12-20 14:58:58 UTC
(In reply to comment #34)
> system-auth please

Here's mine:

auth		required	pam_env.so 
auth		sufficient	pam_ssh.so
auth		[success=1 default=ignore]	pam_krb5.so  ignore_root try_first_pass
auth		required	pam_unix.so try_first_pass likeauth nullok 
auth		optional	pam_permit.so
 
account		[success=1 default=ignore]	pam_krb5.so  ignore_root try_first_pass
account		required	pam_unix.so 
account		optional	pam_permit.so
 
password	required	pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3 
password	[success=1 default=ignore]	pam_krb5.so  ignore_root try_first_pass
password	required	pam_unix.so try_first_pass use_authtok nullok sha512 shadow 
password	optional	pam_permit.so
 
session		optional	pam_ssh.so
session		required	pam_limits.so 
session		required	pam_env.so 
session		optional	pam_mktemp.so
session		[success=1 default=ignore]	pam_krb5.so  ignore_root try_first_pass
session		required	pam_unix.so 
session		optional	pam_permit.so
Comment 36 Paul Osmialowski 2011-12-20 15:11:21 UTC
Mine is:

auth            required        pam_env.so
auth            required        pam_unix.so try_first_pass likeauth nullok
auth            optional        pam_permit.so

account         required        pam_unix.so
account         optional        pam_permit.so

password        required        pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password        required        pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password        optional        pam_permit.so

session         required        pam_limits.so
session         required        pam_env.so
session         required        pam_unix.so
session         optional        pam_permit.so
Comment 37 William Hubbs gentoo-dev 2011-12-29 21:15:29 UTC
@flameeyes:

Do you have any suggestions for this?
Comment 38 Volker R. Müller 2012-01-15 18:55:43 UTC
(In reply to comment #9)
> After modifying the file back to the older version, my server booted correctly.

Same here (stable amd64).

My /etc/pam.d/systemm-auth:

auth		required	pam_env.so 
auth		required	pam_unix.so try_first_pass likeauth nullok 
auth		optional	pam_permit.so
 
account		required	pam_unix.so 
account		optional	pam_permit.so
 
password	required	pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3 
password	required	pam_unix.so try_first_pass use_authtok nullok sha512 shadow 
password	optional	pam_permit.so

session		required	pam_limits.so 
session		required	pam_env.so 
session		required	pam_unix.so 
session		optional	pam_permit.so
Comment 39 William Hubbs gentoo-dev 2012-02-24 17:59:59 UTC
Napoleon,

OpenRC 0.9.9.1 includes a pam fix which I hope will fix your issue.
Please upgrade and let me know if you still can't boot using the
provided /etc/pam.d/start-stop-daemon file.

Thanks,

William
Comment 40 William Hubbs gentoo-dev 2012-03-21 14:43:45 UTC
I am closing this due to no response from the reporter.
Comment 41 Quasimodo 2012-08-13 08:24:12 UTC
I have the same problem with openrc-0.9.8.4. Changing the /etc/pam.d/start-stop-daemon file also solved it for me, but - and there is the rather strange thing - I've got two machines with a similar setup, but only one is experiencing the problem.

They are both Setup in a similar way, which means separate lvm partition for /usr, /var, /opt..., root on sda2, same software level on all packages in @system. One severe difference is that one is a desktop with default/linux/amd64/10.0/desktop profile and the other one is a server with default/linux/amd64/10.0 profile. And the problemativ machine is the server. So maybe, there is something missing in the servers profile, which is causing the error?