Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 856691

Summary: net-im/ejabberd-22.05-r1: repeated crashes caused by acct-user/ejabberd not updating ejabberd user's home
Product: Gentoo Linux Reporter: Phil Stracchino (Unix Ronin) <phils>
Component: Current packagesAssignee: ejabberd Project <ejabberd>
Status: CONFIRMED ---    
Severity: normal CC: phils
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Phil Stracchino (Unix Ronin) 2022-07-06 13:27:45 UTC
net-im/ejabberd-22.05-r1 fails to start, and any attempt to do anything such as check status results in a crash dump.


minbar:root:~:20 # pgrep -alf ejabberd
minbar:root:~:21 # ejabberdctl start
minbar:root:~:22 # ejabberdctl status
=ERROR REPORT==== 6-Jul-2022::09:25:11.760725 ===
ERROR: {error,"Error when reading /dev/null/.erlang.cookie: enotdir"} - []
=SUPERVISOR REPORT==== 6-Jul-2022::09:25:11.760790 ===
    supervisor: {local,net_sup}
    errorContext: start_error
    reason: {{error,"Error when reading /dev/null/.erlang.cookie: enotdir"},
             [{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},
              {auth,init,1,[{file,"auth.erl"},{line,165}]},
              {gen_server,init_it,2,[{file,"gen_server.erl"},{line,848}]},
              {gen_server,init_it,6,[{file,"gen_server.erl"},{line,811}]},
              {proc_lib,init_p_do_apply,3,
                        [{file,"proc_lib.erl"},{line,240}]}]}
    offender: [{pid,undefined},
               {id,auth},
               {mfargs,{auth,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,2000},
               {child_type,worker}]

{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{{error,\"Error when reading /dev/null/.erlang.cookie: enotdir\"},[{auth,init_no_setcookie,0,[{file,\"auth.erl\"},{line,313}]},{auth,init,1,[{file,\"auth.erl\"},{line,165}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,848}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{{error,"Error when reading /dev/null/.erlang.cookie: enotdir"},[{auth,init_no_setcookie,0,[{file,"auth.erl"},{line,313}]},{auth,init,1,[{file,"auth.erl"},{line,165}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,848}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}}}},{kernel,start,[normal,[]]}}})

Crash dump is being written to: /var/log/ejabberd/erl_crash_20220706-092511.dump...done


Reproducible: Always




minbar:root:~:23 # emerge -pqv net-im/ejabberd
[ebuild   R   ] net-im/ejabberd-22.05-r1  USE="captcha mysql pam stun zlib -debug -full-xml -ldap -mssql -odbc -postgres -redis -roster-gw (-selinux) -sip -sqlite -verify-sig"


minbar:root:~:24 # emerge --info
Portage 3.0.30 (python 3.10.5-final-0, default/linux/amd64/17.1, gcc-11.3.0, glibc-2.34-r13, 5.17.9-gentoo-minbar x86_64)
=================================================================
System uname: Linux-5.17.9-gentoo-minbar-x86_64-Intel-R-_Xeon-R-_CPU_E5620_@_2.40GHz-with-glibc2.34
KiB Mem:    24612904 total,  12233004 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Wed, 06 Jul 2022 06:00:01 +0000
Head commit of repository gentoo: 7657f9b3340d2594d5084bfca555dd79da7f8d57
Timestamp of repository guru: Tue, 05 Jul 2022 07:16:31 +0000
Head commit of repository guru: f5a209f57308ebddacf43e98c7a28b60c0e70f04

Head commit of repository mysql: 3925d2fe5eef1e63602a4f520028aa55dca3df08

Timestamp of repository slonko: Tue, 05 Jul 2022 05:01:33 +0000
Head commit of repository slonko: 0b6e59057cdc6b269bbee2d11aec353875c558a4

sh bash 5.1_p16
ld GNU ld (Gentoo 2.37_p1 p2) 2.37
app-misc/pax-utils:        1.3.4::gentoo
app-shells/bash:           5.1_p16::gentoo
dev-java/java-config:      2.3.1::gentoo
dev-lang/perl:             5.34.1-r3::gentoo
dev-lang/python:           3.9.13::gentoo, 3.10.5::gentoo
dev-lang/rust:             1.60.0::gentoo
dev-util/cmake:            3.23.2::gentoo
dev-util/meson:            0.62.2::gentoo
sys-apps/baselayout:       2.8::gentoo
sys-apps/openrc:           0.44.10::gentoo
sys-apps/sandbox:          2.29::gentoo
sys-devel/autoconf:        2.71-r1::gentoo
sys-devel/automake:        1.16.5::gentoo
sys-devel/binutils:        2.37_p1-r2::gentoo
sys-devel/binutils-config: 5.4.1::gentoo
sys-devel/gcc:             11.3.0::gentoo
sys-devel/gcc-config:      2.5-r1::gentoo
sys-devel/libtool:         2.4.7::gentoo
sys-devel/llvm:            14.0.4::gentoo
sys-devel/make:            4.3::gentoo
sys-kernel/linux-headers:  5.18-r1::gentoo (virtual/os-headers)
sys-libs/glibc:            2.34-r13::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts:
    sync-rsync-verify-max-age: 24

guru
    location: /var/db/repos/guru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/guru.git
    masters: gentoo

mysql
    location: /var/db/repos/mysql
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/proj/mysql.git
    masters: gentoo

slonko
    location: /var/db/repos/slonko
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/slonko.git
    masters: gentoo

gentoo-dev-alaric
    location: /var/db/repos/alaric
    sync-type: rsync
    sync-uri: rsync://babylon5/dev-alaric
    masters: gentoo
    priority: 60
    sync-rsync-extra-opts:
    sync-rsync-vcs-ignore: true

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe -mfpmath=sse"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind /var/lib/unifi"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe -mfpmath=sse"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --verbose-conflicts --keep-going"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.osuosl.org                 http://www.gtlib.gatech.edu/pub/gentoo                 http://mirrors.cs.wmich.edu/gentoo                 http://distfiles.gentoo.org                 "
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en_US en"
MAKEOPTS="-j16"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="acl aes amd64 bash-completion bzip2 cdda cddb cli crypt dri elogind fltk fortran gdbm iconv id3tag imagemagick ipv6 jpeg2k libglvnd libtirpc mmx mmxext multilib mysql ncurses nls nptl nsplugin openmp opus pam pcre readline seccomp speex split-usr sse sse2 sse4 sse4_1 sse4_2 ssl theora threads tk tools unicode utils v4l v4l2 vdpau vorbis x264 xattr xpm zlib" ABI_X86="64" ADA_TARGET="gnat_2020" APACHE2_MODULES="actions alias auth_basic authn_core authn_alias authn_anon authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers http2 include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_fcgi proxy_html proxy_http proxy_http2 rewrite setenvif socache_shmcb speling status unique_id unixd userdir vhost_alias xml2enc" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27" USERLAND="GNU" VIDEO_CARDS="mga" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Comment 1 Florian Schmaus gentoo-dev 2022-07-06 15:56:58 UTC
I suspect your ejabberd user has HOME set to /dev/null. Is it possible that you have acct-user/ejabberd-0 installed? If so, does upgrading to acct-user/ejabberd-1 help?
Comment 2 Phil Stracchino (Unix Ronin) 2022-07-06 17:56:06 UTC
Hi Florian,
acct-user/ejabberd is version 1.  However you're correct that user ejabberd's home is /dev/null.

What is the recommended setting here?
Comment 3 Florian Schmaus gentoo-dev 2022-07-06 17:58:28 UTC
With acct-user/ejabberd-1 it should be /var/lib/ejabbberd (was /dev/null with acct-user/ejabberd-0). However I had expect acct-user/ejabberd-1 to automatically update the entry for HOME in the system's user database.
Comment 4 Phil Stracchino (Unix Ronin) 2022-07-06 17:59:30 UTC
Never mind, I removed and re-merged acct-user/ejabberd, and ~ejabberd is now /var/lib/ejabberd.  I'll try re-merging 22.05 now.
Comment 5 Phil Stracchino (Unix Ronin) 2022-07-06 17:59:54 UTC
(In reply to Florian Schmaus from comment #3)
> With acct-user/ejabberd-1 it should be /var/lib/ejabbberd (was /dev/null
> with acct-user/ejabberd-0). However I had expect acct-user/ejabberd-1 to
> automatically update the entry for HOME in the system's user database.

Apparently it doesn't ...
Comment 6 Phil Stracchino (Unix Ronin) 2022-07-06 18:08:07 UTC
Confirmed that was the problem.  Updating from acct-user/ejabberd-0 to acct-user/ejabberd-1 DOES NOT update the home directory of an already-existing ejabberd user, so ejabberd-22.05 fails to start.

Not sure whether this should be patched by net-im/ejabberd, or fixed in acct-user/ejabberd.
Comment 7 Florian Schmaus gentoo-dev 2022-07-07 07:08:40 UTC
Works as intended here:

$ emerge -1 =acct-user/ejabberd-0
$ getent passwd ejabberd                                                                                                                                                                                                                                           
ejabberd:x:980:114:User for net-im/ejabberd:/dev/null:/sbin/nologin

$ emerge -1 =acct-user/ejabberd-1
$ getent passwd ejabberd
ejabberd:x:980:114:User for net-im/ejabberd:/var/lib/ejabberd:/sbin/nologin

I don't expect that you have some custom overrides in /etc/portage?
Comment 8 Phil Stracchino (Unix Ronin) 2022-07-07 15:25:42 UTC
(In reply to Florian Schmaus from comment #7)
> Works as intended here:
> 
> $ emerge -1 =acct-user/ejabberd-0
> $ getent passwd ejabberd                                                    
> 
> ejabberd:x:980:114:User for net-im/ejabberd:/dev/null:/sbin/nologin
> 
> $ emerge -1 =acct-user/ejabberd-1
> $ getent passwd ejabberd
> ejabberd:x:980:114:User for net-im/ejabberd:/var/lib/ejabberd:/sbin/nologin
> 
> I don't expect that you have some custom overrides in /etc/portage?

Nope.  If it worked for you, I have no idea why it didn't for me.
Comment 9 Florian Schmaus gentoo-dev 2022-07-07 16:17:08 UTC
Could you maybe dig into your logs? The should say something like "Updating home for user…" [1]. Can you reproduce the issue? If yes, then the debug logs while emerging acct-user/ejabberd-1 would be interesting, especially those of the pkg_postinst() phase [2].

1: https://github.com/gentoo/gentoo/blob/cbfdb24008d7ff3c84e1e9674dfe1f42e916ad3e/eclass/user.eclass#L403-L404
2: https://github.com/gentoo/gentoo/blob/2588af0e4d08fae6e3ba3fbaa885c6500885444c/eclass/acct-user.eclass#L486
Comment 10 Phil Stracchino (Unix Ronin) 2022-07-07 16:32:21 UTC
(In reply to Florian Schmaus from comment #9)
> Could you maybe dig into your logs? The should say something like "Updating
> home for user…" [1]. Can you reproduce the issue? If yes, then the debug
> logs while emerging acct-user/ejabberd-1 would be interesting, especially
> those of the pkg_postinst() phase [2].
> 
> 1:
> https://github.com/gentoo/gentoo/blob/
> cbfdb24008d7ff3c84e1e9674dfe1f42e916ad3e/eclass/user.eclass#L403-L404
> 2:
> https://github.com/gentoo/gentoo/blob/
> 2588af0e4d08fae6e3ba3fbaa885c6500885444c/eclass/acct-user.eclass#L486

No 'Updating home...' found in emerge.log.  This is all that is in emerge.log for acct-user/ejabberd-1:


1652878282: Started emerge on: May 18, 2022 08:51:21
1652878282:  *** emerge --verbose-conflicts --newuse --update --ask --deep --keep-going --with-bdeps=y --regex-search-auto=y --verbose world
1652878369:  >>> emerge (1 of 3) acct-user/ejabberd-1 to /
1652878369:  === (1 of 3) Cleaning (acct-user/ejabberd-1::/usr/portage/acct-user/ejabberd/ejabberd-1.ebuild)
1652878369:  === (1 of 3) Compiling/Merging (acct-user/ejabberd-1::/usr/portage/acct-user/ejabberd/ejabberd-1.ebuild)
1652878376:  === (1 of 3) Merging (acct-user/ejabberd-1::/usr/portage/acct-user/ejabberd/ejabberd-1.ebuild)
1652878379:  >>> AUTOCLEAN: acct-user/ejabberd:0
1652878379:  === Unmerging... (acct-user/ejabberd-0)
1652878381:  >>> unmerge success: acct-user/ejabberd-0
1652878384:  === (1 of 3) Post-Build Cleaning (acct-user/ejabberd-1::/usr/portage/acct-user/ejabberd/ejabberd-1.ebuild)
1652878384:  ::: completed emerge (1 of 3) acct-user/ejabberd-1 to /


I can force a downgrade to -0, manually adjust the home back to /dev/null, and then re-upgrade, if you wish.
Comment 11 Florian Schmaus gentoo-dev 2022-07-31 07:55:46 UTC
I found the root cause in my logs:

  * Updating home for user 'ejabberd' ...
  *  - Home: /var/lib/ejabberd
 usermod: user ejabberd is currently used by process 434
  * ejabberd is in use, cannot update home
  * There was an error when attempting to update the home directory for ejabberd
  * Please update it manually on your system (as root):
  *          usermod -d "/var/lib/ejabberd" "ejabberd"
Comment 12 Phil Stracchino (Unix Ronin) 2022-07-31 16:39:25 UTC
(In reply to Florian Schmaus from comment #11)
> I found the root cause in my logs:
> 
>   * Updating home for user 'ejabberd' ...
>   *  - Home: /var/lib/ejabberd
>  usermod: user ejabberd is currently used by process 434
>   * ejabberd is in use, cannot update home
>   * There was an error when attempting to update the home directory for
> ejabberd
>   * Please update it manually on your system (as root):
>   *          usermod -d "/var/lib/ejabberd" "ejabberd"

Aaaaaaahhhhh.  Can't update the user's home directory while it's in use.  Makes perfect sense.

So to work without manual fixing, this needs to stop ejabberd while it updates the user, then restart it.

This is something to bear in mind for nearly every acct-user ebuild, I imagine.