Cannot load Cherokee admin anymore. This is on at least two servers I'm using with different cherokee.conf This used to work fine on both but as I haven't tried it since the last upgrade, this might be an issue of the most recent version. # cherokee-admin --bind=179.99.61.52 --port=9090 -x Cherokee Web Server 1.2.101 (Aug 13 2012): Listening on port 179.99.61.52:9090, TLS disabled, IPv6 enabled, using epoll, 4096 fds system limit, max. 2041 connections, 4 threads, 510 connections per thread, standard scheduling policy Login: User: admin One-time Password: 3UOGPW1PFr6s6tSc Web Interface: URL: http://179.99.61.52:9090/ DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4020 Port=4000 DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4026 Port=4000 DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4042 Port=4000 DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4048 Port=4000 DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4054 Port=4000 ^CCherokee-admin is exiting.. Reproducible: Always Steps to Reproduce: 1. Launch cherokee-admin 2. Connect from a browser, password dialog pop-up up 3. enter temporary password 4. waiting for the admin to load but it doesn't Actual Results: after a long wait I get 503 503 Service Unavailable Cherokee web server 1.2.101 (Gentoo Linux), Port 9090 Expected Results: Load the admin
This is the output using the -t option. Without the -t these lines do NOT appear: DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4020 Port=4000 I've eventually hit ctrl-c so that explains the: ^CCherokee-admin is exiting..
(In reply to comment #0) > This used to work fine on both but as I haven't tried it since the last > upgrade, this might be an issue of the most recent version. Can you tell me what version worked and which one didn't? Also provide your emerge --info.
The last time I needed the admin interface was a few months back so I'm unsure. Here the output of emerge --info www-servers/cherokee Portage 2.1.11.31 (default/linux/amd64/10.0, gcc-4.5.4, glibc-2.15-r3, 3.1.5-gentoo x86_64) ================================================================= System Settings ================================================================= System uname: Linux-3.1.5-gentoo-x86_64-Intel-R-_Core-TM-_i7-2600_CPU_@_3.40GHz-with-gentoo-2.1 Timestamp of tree: Mon, 05 Nov 2012 02:45:01 +0000 ld GNU ld (GNU Binutils) 2.22 app-shells/bash: 4.2_p37 dev-lang/python: 2.7.3-r2, 3.2.3 dev-util/cmake: 2.8.6-r4 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.9.8.4 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.68 sys-devel/automake: 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.5.4 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=native -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=native -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://rush.tisys.org/pub/gentoo/" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rush.tisys.org/gentoo-portage" USE="acl amd64 apache2 berkdb bzip2 cli cracklib crypt cups cxx dri fortran gdbm iconv ipv6 jpeg mmx modules mudflap multilib ncurses nls nptl openmp pam pcre png pppd readline session sse sse2 ssl tcpd threads unicode zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON ================================================================= Package Settings ================================================================= www-servers/cherokee-1.2.101-r2 was built with the following: USE="admin ipv6 (multilib) nls pam php rrdtool ssl -debug -ffmpeg -geoip -ldap -mysql -static -static-libs" nects ~ # FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://rush.tisys.org/pub/gentoo/" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rush.tisys.org/gentoo-portage" USE="acl amd64 apache2 berkdb bzip2 cli cracklib crypt cups cxx dri fortran gdbm iconv ipv6 jpeg mmx modules mudflap multilib ncurses nls nptl openmp pam pcre png pppd readline session sse sse2 ssl tcpd threads unicode zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON ================================================================= Package Settings ================================================================= www-servers/cherokee-1.2.101-r2 was built with the following: USE="admin ipv6 (multilib) nls pam php rrdtool ssl -debug -ffmpeg -geoip -ldap -mysql -static -static-libs"
(In reply to comment #3) > The last time I needed the admin interface was a few months back so I'm > unsure. > Here the output of emerge --info www-servers/cherokee > Thanks. Unfortunately I can't reproduce this on a nearly identical system. Let's try this: 1) start up cherokee-admin 2) post `netstat -ntlp` 3) start up `tcpdump -n -i eth0 tcp and port 9090` and try to connect, let's see if its actually communicating. There's a lot here, so my may want to just post a few lines to make sure there are packets being exchanged. 4) if it is, then let's see why its hanging, do `strace -f -o cherokee.log cherokee-admin --bind=179.99.61.52 --port=9090 -x` and post that log up to where it hangs.
Created attachment 328510 [details] strace -f -o cherokee.log cherokee-admin --bind=179.99.61.52 --port=9090 -x As requested launched: strace -f -o cherokee.log cherokee-admin --bind=179.99.61.52 --port=9090 -x
Created attachment 328512 [details] requested tcpdump
Added the attachments you requested. Had there been no physical connection, I wouldn't have received the login prompt and finally the 503 response. Let me know what else I could be of use in.
(In reply to comment #7) > Added the attachments you requested. Had there been no physical connection, > I wouldn't have received the login prompt and finally the 503 response. Let > me know what else I could be of use in. I compared your strace to mine. I can see things are going horribly bad but I don't know why. You main worker pid=11172 stalls a lot, and finally gets killed before making a loopback connection to continue the work. To see this do cat cherokee.log | grep 11172 There you see the final stall on a select followed by being kiled by its parten: 11172 clone(child_stack=0x7f304f06cff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f304f06d9d0, tls=0x7f304f06d700, child_tidptr=0x7f304f06d9d0) = 11177 11172 select(4, [3], [], [], NULL <unfinished ...> ... 11171 kill(11172, SIG_0) = 0 11171 kill(11172, SIG_0) = 0 11171 kill(11172, SIG_0) = 0 11171 kill(11172, SIG_0) = 0 11171 kill(11172, SIG_0) = 0 11171 kill(11172, SIG_0) = 0 ... Compare this to my working version where you get ... 3139 clone(child_stack=0x36269f09ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x36269f0a9d0, tls=0x36269f0a700, child_tidptr=0x36269f0a9d0) = 3144 3139 select(4, [3], [], [], NULL) = 1 (in [3]) 3139 accept(3, {sa_family=AF_INET, sin_port=htons(58581), sin_addr=inet_addr("127.0.0.1")}, [16]) = 4 3139 mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x36268f09000 3139 mprotect(0x36268f09000, 4096, PROT_NONE) = 0 ... So you can see an accepted connection on 127.0.0.1 and then more work being done on mine but not yours. Okay four more thing (sorry I'd probably have this done if I could reproduce it): 1) let me see your config files. I can see from the strace that they've been edited. Hopefully I might be able to reproduce using them. 2) Make sure your python-2.7 has threading support. I'm 99% sure but let's be 100%. 3) It may not be related, but I saw a lot of ipv6 whereas on mine I had just plain old ipv4. If you can, try turning off ipv6 support server wide: echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6 I'm 99% sure this isn't it, but I want to eliminate differences. 4) Repeat what you did in your original description. When you see the message: DEBUG: SIGUSR1 invokes the console.. SIGUSR2 prints a backtrace.. Server 1.2.101 running.. PID=4020 Port=4000 wait till it just sits there, then do `kill -USR2 4020`. Of course, replace the pid with whatever it cherokee-admin gives you. Then cut and paste the backtrace. Maybe do it a few times to make sure you are catching the code in the same place. Its hard working backwards from your strace to figure out where in the code its happening.
Thanks for the reply and I hope we'll get to the bottom of it all. 1) I renamed cherokee.conf so it won't be accessed but the same happens so I won't blame my config. [09/11/2012 15:01:26.138] (critical) config_reader.c:65 - Could not access file | The configuration file '/etc/cherokee/cherokee.conf' could not be accessed. Most probably the server user does not have enough permissions to read it. Cherokee Web Server 1.2.101 (Nov 7 2012): Listening on port 179.99.61.50:9090, TLS disabled, IPv6 enabled, using epoll, 4096 fds system limit, max. 2041 connections, caching I/O, 4 threads, 510 connections per thread, standard scheduling policy Login: User: admin One-time Password: yau4cVyP73T6O7k4 Web Interface: URL: http://179.99.61.50:90/ 2) It does: [U] dev-lang/python Available versions: (2.5) 2.5.4-r4 (2.6) 2.6.8 (2.7) 2.7.3-r2 (3.1) 3.1.5 (3.2) 3.2.3 ~3.2.3-r1 (3.3) **3.3.0 {{-berkdb build doc elibc_uclibc examples gdbm ipv6 +ncurses +readline sqlite +ssl +threads tk +wide-unicode wininst +xml}} Installed versions: 2.7.3-r2(2.7)(09:26:57 AM 06/15/2012)(gdbm ipv6 ncurses readline ssl threads wide-unicode xml -berkdb -build -doc -elibc_uclibc -examples -sqlite -tk -wininst) 3.1.4-r3(3.1)(11:35:04 AM 12/08/2011)(gdbm ipv6 ncurses readline ssl threads wide-unicode xml -build -doc -elibc_uclibc -examples -sqlite -tk -wininst) 3.2.3(3.2)(06:52:19 AM 05/19/2012)(gdbm ipv6 ncurses readline ssl threads wide-unicode xml -build -doc -elibc_uclibc -examples -sqlite -tk -wininst) and while we're at it, here's are Cherokee's compilation options: [I] www-servers/cherokee Available versions: (~)1.2.101-r1^t (~)1.2.101-r2^t {{admin coverpage debug ffmpeg geoip ipv6 kernel_linux ldap mysql nls pam php rrdtool ssl static static-libs}} Installed versions: 1.2.101-r2^t(05:32:29 PM 11/07/2012)(admin ipv6 kernel_linux mysql nls pam php rrdtool ssl -debug -ffmpeg -geoip -ldap -static -static-libs) 3) Guess what? It was the 1%... I'll try to re-compile without ipv6 support and see if it that works without the server-wide change.
> 3) Guess what? It was the 1%... I'll try to re-compile without ipv6 support > and see if it that works without the server-wide change. I would conclude then that its a bug in cherokee and should go upstream. I'll hack away at the code once I get your result.
I've tested a few times and can confirm that the ipv6 is the cause of the problem in cherokee-admin. Specifically: when I did echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6 it stopped working and when I recompiled on gentoo without ipv6 support it worked.
Thank you for your help :) I can now workaround what I need to do.
Upstream is aware of the issue: http://code.google.com/p/cherokee/issues/detail?id=719
(In reply to A Collector from comment #12) > Thank you for your help :) I can now workaround what I need to do. I'm trying to work around it, but I still can't manage to. Can you please explain the workaround step-by-step?
Sorry, step by step would likely depends on your system settings. Turn off ipv6 support and recompile cherokee and it should work. Mind Cherokee doesn't seem very active so u might wanna consider another server.
(In reply to A Collector from comment #15) > Sorry, step by step would likely depends on your system settings. Turn off > ipv6 support and recompile cherokee and it should work. Mind Cherokee > doesn't seem very active so u might wanna consider another server. I think this is fixed. (In reply to A Collector from comment #11) > I've tested a few times and can confirm that the ipv6 is the cause of the > problem in cherokee-admin. > Specifically: when I did > echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6 > it stopped working and when I recompiled on gentoo without ipv6 support it > worked. This is fixed in 1.2.104 that I just added to the tree. Upstream never really closed that bug so I just tested directly. Sorry it took so long :( But its done.
