After emerging dev-java/tomcat-native-1.1.10 www-servers/tomcat-6.0.14-r1 didn't continue to start after loading the native library catalina.out: Nov 6, 2007 1:29:05 AM org.apache.catalina.core.AprLifecycleListener init INFO: Loaded Apache Tomcat Native library 1.1.10. Nov 6, 2007 1:29:05 AM org.apache.catalina.core.AprLifecycleListener init INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. No further output in catalina.out after waiting for some minutes. The solution was to disable SSLEngine in the AprLifecycleListener directive. server.xml: <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" /> catalina.out: Nov 6, 2007 1:33:24 AM org.apache.catalina.core.AprLifecycleListener init INFO: Loaded Apache Tomcat Native library 1.1.10. Nov 6, 2007 1:33:24 AM org.apache.catalina.core.AprLifecycleListener init INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Nov 6, 2007 1:33:24 AM org.apache.coyote.http11.Http11AprProtocol init INFO: Initializing Coyote HTTP/1.1 on http-8080 Nov 6, 2007 1:33:24 AM org.apache.coyote.ajp.AjpAprProtocol init INFO: Initializing Coyote AJP/1.3 on ajp-8009 Nov 6, 2007 1:33:24 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 388 ms Nov 6, 2007 1:33:24 AM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Nov 6, 2007 1:33:24 AM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0-snapshot Nov 6, 2007 1:33:24 AM org.apache.coyote.http11.Http11AprProtocol start INFO: Starting Coyote HTTP/1.1 on http-8080 Nov 6, 2007 1:33:24 AM org.apache.coyote.ajp.AjpAprProtocol start INFO: Starting Coyote AJP/1.3 on ajp-8009 Nov 6, 2007 1:33:24 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 394 ms I have dev-libs/openssl-0.9.8d installed. Maybe there is some dependency missing in the ebuild? Reproducible: Always java-config -L The following VMs are available for generation-2: *) Sun JDK 1.6.0.03 [sun-jdk-1.6] No generation-1 vm, tomcat and tomcat-native were build with 1.6.0.03. Portage 2.1.3.16 (default-linux/amd64/2007.0, gcc-4.1.1, glibc-2.5-r0, 2.6.22-gentoo-r9 x86_64) ================================================================= System uname: 2.6.22-gentoo-r9 x86_64 Intel(R) Xeon(R) CPU E5335 @ 2.00GHz Timestamp of tree: Fri, 02 Nov 2007 17:30:09 +0000 app-shells/bash: 3.2_p17 dev-java/java-config: 1.3.7, 2.0.33-r1 dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 1.12.9 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.61 sys-devel/automake: 1.9.6-r2, 1.10 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/terminfo" CXXFLAGS="-march=nocona -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="acl amd64 bash-completion berkdb bitmap-fonts cli cracklib crypt dri fortran gdbm gpm iconv imap ipv6 isdnlog java ldap midi mmx mudflap ncurses nls nptl nptlonly openmp pam pcre perl postgres pppd python readline reflection samba session spl sse sse2 ssl svg tcpd truetype truetype-fonts type1-fonts unicode xml xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i810 mach64 mga neomagic nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa vga via vmware voodoo" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
Confirmed it just hangs here as well. Pretty odd because it doesn't finish initializing to the point where it's listening on ports. However it stays running and responds to being stopped by init script. Pretty odd, and it used to work. Not sure off hand what's causing it to no longer work. Will look further into it as time permits.
Bumped to 1.1.12, and unfortunately it still hangs :( Will have to look into this further when time permits
same problem with tomcate-native, my versions: sys-kernel/gentoo-sources-2.6.24 dev-libs/openssl-0.9.8g dev-java/sun-jdk-1.6.0.04 dev-java/tomcat-native-1.1.12 www-servers/tomcat-5.5.25-r1 -> no problem at all www-servers/tomcat-6.0.14-r1 -> no problem with SSLEngine="off" at startup, works. www-servers/tomcat-6.0.14-r1 -> problem with SSLEngine="on" at startup of tomcat-6 in this line at /etc/tomcat-6/server.xml: <!--APR library loader. Documentation at /docs/apr.html --> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> and "./configure" makes a lot of "command not found"-noises: ggpxpe013 ~ # ebuild /usr/portage/dev-java/tomcat-native/tomcat-native-1.1.12.ebuild compile >>> Existing ${T}/environment for 'tomcat-native-1.1.12' will be sourced. >>> Run 'clean' to start with a fresh environment. * tomcat-native-1.1.12-src.tar.gz RMD160 SHA1 SHA256 size ;-) ... [ ok ] * checking ebuild checksums ;-) ... [ ok ] * checking auxfile checksums ;-) ... [ ok ] * checking miscfile checksums ;-) ... [ ok ] * checking tomcat-native-1.1.12-src.tar.gz ;-) ... [ ok ] >>> Checking tomcat-native-1.1.12-src.tar.gz's mtime... >>> WORKDIR is up-to-date, keeping... >>> Compiling source in /var/tmp/portage/dev-java/tomcat-native-1.1.12/work/tomcat-native-1.1.12-src ... * econf: updating tomcat-native-1.1.12-src/jni/native/build/config.sub with /usr/share/gnuconfig/config.sub * econf: updating tomcat-native-1.1.12-src/jni/native/build/config.guess with /usr/share/gnuconfig/config.guess ./configure --prefix=/usr --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --with-apr=/usr/bin/apr-1-config --with-ssl=/usr --build=i686-pc-linux-gnu checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking target system type... i686-pc-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking for working mkdir -p... yes Tomcat Native Version: 1.1.12 checking for chosen layout... tcnative checking for APR... yes setting CC to "i686-pc-linux-gnu-gcc" setting CPP to "i686-pc-linux-gnu-gcc -E" checking for a BSD-compatible install... /usr/bin/install -c checking for JDK location (please wait)... /opt/sun-jdk-1.6.0.04 from environment checking Java platform... checking Java platform... checking for sablevm... NONE adding "-I/opt/sun-jdk-1.6.0.04/include" to TCNATIVE_PRIV_INCLUDES checking os_type directory... linux adding "-I/opt/sun-jdk-1.6.0.04/include/linux" to TCNATIVE_PRIV_INCLUDES checking for i686-pc-linux-gnu-gcc... i686-pc-linux-gnu-gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether i686-pc-linux-gnu-gcc accepts -g... yes checking for i686-pc-linux-gnu-gcc option to accept ISO C89... none needed checking for OpenSSL library... using openssl from /usr/lib and /usr/include checking OpenSSL library version... ok checking for OpenSSL DSA support... yes setting TCNATIVE_LDFLAGS to "-lssl -lcrypto" adding "-DHAVE_OPENSSL" to CFLAGS setting TCNATIVE_LIBS to "" setting TCNATIVE_LIBS to " /usr/lib/libapr-1.la -luuid -lrt -lcrypt -lpthread -ldl" ./configure: line 4315: AA: command not found ./configure: line 4316: ALSA_CARDS: command not found ./configure: line 4317: ALSA_PCM_PLUGINS: command not found ./configure: line 4318: ANT_HOME: command not found ./configure: line 4319: ANT_OPTS: command not found ./configure: line 4320: ANT_TASKS: command not found ...
(In reply to comment #3) > same problem with tomcate-native, my versions: If you read the last comment by me before your post. I was already aware 1.1.12 also had the same problem.Reporting a known problem again doesn't really do anything. Really not sure what's going on. Haven't had a chance to fully debug it. Will see about making some time for that ASAP. It's pretty odd, because it used to work fine. Not sure when it stopped working or what caused that.
So far a workaround is to disable ssl for apr (tomcat-native) in "server.xml": <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" /> and when using SSL to change the <Connector protocol="org.apache.coyote.http11.Http11AprProtocol" port="8443" ... /> to <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="8443" ... /> Works for me. Still have apr for http. Hope it helps other tomcat-users for now. And it maybe "localizes" the problem with tomcat-native.
(In reply to comment #5) > So far a workaround is to disable ssl for apr (tomcat-native) in "server.xml": You do realize the most of the point to Tomcat using APR is for SSL acceleration. Unless you are serving up large static files. For most installs, I doubt there would be a noticeable performance increase. > Works for me. Still have apr for http. Hope it helps other tomcat-users for > now. And it maybe "localizes" the problem with tomcat-native. Might be a work around, but surely not a fix. Does nothing for those using it for SSL acceleration. However that the SSL bit causes the hang, and without doesn't. Could mean that their is some problem in the native connection of ssl. Maybe coded against and older version or something.
I played along and "tomcat-native" with apr/ssl works for me: emerge -ev world openssl genrsa -out tomcat_privkey.pem 2048 openssl req -new -x509 -key tomcat_privkey.pem -out tomcat_cacert.pem -days 1095 cp *.pem /etc/tomcat-6/ chown tomcat:tomcat /etc/tomcat-6/*.pem nano /etc/tomcat-6/server.xml (modified, added lines): <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <Connector protocol="org.apache.coyote.http11.Http11AprProtocol" port="8443" minSpareThreads="5" maxSpareThreads="75" enableLookups="true" disableUploadTimeout="true" acceptCount="100" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" SSLCertificateFile="/etc/tomcat-6/tomcat_cacert.pem" SSLCertificateKeyFile="/etc/tomcat-6/tomcat_privkey.pem" clientAuth="false" sslProtocol="TLS" /> It's a pity about the load times then (several restarts of tomcat per line): Feb 6, 2008 10:22:57 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 189956 ms ... INFO: Initialization processed in 1520 ms INFO: Initialization processed in 71551 ms INFO: Initialization processed in 144499 ms INFO: Initialization processed in 3937 ms INFO: Initialization processed in 78990 ms INFO: Initialization processed in 127022 ms (normally <4000 ms) But it works, this matters for me. Hope it helps.
Just committed 1.1.13 and seems tomcat was able to fully start this time :) Really not sure what's up with 1.1.11 and 1.1.12. Please test and confirm if this version works for you or not. Will close bug after a few confirmations, like from reporter or commenters. Thanks
I tested tomcat-native-1.1.12.ebuild and tomcat-native-1.1.13.ebuild with exactly the same environment as in my bug description but tomcat still doesn't start until I disable SSLEngine in the AprLifecycleListener directive. I found it strange that tomcat reported INFO: Loaded Apache Tomcat Native library 1.1.12. after I emerged tomcat-native-1.1.13.ebuild. But the version info changes when I emerge 1.1.10 - so 1.1.13 perhaps displays 1.1.12 too?
(In reply to comment #9) > I found it strange that tomcat reported > INFO: Loaded Apache Tomcat Native library 1.1.12. > after I emerged tomcat-native-1.1.13.ebuild. But the version info changes when > I emerge 1.1.10 - so 1.1.13 perhaps displays 1.1.12 too? Yes, upstream screwed up there, and didn't bump version :( What exactly is the problem you are experiencing? Because it starts here all day long on many machines with tomcat-native 1.1.13 merged (despite showing up as 1.1.12). I no longer experience the hanging and emerging past versions like 1.1.12 no longer hang. What version of Tomcat are you using? Seems to be getting picky there. I think >=1.1.12 only work with > 6.0.14. Might add a runtime dependency of Tomcat and make it version specific.
I'm using tomcat-6.0.14-r1, apr-1.2.12, openssl-0.9.8g and sun-jdk-1.6.0.03 on amd64. After waiting a little longer I can confirm that tomcat starts but waits for at least 30 seconds to several minutes after loading tomcat-native. catalina.out: log4j:WARN No such property [datePattern] in org.apache.log4j.RollingFileAppender. Feb 18, 2008 7:13:38 PM org.apache.catalina.core.AprLifecycleListener init INFO: Loaded Apache Tomcat Native library 1.1.12. Feb 18, 2008 7:13:38 PM org.apache.catalina.core.AprLifecycleListener init INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. [wait for 10 minutes...] Feb 18, 2008 7:23:56 PM org.apache.coyote.http11.Http11AprProtocol init INFO: Initializing Coyote HTTP/1.1 on http-12001 Feb 18, 2008 7:23:56 PM org.apache.coyote.ajp.AjpAprProtocol init INFO: Initializing Coyote AJP/1.3 on ajp-12101 Feb 18, 2008 7:23:56 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 618870 ms Feb 18, 2008 7:23:56 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Feb 18, 2008 7:23:56 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0-snapshot Feb 18, 2008 7:23:56 PM org.apache.catalina.startup.HostConfig deployDescriptor Feb 18, 2008 7:23:57 PM org.apache.catalina.startup.HostConfig deployDescriptor Feb 18, 2008 7:23:57 PM org.apache.coyote.http11.Http11AprProtocol start INFO: Starting Coyote HTTP/1.1 on http-12001 Feb 18, 2008 7:23:57 PM org.apache.coyote.ajp.AjpAprProtocol start INFO: Starting Coyote AJP/1.3 on ajp-12101 Feb 18, 2008 7:23:57 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 756 ms Could you give me a hint how to debug what tomcat is doing while waiting? My guts say it has something to do with ipv6...
(In reply to comment #11) > I'm using tomcat-6.0.14-r1, apr-1.2.12, openssl-0.9.8g and sun-jdk-1.6.0.03 on > amd64. After waiting a little longer I can confirm that tomcat starts but waits > for at least 30 seconds to several minutes after loading tomcat-native. Yeah and I have to go back and re-confirm it was actually hanging for me verses being impatient :) > Could you give me a hint how to debug what tomcat is doing while waiting? Being as how I really don't even using APR. I haven't a clue what could be causing that. Sorry. > My > guts say it has something to do with ipv6... Possibly, I have it on one machine here but not another. Will look into it as time permits.
(In reply to comment #11) > Could you give me a hint how to debug what tomcat is doing while waiting? My > guts say it has something to do with ipv6... > That sounds reasonable. although I have -ipv6 in my global USE. and im also getting this line: INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
I am going to assume any delays on start are some sort of network time out or etc. Not 100% where it comes from. But that Tomcat does start, any delays are moot. If the delays are of concern. It's very likely a general issue, nothing really specific to Tomcat on Gentoo. Since it seems most of this bug was just impatience on tomcat starting, which includes myself :). Pretty sure this is invalid, and we can close as such.
Closing this as invalid for now. So we can stabilize 1.1.13 per security bug 209410. Willing to re-open if we can prove Tomcat actually hangs. And/or if the delay is something Gentoo specific. Otherwise potential upstream issue/bug.
I've found this thread: http://www.nabble.com/Very-Slow-Startup-with-APR-tt12245879.html#a12245879 This is an upstream bug but: It seems like a /dev/random blocking issue. I think it might help if there was a way to tell the ebuild to use the non-blocking /dev/urandom instead of /dev/random This way this bug (that still exists) won't affect us.
Simply add dev-libs/apr urandom to /etc/portage/package.use then emerge apr and problem is solved. HTH. Cheers, David Sperling
Nice job tracking that down, thanks David for a suggestion as to how to fix/avoid.
