| Summary: | www-servers/tomcat-7.0.X - classloading issues due to too much jars in initial classpath | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Marc <gigerstyle> |
| Component: | [OLD] Java | Assignee: | Java team <java> |
| Status: | RESOLVED FIXED | ||
| Severity: | major | CC: | bfx81, heavymetal, marco.dr |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=551050 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | tomcat.init file with fixed classloading | ||
I can confirm the problem you're talking about (specifically https://bugs.gentoo.org/show_bug.cgi?id=462096) I've edited my init.d/tomcat-7 script with a temporary fixed classpath: CLASSPATH="/usr/share/tomcat-7/bin/bootstrap.jar:/usr/share/tomcat-7/bin/tomcat-juli.jar:/usr/share/eclipse-ecj-3.7/lib/ecj.jar:/usr/share/tomcat-servlet-api-3.0/lib/el-api.jar:/usr/share/tomcat-servlet-api-3.0/lib/jsp-api.jar:/usr/share/tomcat-servlet-api-3.0/lib/servlet-api.jar" This solve now the classloading of my jdbc driver, but as you stated, probably the best would be just a CLASSPATH="$CATALINA_HOME/bin/bootstrap.jar:$CATALINA_HOME/bin/tomcat-juli.jar" The problem now, i think, is related to the following jars /usr/share/eclipse-ecj-3.7/lib/ecj.jar /usr/share/tomcat-servlet-api-3.0/lib/el-api.jar /usr/share/tomcat-servlet-api-3.0/lib/jsp-api.jar /usr/share/tomcat-servlet-api-3.0/lib/servlet-api.jar that needs to be in $CATALINA_HOME/lib. I've this idea... I ask to the java-team to tell me if it's a feasible solution: * while emerging tomcat, the package is aware about his own dependencies, so the ebuild could symlink these jars in $CATALINA_HOME/lib * updating a tomcat dependency (es servlet-api): * a post-inst warning messages about the possible broken link * (or better) check the symlink and update accordingly thanks and let me know... Fabio *** Bug 462096 has been marked as a duplicate of this bug. *** Created attachment 362526 [details]
tomcat.init file with fixed classloading
Fixed the classloading excluding from the classpath all the jar in /usr/share/tomcat-7/lib/
Works for me in production machine
ping (and happy new year) Hi Fabio, Happy new year to you too! Ok, your change to the init script will lead to the following initial classpath: /usr/share/tomcat-7/bin/bootstrap.jar:/usr/share/tomcat-7/bin/tomcat-juli.jar:/usr/share/eclipse-ecj-4.2/lib/ecj.jar:/usr/share/tomcat-servlet-api-3.0/lib/el-api.jar:/usr/share/tomcat-servlet-api-3.0/lib/jsp-api.jar:/usr/share/tomcat-servlet-api-3.0/lib/servlet-api.jar That looks much better and seems to work. But, I'm not sure if that is the proper way how to do it because I think we still have to much jars in the bootstrap / system classpath. If we look at the classloader documentation of tomcat (http://tomcat.apache.org/tomcat-7.0-doc/class-loader-howto.html) we can read that additional jars like ecj / servlet-api etc should be applied at first in the common classloader and not before. So my proposal would be one of the two solutions below: 1) Simply symlink the required jars into the /usr/share/tomcat-7/lib during install time and be happy as long as the symlinks are valid. (If I'm not wrong an update of ecj will break the tomcat installation anyway since the path to it is hardcoded in /usr/share/tomcat-7/package.env. Or will that file also be updated when ecj is updated?) Another drawback of this solution and the currently existing one is the handling of the $TOMCAT_EXTRA_JARS: They will also be added to the bootstrap / system classpath which may yield to the same classloading issues I had. To workaround that see 2) below: 2) (seems to work for me) a) export the CLASSPATH=/usr/share/tomcat-7/bin/bootstrap.jar:/usr/share/tomcat-7/bin/tomcat-juli.jar in the init script (jars may be dynamically resolved if required) b) Do not symlink the jars as proposed in 1) c) build TOMCAT_EXTRA_CLASSPATH=`java-config --classpath eclipse-ecj-4.2` + servlet-api etc + TOMCAT_EXTRA_JARS defined in /etc/conf.d/tomcat-7-xyz in the init script (comma-separated paths to the jars) d) set additional system property in the init script: -DTOMCAT_EXTRA_CLASSPATH=${TOMCAT_EXTRA_CLASSPATH} (required for the step e) below since catalina.properties do not support env variables.) e) set common.loader to common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${TOMCAT_EXTRA_CLASSPATH} in /etc/tomcat-7-xyz/catalina.properties during install of the instance. The issue that may arise with this solution is the length of the TOMCAT_EXTRA_CLASSPATH system property when many jars are added by the user (what is the limit?) May be workarounded by 2b): 2b) add an additional path like for example ${catalina.base}/extralibs/*.jar to common.loader in /etc/tomcat-7-xyz/catalina.properties and symlink all required jars in the init script on startup. What do you think? Thanks, Marc Seems the best way to handle this is to update init script to only have the 2 jars on the classpath, that ship with tomcat. Then symlink serlvet-api into lib. Also do the same with eclipse-ecj but using the location of the eselect managed symlink, that way it does not break when eclipse-ecj version changes. That should address all issues. The rest of the jars are already in lib so should be loaded normally. This should allow tomcat to function as normal, loaded classes with a minimal classpath like upstream. commit 480ab71a5cb19f40a74ffaa947b954b0ea4b2783 Author: James Le Cuirot <chewi@gentoo.org> Date: Sun Aug 16 14:39:50 2015 +0100 www-servers/tomcat: Version bumps, fix bug #453212, general cleanup * Version bumps for 7.0.63 and 8.0.24. * Start with a minimal classpath by prepending remaining jars to common.loader instead. Fixes bug #453212. * Unbundle javamail at runtime and servlet-api at build time. * Clean up test handling and add missing easymock dependency. * Fix handling of websockets flag under 7. ---------------- Sorry this took so long. After hearing about it myself, I wanted to make sure it was done in a clean way that preferably didn't involve symlinks. Luckily you can dynamically set Tomcat's common loader search path. It works for 6, 7 and 8, though 6 still needed to have jsp-api.jar and servlet-api.jar in the initial classpath. wltjr mentioned eselect-ecj and I thought that was already being used here but it turns out the ecj SLOT is hardcoded. Given that gnu_andrew would prefer a fixed version of ecj for building icedtea, this begs the question as to why we even have eselect-ecj. I should probably take the blame, I suspect I was the original author! I'll save this debate for another time though. I fixed a few other things while I was at it so please give this a try and report back. Don't forget to copy a fresh init script from /usr/share/tomcat-?/gentoo/tomcat.init. I'll close this when we mark these stable. Hi James, I've installed 8.0.24 and looked how the classpath's are assembled now. In principle it is now exactly the same setup as with my manual changes I had to apply on my production system. So from my side I would say that this fixes my issues. Apart from that I noticed the following two things: - Why is java-mail a runtime dependecy and therefore included in the common.loader classpath? And if it will stay as it is now, shouldn't then the acitvation.jar be included in the classpath too, no? - Why does the eclipse-ecj ebuild install the jar twice with different names?: ls -latr /usr/share/eclipse-ecj-4.4/lib/ total 3760 -rw-r--r-- 1 root root 1917133 Aug 22 11:12 eclipse-ecj.jar -rw-r--r-- 1 root root 1917133 Aug 22 11:12 ecj.jar The two points shouldn't hurt (me), I'm just curious. Thanks for your work and kind regards, Marc (In reply to Marc from comment #8) > - Why is java-mail a runtime dependency and therefore included in the > common.loader classpath? And if it will stay as it is now, shouldn't then > the activation.jar be included in the classpath too, no? A few javamail classes were duplicated in the tomcat source tree and I thought it would be best to unbundle them. I forget exactly which jar these classes were in before but they were already in the classpath. I'm not sure what you mean about activation.jar. I know it's part of JAF but I don't really know what that is since my Java knowledge is surprisingly limited. :( > - Why does the eclipse-ecj ebuild install the jar twice with different > names?: That was a harmless duplication bug in eclipse-ecj 4.4.1. It was fixed in 4.4.2 and I just marked that stable for you. Oops, forgot I said I'd only close this when stable. Oh well. I'll get it stabled once it's been long enough anyway. (In reply to James Le Cuirot from comment #9) > (In reply to Marc from comment #8) > > - Why is java-mail a runtime dependency and therefore included in the > > common.loader classpath? And if it will stay as it is now, shouldn't then > > the activation.jar be included in the classpath too, no? > > A few javamail classes were duplicated in the tomcat source tree and I > thought it would be best to unbundle them. I forget exactly which jar these > classes were in before but they were already in the classpath. I'm not sure > what you mean about activation.jar. I know it's part of JAF but I don't > really know what that is since my Java knowledge is surprisingly limited. :( Just for the records: I did a quick look at these "duplicated classes". These classes are just dummy implementations/interfaces so that tomcat can be compiled without having to download the javamail jar manually. Javamail itself is just a compile dependency for org.apache.naming.factory.SendMailFactory which is an optional configurable resource for sending mails. In the standard configuration it is not active and therefore javamail is in this case not required during runtime. The "perfect" solution would be to introduce a new USE flag "javamail" but as already said I'm fine if its included by default (and more important loaded from the right classloader). Regarding activation.jar: from what I read in the javamail doc (https://javamail.java.net/docs/README.txt) it is no longer required from jdk 6 on so we can safely ignore it. > > > - Why does the eclipse-ecj ebuild install the jar twice with different > > names?: > > That was a harmless duplication bug in eclipse-ecj 4.4.1. It was fixed in > 4.4.2 and I just marked that stable for you. Cool, thanks! Kind regards, Marc (In reply to Marc from comment #11) > Just for the records: > I did a quick look at these "duplicated classes". These classes are just dummy > implementations/interfaces so that tomcat can be compiled without having to > download the javamail jar manually. Javamail itself is just a compile > dependency for org.apache.naming.factory.SendMailFactory which is an > optional configurable resource for sending mails. Thanks for finding that! I should have looked at the actual contents. This approach is very rare in the Java world. > The "perfect" solution would be to introduce a new USE flag "javamail" but > as already said I'm fine if its included by default (and more important > loaded from the right classloader). I'll do that for the next version then. Results y of tests i've run on tomcat-6 (follow-up from IRC) : https://bz.apache.org/bugzilla/show_bug.cgi?id=50677 That convenient -Dvar=value is not backported to tomcat-6 it seems Pasting the calculated classpath in catalina.properties instead of ${gentoo.classpath} resolves the issue... but is not what we want :( Furthermore adding /usr/share/tomcat-6/lib/el-api.jar to system CLASSPATH in init script seems mandatory... don't know why [[ 6 = 6 ]] && CLASSPATH+=":/usr/share/tomcat-servlet-api-2.5/lib/jsp-api.jar:/usr/share/tomcat-servlet-api-2.5/lib/servlet-api.jar:/usr/share/tomcat-6/lib/el-api.jar" el-api.jar should come from tomcat-servlet-api, which tomcat-servlet-api-2.5 is not providing. I opened bug 558728 to address the missing el-api.jar from tomcat-servlet-api-2.5. Tomcat 6 presently builds and ships that jar, which also needs to be modified to use the jar from tomcat-servet-api-2.5. Just committed 6.0.44-r2, which fixes the above by backporting a change from 7. I have also fixed bug #558728. commit 540f24964bd0e5d291cfdb1f621a8ef00cc03e70 Author: Miroslav Šulc <fordfrog@gentoo.org> Date: Fri Feb 19 22:45:21 2016 +0100 www-servers/tomcat: removed oracle-javamail from deps and hence from tomcat classpath (bug #453212) this jar should not be on tomcat's classpath, if one needs it on global classpath, he/she should use TOMCAT_EXTRA_JARS in /etc/conf.d/tomcat-SLOT[-suffix] to put it on the global classpath. the before-the-fix version of the ebuilds made it impossible to put javamail jar inside web application as that resulted in conflict which could be avoided only by removal of javamail jar from the global classpath Package-Manager: portage-2.2.27 now the gentoo classpath contains this: -Dgentoo.classpath=/usr/share/eclipse-ecj-4.5/lib/ecj.jar,/usr/share/tomcat-servlet-api-4.0/lib/el-api.jar,/usr/share/tomcat-servlet-api-4.0/lib/jsp-api.jar,/usr/share/tomcat-servlet-api-4.0/lib/servlet-api.jar this fix is in following ebuilds: tomcat-7.0.68-r1 tomcat-8.0.32-r1.ebuild tomcat-9.0.0_alpha3-r1.ebuild i suppose this should be ok. if you need to apply this fix immediately, just apply appropriate keyword for these ebuilds. |
The tomcat init script calls java-config --classpath tomcat-7 to assemble the inital classpath. java-config itself gets the classpath for tomcat from /usr/share/tomcat-7/package.env . In this file the CLASSPATH consist of tons of jars which MUST NOT be included in the initial classpath. The classpath there MUST contain only the following two jars: /usr/share/tomcat-7/bin/bootstrap.jar:/usr/share/tomcat-7/bin/tomcat-juli.jar That's it, no more and no less is necessary. Everything else is a bug and leads to problems as I have now. The two jar's takes care of everything else. Why? If you add a container-managed resource like the following <Context path="/archiva" docBase="apache-archiva"> <Resource name="mail/Session" auth="Container" type="javax.mail.Session" mail.smtp.host="localhost"/> </Context> in either conf/server.xml or in conf/Catalina/localhost for a webapp you will get an ClassNotFoundException, independent where the mail.jar is copied to (I've proved with lsof that the jar was really loaded). The reason for this behavior is that tomcat builds the necessary classloader hierarchy with an exactly defined set of jars. It seems that some dependent classes are now either unknown to the actual classloader or more probably because the same class is loaded by multiple classloaders which leads to the same exception if they get in touch together. I have no idea how to solve this issue without loosing my configuration with every tomcat update... Reproducible: Always