Classpath issue between jetty-maven-plugin and tom

Question

I'm working on an app using :

jetty-maven-plugin:9.3.2.v20150730
tomcat-jdbc:8.0.8 (which has tomcat-juli as dependency)

After trying to upgrade the tomcat-jdbc jar to any version beyond 8.0.9+ I get the following error:

java.util.ServiceConfigurationError: org.apache.juli.logging.Log: Provider org.eclipse.jetty.apache.jsp.JuliLog not a subtype

Looking the changelog between those 2 versions I've found something that looks suspicious:

"Add a simple ServiceLoader based discovery mechanism to the JULI LogFactory to make it easier to use JULI and Tomcat components that depend on JULI (such as Jasper) independently from Tomcat. Patch provided by Greg Wilkins. (markt)"

Also, I've found that a new system property was introduced in the Apache Tomcat JDBC Connection Pool:

org.apache.tomcat.jdbc.pool.onlyAttemptCurrentClassLoader

"Controls classloading of dynamic classes, such as JDBC drivers, interceptors and validators. If set to false, default value, the pool will first attempt to load using the current loader (i.e. the class loader that loaded the pool classes) and if class loading fails attempt to load using the thread context loader. Set this value to true, if you wish to remain backwards compatible with Apache Tomcat 8.0.8 and earlier, and only attempt the current loader. If not set then the default value is false."

Unfortunately, starting the plugin with jetty:run using this property didn't resolve the issue.

Any help would be appreciated! Thanks!

Stack trace and dependencies tree :

Caused by: 
java.util.ServiceConfigurationError: org.apache.juli.logging.Log: Provider org.eclipse.jetty.apache.jsp.JuliLog not a subtype
    at java.util.ServiceLoader.fail(ServiceLoader.java:239)
    at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.juli.logging.LogFactory.(LogFactory.java:78)
    at org.apache.juli.logging.LogFactory.(LogFactory.java:66)
    at org.apache.tomcat.jdbc.pool.DataSourceFactory.(DataSourceFactory.java:58)
    at local.ristretto.persistence.datasource.mail.MailDataSourceConfiguration.dataSource(MailDataSourceConfiguration.java:31)
    at local.ristretto.persistence.datasource.mail.MailDataSourceConfiguration$$EnhancerBySpringCGLIB$$497970dd.CGLIB$dataSource$0()
    at local.ristretto.persistence.datasource.mail.MailDataSourceConfiguration$$EnhancerBySpringCGLIB$$497970dd$$FastClassBySpringCGLIB$$2ba2dde9.invoke()
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:309)
    at local.ristretto.persistence.datasource.mail.MailDataSourceConfiguration$$EnhancerBySpringCGLIB$$497970dd.dataSource()
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1120)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1044)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:942)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:813)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:464)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:956)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:747)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:434)
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:798)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:530)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:771)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:342)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1368)
    at org.eclipse.jetty.maven.plugin.JettyWebAppContext.startWebapp(JettyWebAppContext.java:320)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1335)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:735)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:511)
    at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart(JettyWebAppContext.java:403)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:405)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:372)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty(AbstractJettyMojo.java:457)
    at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute(AbstractJettyMojo.java:328)
    at org.eclipse.jetty.maven.plugin.JettyRunMojo.execute(JettyRunMojo.java:170)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

[INFO] |  +- org.springframework.data:spring-data-jpa:jar:1.8.2.RELEASE:compile               
[INFO] |  |  +- org.springframework.data:spring-data-commons:jar:1.10.2.RELEASE:compile       
[INFO] |  |  +- org.springframework:spring-orm:jar:4.1.7.RELEASE:compile                      
[INFO] |  |  |  \- org.springframework:spring-jdbc:jar:4.1.7.RELEASE:compile                  
[INFO] |  |  +- org.springframework:spring-tx:jar:4.1.7.RELEASE:compile                       
[INFO] |  |  \- org.aspectj:aspectjrt:jar:1.8.6:compile                                       
[INFO] |  +- org.postgresql:postgresql:jar:9.4-1202-jdbc42:compile                            
[INFO] |  +- org.apache.tomcat:tomcat-jdbc:jar:8.0.9:compile                                 
[INFO] |  |  \- org.apache.tomcat:tomcat-juli:jar:8.0.9:compile                              
[INFO] |  +- org.hibernate:hibernate-entitymanager:jar:5.0.1.Final:compile                    
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.0.Final:compile                       
[INFO] |  |  +- org.hibernate:hibernate-core:jar:5.0.1.Final:compile                          
[INFO] |  |  |  +- antlr:antlr:jar:2.7.7:compile                                              
[INFO] |  |  |  \- org.jboss:jandex:jar:1.2.2.Final:compile                                   
[INFO] |  |  +- dom4j:dom4j:jar:1.6.1:compile                                                 
[INFO] |  |  |  \- xml-apis:xml-apis:jar:1.0.b2:compile                                       
[INFO] |  |  +- org.hibernate.common:hibernate-commons-annotations:jar:5.0.0.Final:compile    
[INFO] |  |  +- org.hibernate.javax.persistence:hibernate-jpa-2.1-api:jar:1.0.0.Final:compile 
[INFO] |  |  +- org.apache.geronimo.specs:geronimo-jta_1.1_spec:jar:1.1.1:compile             
[INFO] |  |  \- org.javassist:javassist:jar:3.18.1-GA:compile                                 
[INFO] |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.6.1:compile                    
[INFO] |     +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.1:compile              
[INFO] |     \- com.fasterxml.jackson.core:jackson-core:jar:2.6.1:compile                     

[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.12:compile                                      
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.12:compile                                        
[INFO] |  +- org.slf4j:log4j-over-slf4j:jar:1.7.12:compile                                    
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.12:compile                                           
[INFO] |  +- ch.qos.logback:logback-classic:jar:1.1.3:compile                                 
[INFO] |  |  \- ch.qos.logback:logback-core:jar:1.1.3:compile                                 
[INFO] |  +- org.wicketstuff:wicketstuff-logback:jar:6.20.0:compile                           
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.4:compile                                 
[INFO] |  +- org.apache.commons:commons-collections4:jar:4.0:compile                          
[INFO] |  +- commons-io:commons-io:jar:2.4:compile                                            
[INFO] |  +- commons-codec:commons-codec:jar:1.10:compile                                     
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.2:compile                            
[INFO] |  |  \- commons-collections:commons-collections:jar:3.2.1:compile                     
[INFO] |  \- com.google.guava:guava:jar:18.0:compile                                          
[INFO] +- junit:junit:jar:4.12:test                                                           
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test                                          
[INFO] +- org.springframework:spring-test:jar:4.1.7.RELEASE:test                              
[INFO] |  \- org.springframework:spring-core:jar:4.1.7.RELEASE:compile                        
[INFO] +- org.springframework:spring-web:jar:4.1.7.RELEASE:compile                            
[INFO] |  +- org.springframework:spring-aop:jar:4.1.7.RELEASE:compile                         
[INFO] |  |  \- aopalliance:aopalliance:jar:1.0:compile                                       
[INFO] |  +- org.springframework:spring-beans:jar:4.1.7.RELEASE:compile                       
[INFO] |  \- org.springframework:spring-context:jar:4.1.7.RELEASE:compile                     
[INFO] |     \- org.springframework:spring-expression:jar:4.1.7.RELEASE:compile               
[INFO] \- javax:javaee-web-api:jar:7.0:provided

Answer 1

It appears, that this is caused by Jetty using a modified Apache JSP package Jasper. It has its own version of Juli logging (check lib/apache-jsp/org.mortbay.jasper.apache-jsp-8.0.27.jar under Jetty home directory) which seems to cause all the confusion. I think, that Jetty's default webapp class loading logic causes Juli LogFactory from your webapp to be loaded everytime that class is requested. It's all good and dandy when that LogFactory has to load tomcat-jdbc classes. It gets tricky when it gets passed Jetty's modified Jasper classes, because it realises that there are type conflicts.

How you can solve this? I think there are 2 ways to do so:

1. Change Jetty webapp classloader logic

You can modify classloader logic, to always delegate class loading to parent classloader. You just need to call setParentLoaderPriority(true) on particular WebAppContext. You can do that either by Jetty configuration or by including jetty-web.xml file with your war.

2. Exclude tomcat-juli dependency

Using Maven's Dependency exclusions you can skip any transitive dependency you do not like. So just include tomcat-jdbc without tomcat-juli and you're done.

The only problem is that both solutions force you to use Jetty's modified Juli package. Haven't seen a problem with such scenario yet, but I guess at some point you might find you're missing some feature, because Jetty's modified Juli wasn't synced with Apache latest version.

Answer 2

The spring boot starter web dependency will include tomcat by default. This will confuse jetty when you start it, since it uses different versions of the juli lib.

Simply fix in your pom.xml, exlude the starter-tomcat and separately include the jetty dependency:

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-tomcat</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-jetty</artifactId>
    </dependency>