JBoss 4.2 startup throws NoClassDefFoundError

I am having trouble starting a JBoss 4.2.2 server - it is producing the following error message (from org.jboss.kernel.plugins.dependency.AbstractKernelController) :

Error installing to Described: name=WSDeployerHook_JAXWS_EJB3 state=Not Installed mode=Manual requiredState=Create
java.lang.NoClassDefFoundError: org/jboss/ejb3/interceptors/direct/IndirectContainer

Looking through the JBoss source code, I have turned on TRACE level logging for certain classes, though all it is telling me is that it can't find the IndirectContainer (these lines all come from org.jboss.mx.loading.LoadMgr3):

Begin beginLoadTask, task=org.jboss.mx.loading.ClassLoadingTask@c8aeb3{classname: org.jboss.ejb3.interceptors.direct.IndirectContainer, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@16925b0{ url=null ,addedOrder=2}, loadedClass: nullnull, loadOrder: 2147483647, loadException: null, threadTaskCount: 0, state: 0, #CCE: 0}
End beginLoadTask, ClassNotFoundException
Run failed with exception, loadTask=org.jboss.mx.loading.ClassLoadingTask@10e468f{classname: org.jboss.ejb3.EJBContainer, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@16925b0{ url=null ,addedOrder=2}, loadedClass: nullnull, loadOrder: 2147483647, loadException: java.lang.NoClassDefFoundError: org/jboss/ejb3/interceptors/direct/IndirectContainer, threadTaskCount: 0, state: 1, #CCE: 1}
Notifying task of thread completion, loadTask:org.jboss.mx.loading.ClassLoadingTask@10e468f{classname: org.jboss.ejb3.EJBContainer, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@16925b0{ url=null ,addedOrder=2}, loadedClass: nullnull, loadOrder: 2147483647, loadException: java.lang.NoClassDefFoundError: org/jboss/ejb3/interceptors/direct/IndirectContainer, threadTaskCount: 0, state: 1, #CCE: 1}
End nextTask(0), loadTask=org.jboss.mx.loading.ClassLoadingTask@10e468f{classname: org.jboss.ejb3.EJBContainer, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@16925b0{ url=null ,addedOrder=2}, loadedClass: nullnull, loadOrder: 2147483647, loadException: java.lang.NoClassDefFoundError: org/jboss/ejb3/interceptors/direct/IndirectContainer, threadTaskCount: 0, state: 4, #CCE: 1}
Begin endLoadTask, task=org.jboss.mx.loading.ClassLoadingTask@10e468f{classname: org.jboss.ejb3.EJBContainer, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@16925b0{ url=null ,addedOrder=2}, loadedClass: nullnull, loadOrder: 2147483647, loadException: java.lang.NoClassDefFoundError: org/jboss/ejb3/interceptors/direct/IndirectContainer, threadTaskCount: 0, state: 4, #CCE: 1}
registerLoaderThread, ucl=org.jboss.mx.loading.UnifiedClassLoader3@56182f{ url=file:/var/data/applications/jboss/server/jbossgpa1/deploy/jboss-bean.deployer/ ,addedOrder=5}, t=Thread[main,5,jboss], prevT=null
Begin beginLoadTask, task=org.jboss.mx.loading.ClassLoadingTask@159d10{classname: org.jboss.util.JBossStringBuilder, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@56182f{ url=file:/var/data/applications/jboss/server/jbossgpa1/deploy/jboss-bean.deployer/ ,addedOrder=5}, loadedClass: nullnull, loadOrder: 2147483647, loadException: null, threadTaskCount: 0, state: 0, #CCE: 0}
scheduleTask(1), created subtask: {t=Thread[main,5,jboss], ucl=org.jboss.mx.loading.UnifiedClassLoader3@56182f{ url=file:/var/data/applications/jboss/server/jbossgpa1/deploy/jboss-bean.deployer/ ,addedOrder=5}, name=org.jboss.util.JBossStringBuilder, requestingThread=Thread[main,5,jboss], order=5, releaseInNextTask=false}

I have used TextPad to search the whole folder structure including the .jars etc and there is not a single mention of IndirectContainer outside of the log files.

I'm guessing that a library's dependency is not being satisfied, but I am unsure what the next step should be to diagnose this problem. Any ideas/suggestions?

(running on RHEL4)

I'm running JBoss (MX MicroKernel) [4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)]

I started getting a bit desparate, so I connected the debugger and created a conditional breakpoint ("org.jboss.ejb3.interceptors.direct.IndirectContainer".equals(classname)) on the constructor of org.jboss.mx.loading.ClassLoadingTask. When the breakpoint was hit, I went back down the stack to find the earliest mention of IndirectContainer:

On the left hand side you can see (I hope - it's not looking very clear on my small screen) that below the call to UnifiedClassloader3.classLoadInternal() is a native call (which I can't see any values of), and then there is a call to UnifiedClassloader3.defineClass():

In this call, you can see highlighted (in blue, bottom right) jboss-ejb3-core-client.jar - I don't work with Classloaders much, but I am assuming that the decision to instantiate a ClassLoadingTask for IndirectContainer was taken because of the contents of this .jar file.

As far as I can see there is no mention of IndirectContainer in this .jar.

Hopefully that's enough detail for someone to recognise the problem.

I found the solution after a lot of debugging and checking of source code.

Our appserver's classpath included jboss-ejb3-core-client.jar from JBoss 5.1.0.GA, so that it could talk to EJBs exposed on JBoss5.1.0 appservers. There is also jboss-ejb3-client.jar in our classpath. Both contain the class org.jboss.ejb3.EJBContainer but only the class in the first .jar implements org.jboss.ejb3.interceptors.direct.IndirectContainer, which we didn't have in our classpath.

In my original question, I wrote:

As far as I can see there is no mention of IndirectContainer in this .jar.

...which was obviously wrong.

Therefore, I have written another question asking if there are any tools that could have identified this much more quickly. If there aren't, I'll write one.