I have a application running in AWS BeanStalk (Tomcat8 + SpringMVC + PostgreSQL + EclipseLink).
The application is running ok but after some time running (2 hours) one error starting to happen:
The intercepted error is:
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: Exception [EclipseLink-28013] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException Exception Description: Unable to deploy PersistenceUnit ###### in invalid state [DeployFailed]. Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException Exception Description: Deployment of PersistenceUnit ###### failed. Close all factories for this PersistenceUnit. Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException Exception Description: Attempted to redeploy a session named file: ####### without closing it.
The basic error is:
Attempted to redeploy a session without closing it.
The complete stack error is:
[EL Severe]: ejb: 2015-03-25 13:30:25.813--ServerSession(490254137)--Thread(Thread[http-nio-8080-exec-16,5,main])--javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Unable to deploy PersistenceUnit [XpHubPU] in invalid state [DeployFailed].
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit [XpHubPU] failed. Close all factories for this PersistenceUnit.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file: #################################### without closing it.
19:58:09,271 DEBUG [DefaultListableBeanFactory] Returning cached instance of singleton bean 'globalDefaultExceptionHandler'
19:58:09,271 DEBUG [ExceptionHandlerExceptionResolver] Invoking @ExceptionHandler method: public org.springframework.web.servlet.ModelAndView com.#########.common.controller.GlobalDefaultExceptionHandler.defaultErrorHandler(javax.servlet.http.HttpServletRequest,java.lang.Exception) throws java.lang.Exception
19:58:09,271 DEBUG [DispatcherServlet ] Handler execution resulted in exception - forwarding to resolved error view: ModelAndView: reference to view with name 'util/error'; model is {exception=org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: Exception [EclipseLink-28013] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Unable to deploy PersistenceUnit [XpHubPU] in invalid state [DeployFailed].
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit #################### failed. Close all factories for this PersistenceUnit.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file: ############################### without closing it., url=###################}
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: Exception [EclipseLink-28013] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Unable to deploy PersistenceUnit [XpHubPU] in invalid state [DeployFailed].
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit [XpHubPU] failed. Close all factories for this PersistenceUnit.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:######################################### without closing it.
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:457)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy85.storeIsActive(Unknown Source)
at com.xphub.modules.hub.controller.HubStoreController.storeIsActive(HubStoreController.java:483)
at com.xphub.modules.hub.controller.HubStoreController.home(HubStoreController.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:673)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1085)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1556)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1513)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.persistence.PersistenceException: Exception [EclipseLink-28013] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Unable to deploy PersistenceUnit ############## in invalid state [DeployFailed].
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit ################# failed. Close all factories for this PersistenceUnit.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:###################################### without closing it.
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:547)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:204)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:304)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:336)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:302)
at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:449)
at com.xphub.config.MultiTenantJpaTransactionManager.createEntityManagerForTransaction(MultiTenantJpaTransactionManager.java:55)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:369)
... 84 more
Caused by: Exception [EclipseLink-28013] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Unable to deploy PersistenceUnit [XpHubPU] in invalid state [DeployFailed].
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit [XpHubPU] failed. Close all factories for this PersistenceUnit.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:####################### without closing it.
at org.eclipse.persistence.exceptions.EntityManagerSetupException.cannotDeployWithoutPredeploy(EntityManagerSetupException.java:190)
... 92 more
I search for answers and found few results about this error and all information is old, 2008 and older like that: https://bugs.eclipse.org/bugs/show_bug.cgi?id=256284
I am using:
- Eclipselink - org.eclipse.persistence.jpa - 2.5.2
- Spring MVC - many features - 4.1.4.RELEASE
- PostgreSQL - org.postgresql - 9.3-1100-jdbc41
And using a extending JpaTransactionManager class to tenancy that is:
public class MultiTenantJpaTransactionManager extends JpaTransactionManager {
protected static final Logger log = Logger.getLogger(MultiTenantJpaTransactionManager.class);
@Autowired
CurrentTenantResolverService currentTenantResolverService;
/**
* NOTE:
* Maybe it would be also possible to replace existing entityManager in transaction with new but it
* isn't a good idea because of rollback and other problems.
* So when new tenant is required always start new transaction for it.
*
*/
@Override
protected EntityManager createEntityManagerForTransaction() {
EntityManager em = super.createEntityManagerForTransaction();
boolean refreshed = false;
ServerSession ss = ((EntityManagerImpl) em.getDelegate()).getServerSession();
Map sessionProp = ss.getProperties();
// don't run it if tenant didn't change
// it should be quite faster then
String OrgId = currentTenantResolverService.getCurrentTenantId();
// set new tenant as property for actual session
// while refreshing metadata it will be used from actual session for new session
sessionProp.put(PersistenceUnitProperties.MULTITENANT_PROPERTY_DEFAULT, OrgId);
MetadataRefreshListener mrl = ((EntityManagerImpl) em.getDelegate()).getServerSession().getRefreshMetadataListener();
// metadata refresh listener is empty if it was already run for actual transaction (same entity manager)
// because it is placed in createEntityManagerForTransaction now this shouldn't happen but to be sure...
if (mrl != null) {
Map<String, Object> prop = new HashMap<String, Object>();
// metadata will be refreshed for next created entity manager
mrl.triggerMetadataRefresh(prop);
refreshed = true;
}
// if metadata for "old" entity manager wasn't refreshed we don't need to create a new one
return refreshed ? super.createEntityManagerForTransaction() : em;
}
I was trying everithing to solve this problem, but don't found any approach.
Anyone here have an idea about this situation?
Update 1
After a several tests, I could simulate the problem using apache AB to generate concurrency and some requests to my application.
No other loggin information showed.
Running the same load test without extending JpaTransactionManager with class MultiTenantJpaTransactionManager no problem happen.
I think the problem is with the persistence unit name or something about that. Even stopping the requests, after cause the problem, the error still there until restart server.