Spring and Hibernate suddenly set the transaction

Question

We have an application running on JBoss 4.2.3, using Spring 2.5.2 and Hibernate 3.2.6.ga. This is running on Linux JEE01 2.6.16.60-0.54.5-smp, using its own user. Writing to a Oracle 10G database on another machine.

We're using a standard view -> service -> dao layering. Where each dao is annotated with @Repository.

This is all running 24/7 without many problems, but every several days and sometimes a couple of times in one day the whole system goes into a bad state where nothing can be written to the database anymore. These stacktraces appear in the logs:

org.springframework.dao.InvalidDataAccessApiUsageException: Write operations are not
      allowed in read-only mode (FlushMode.NEVER/MANUAL): 
Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction
      definition.

We scanned the complete system, and there is one place in the system where the flushmode is temporarely set to MANUAL after which a finally block set its back to its original value. This is because we do not want to flush the state to the database before this query runs. So we can't change this very easily. The normal FlushMode is set to AUTO and on several places we temporarily set it to COMMIT and switching it back to the default again.

Only a server restart restores the system back to working order.

The question is: why does the system set all transactions to readonly/manual flush mode? I googled this but could not find a solution.

This is our spring and hibernate configuration (only relevants part showing):

     <bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
  <property name="dataSource">
   <ref bean="datasourceName" />
  </property>
  <property name="configLocation">
   <value>classpath:hibernate.cfg.xml</value>
  </property>
 </bean>

 <bean id="hibernateInterceptor"
  class="org.springframework.orm.hibernate3.HibernateInterceptor">
  <property name="sessionFactory" ref="sessionFactory" />
 </bean>

 <tx:annotation-driven transaction-manager="txManager" /> 

 <tx:advice id="txAdvice" transaction-manager="txManager" >
  <!-- the transactional semantics... -->
  <tx:attributes >
   <!-- all methods starting with 'get' are read-only -->
   <tx:method name="approve*" read-only="false"
    propagation="REQUIRED" rollback-for="java.lang.Exception" />
   <tx:method name="update*" read-only="false"   
    propagation="REQUIRED" rollback-for="java.lang.Exception"/>
   <tx:method name="save*" read-only="false"
    propagation="REQUIRED" rollback-for="java.lang.Exception"/>
            <tx:method name="delete*" read-only="false"
    propagation="REQUIRED" rollback-for="java.lang.Exception" />
   <!-- other methods use the default transaction settings (see below) -->
   <tx:method name="*" read-only="true" propagation="REQUIRED" />
  </tx:attributes>
 </tx:advice>

 <aop:config>
  <aop:pointcut id="serviceMethods"
   expression="execution(* com.myapplication.service.*.*(..))"  />
  <aop:advisor advice-ref="txAdvice" 
   pointcut-ref="serviceMethods" />
 </aop:config>

 <bean id="txManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager" >
  <property name="sessionFactory" ref="sessionFactory"  />
 </bean>

-- end of spring config --

-- hibernate configuation --
<hibernate-configuration>
    <session-factory name="">
        <property name="dialect">org.hibernate.dialect.Oracle10gDialect</property>
        <property name="show_sql">false</property>
        <property name="use_outer_join">false</property>
        <property name="hibernate.cache.use_query_cache">true</property>
        <property name="hibernate.cache.use_second_level_cache">true</property>
        <property name="cache.provider_class">org.hibernate.cache.EhCacheProvider</property>
        <property name="hibernate.connection.SetBigStringTryClob">true</property>
        <property name="hibernate.jdbc.batch_size">0</property>
    </session-factory>
    <mapping ----/>
</hibernate-configuration>

This is the stacktrace:

org.springframework.dao.InvalidDataAccessApiUsageException: Write operations are not allowed in read-only mode (FlushMode.NEVER/MANUAL): Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction definition.
at org.springframework.orm.hibernate3.HibernateTemplate.checkWriteOperationAllowed(HibernateTemplate.java:1137)
at org.springframework.orm.hibernate3.HibernateTemplate$16.doInHibernate(HibernateTemplate.java:701)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:699)
at nl.company.myapp.dao.impl.GenericDAOImpl.save(GenericDAOImpl.java:94)
at nl.company.myapp.dao.impl.CallDAOImpl.save(CallDAOImpl.java:266)
at nl.company.myapp.dao.impl.CallDAOImpl.save(CallDAOImpl.java:47)
at nl.company.myapp.service.impl.CallServiceImpl.saveCall(CallServiceImpl.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy142.saveCall(Unknown Source)
at nl.company.myapp.view.bean.call.CallDetailBean.doSave(CallDetailBean.java:319)
at nl.company.myapp.view.bean.EditModeAwareBean.save(EditModeAwareBean.java:151)
at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.apache.el.parser.AstValue.invoke(AstValue.java:131)
at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
at com.sun.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:68)
at org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
at org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:190)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:458)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:763)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at nl.company.myapp.view.audit.AuditFilter.doFilter(AuditFilter.java:88)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._invokeDoFilter(TrinidadFilterImpl.java:238)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:195)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:138)
at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:595)

This all works fine

Answer 1

My guess would be that it is Spring that does this for you. I seem to remember this being done by the OpenSessionInViewFilter in Spring. Are you using that?

Answer 2

aseesing, i check your configuration. in spring context part, you use

<!-- other methods use the default transaction settings (see below) -->
   <tx:method name="*" read-only="true" propagation="REQUIRED" />

usually, only some accesses are read-only types, such as get/find/query, etc. i suggest use

<!--default is required transaction-->
<tx:method name="*"/>

another thing is, do you use opensessioninview pattern? flush mode could be set in opensessioninview properly. you can use filter in web.xml or use spring interceptor in application context config.

Answer 3

This is from memory so it's not very detailed. There are two ways of setting the flushmode. I think a Spring way and a Hibernate way. Because we were using the (Spring?) way only once and the Hibernate way all the other times it made the internal state of Hibernate wrong somehow. Which created my problems. When we made everything consistent the problem went away.

Answer 4

Because you are using an execution pointcut on the implementation and you are utilizing a generic DAO, perhaps this is not handling transactions as you would expect and this is just a manifestation of the real issue.

Please ensure that Spring is proxying your DAO as you expect on that call. You may need to use the target(beanid) syntax to enable the proper transactions for your generic DAO.

Answer 5

in your web.xml put :

<filter>
    <filter-name>openSessionInViewFilter</filter-name>
    <filter-class>org.springframework.orm.hibernate3.support.OpenSessionInViewFilter</filter-class>
    <init-param>
        <param-name>flushMode</param-name>
        <param-value>AUTO</param-value>
    </init-param>
</filter>

Answer 6

This exception comes from the following code in Spring's HibernateTemplate class:

protected void checkWriteOperationAllowed(Session session) throws InvalidDataAccessApiUsageException {
    if (isCheckWriteOperations() && getFlushMode() != FLUSH_EAGER &&
            session.getFlushMode().lessThan(FlushMode.COMMIT)) {
        throw new InvalidDataAccessApiUsageException(
                "Write operations are not allowed in read-only mode (FlushMode.NEVER/MANUAL): "+
                "Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction definition.");
    }
}

The rationale for this check is explained as:

This is a new consistency check introduced in Spring 1.1.

Invoking HibernateTemplate's save/update/delete methods on a Spring-managed Session in FlushMode.NEVER is potentially dangerous: It means that you are:

• either doing this in a Spring-managed read-only transaction, which will never flush the Hibernate Session, i.e. never flush your save/update/delete calls. The new check makes you aware that you won't persist your changes in that situation.

• or working with some other thread-bound Session in FlushMode.NEVER, for example the OpenSessionInViewFilter. If you're overriding closeSession there to flush after view rendering, you should override getSession too, setting the Session to FlushMode.AUTO.

I strongly recommend against the latter, though. If you're using OpenSessionInViewFilter, combine it with middle tier transactions rather than let the filter flush at request completion.

Does any of this ring a bell?

It is possible that there is some bug in the your code or in Spring ORM. To disable this check you can call setCheckWriteOperations(false).