Spring Batch的 - JdbcCursorItemReader抛出的OutOfMemor

我写使用Spring Batch的处理从MySQL数据库表7637064行的程序。 我已经与小表的成功，但该表中的大量行导致的OutOfMemoryError异常时JdbcCursorItemReader试图打开光标。

我可以通过它抛出一个更大的XMX可能解决这个问题，但在我看来，Spring Batch的应该有一种方法来处理这一点，我可能只是缺少配置的一个关键部分。

Spring Batch的配置：

  <job id="reportJob" xmlns="http://www.springframework.org/schema/batch">
    <step id="largeTableTransfer">
      <tasklet>
        <chunk reader="largeTableReader" processor="largeTableTransformer" writer="largeTableWriter"
          commit-interval="10" />
      </tasklet>
    </step>
  </job>

  <bean id="largeTableReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
    <property name="dataSource" ref="inputDataSource" />
    <property name="sql" value="select * from largeTable" />
    <property name="rowMapper">
      <bean class="myproject.reader.largeTableRowMapper" />
    </property>
  </bean>
  <bean id="largeTableTransformer" class="myproject.transformer.LargeTableTransformer" />
  <bean id="largeTableWriter" class="myproject.writer.JdbcLargeTableWriter">
    <property name="dataSource" ref="outputDataSource" />
  </bean>

在JdbcCursorItemReader设置FETCHSIZE似乎没有任何效果。 这允许它运行到完成的唯一事情是中的maxRows设置为一个小数目，但随后只行的这一数字得到处理。

相关的堆栈跟踪：

2012-11-21 11:25:29,931 DEBUG [org.springframework.batch.core.repository.dao.JdbcStepExecutionDao] - <Truncating long message before update of StepExecution, original message is: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2734)
    at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
    at java.util.ArrayList.add(ArrayList.java:351)
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2821)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
    at org.springframework.batch.item.database.JdbcCursorItemReader.openCursor(JdbcCursorItemReader.java:125)
    at org.springframework.batch.item.database.AbstractCursorItemReader.doOpen(AbstractCursorItemReader.java:401)
    at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:134)
    at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:93)
    at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:301)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:192)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)
    at ifpress.ams2amx.ExampleJobConfigurationTests.testLaunchJob(ExampleJobConfigurationTests.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Answer 1:

有一个在MySQL JDBC驱动程序的问题，这将导致整个数据集到内存中加载不管已传递到语句创建方法的参数。 见http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-implementation-notes.html如何正确地打开游标。

这是我如何做到这一点：

<bean class="org.springframework.batch.item.database.JdbcCursorItemReader">
  <property name="verifyCursorPosition" value="false" />
   <property name="dataSource" ref="remoteDataSource" />
   <property name="rowMapper">
     <bean class="org.springframework.jdbc.core.SingleColumnRowMapper" />
   </property>
   <property name="fetchSize">
     <util:constant static-field="java.lang.Integer.MIN_VALUE" />
   </property>
   <property name="sql">
     <value>SELECT foo, bar FROM baz</value>
  </property>
</bean>

文章来源: Spring Batch - JdbcCursorItemReader throwing OutOfMemoryError with large MySQL table