I've created: https://jira.spring.io/browse/BATCH-2778
I am developing Spring Batch + Redis (Spring Data Redis) example. In this example, I'm reading student.csv
file and storing all the data as is in Redis DB. I wanted to used dateOfBirth
as Date
and I am sure that I need to do some date logic conversion to store value Date in Redis
.
As per my analysis, it looks like I wont be able to use @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd")
, because I am not dealing with JSON at all. I think I should be using the Converter etc to convert the Byte[]
to Date
and Date
to Byte[]
etc.
Infact I tried these options, but doesn't working. Any quick help?
Note: I don't want to save dateOfBirth as String value.
org.springframework.batch.item.file.FlatFileParseException: Parsing error at line: 2 in resource=[URL [file:c:/Videos/student.csv]], input=[1,John,Doe,05-12-1988 12:34:45]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:184) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.read(AbstractItemCountingItemStreamItemReader.java:89) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader$$FastClassBySpringCGLIB$$ebb633d0.invoke(<generated>) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader$$EnhancerBySpringCGLIB$$87932856.read(<generated>) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:91) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.read(SimpleChunkProvider.java:157) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:116) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:110) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:69) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:394) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_162]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_162]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at com.sun.proxy.$Proxy69.run(Unknown Source) [na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:792) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1242) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1230) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at com.myexample.DateBatchPocApplication.main(DateBatchPocApplication.java:12) [classes/:na]
Caused by: org.springframework.validation.BindException: org.springframework.validation.BeanPropertyBindingResult: 1 errors
Field error in object 'target' on field 'dateOfBirth': rejected value [05-12-1988 12:34:45]; codes [typeMismatch.target.dateOfBirth,typeMismatch.dateOfBirth,typeMismatch.java.util.Date,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [target.dateOfBirth,dateOfBirth]; arguments []; default message [dateOfBirth]]; default message [Failed to convert property value of type 'java.lang.String' to required type 'java.util.Date' for property 'dateOfBirth'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'java.lang.String' to required type 'java.util.Date' for property 'dateOfBirth': no matching editors or conversion strategy found]
at org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper.mapFieldSet(BeanWrapperFieldSetMapper.java:200) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:43) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:180) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
... 56 common frames omitted
StudentJob.java
@Configuration
public class StudentJob {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private FlatFileItemReader<Student> studentReader;
@Autowired
private StudentWritter studentWritter;
@Bean
public Job readstudentCSVFileJob() {
return jobBuilderFactory.get("readstudentCSVFileJob").incrementer(new RunIdIncrementer())
.start(countryCurrStepOne()).build();
}
@Bean
public Step countryCurrStepOne() {
return stepBuilderFactory.get("studentStepOne").<Student, Student>chunk(5).reader(studentReader)
.writer(studentWritter).build();
}
}
StudentBatchConfig.java
@Slf4j
@Configuration
public class StudentBatchConfig {
@Bean(destroyMethod="")
@StepScope
public FlatFileItemReader<Student> studentReader(@Value("${input.student.path}") Resource resource) throws IOException {
log.debug("Resource Path : "+resource.getFile());
FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>();
itemReader.setName("STUDENT_READER");
itemReader.setResource(resource);
itemReader.setLineMapper(studentlineMapper());
itemReader.setLinesToSkip(1);
return itemReader;
}
@Bean
public LineMapper<Student> studentlineMapper() {
// Delimited Line Tokenizer
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setNames("id", "firstName", "lastName", "dateOfBirth");
lineTokenizer.setIncludedFields(new int[] {0,1,2,3});
// Bean Wrapper Field SetMapper
BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(Student.class);
DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(fieldSetMapper);
return lineMapper;
}
@Bean
public StudentWritter studentWritter() {
return new StudentWritter();
}
}
StudentWritter.java
public class StudentWritter implements ItemWriter<Student>{
@Autowired
private StudentRepository studentRepository;
@Override
public void write(List<? extends Student> students) throws Exception {
studentRepository.saveAll(students);
}
}
EDIT-1:
@Slf4j
@Configuration
public class StudentBatchConfig {
private static final SimpleDateFormat format = new SimpleDateFormat("dd-MM-yyyy HH:mm");
@Bean(destroyMethod="")
@StepScope
public FlatFileItemReader<Student> countryReader(@Value("${input.student.path}") Resource resource) throws IOException {
log.debug("Resource Path : "+resource.getFile());
FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>();
itemReader.setName("STUDENT_READER");
itemReader.setResource(resource);
itemReader.setLineMapper(studentLineMapper());
itemReader.setLinesToSkip(1);
return itemReader;
}
@SuppressWarnings("rawtypes")
@Bean
public LineMapper<Student> studentLineMapper() {
DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>();
// Delimited Line Tokenizer
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setNames("id", "firstName", "lastName", "dateOfBirth");
lineTokenizer.setIncludedFields(new int[] {0,1,2,3});
// Date parsing logic has been added
CustomDateEditor customDateEditor = new CustomDateEditor(format, false);
HashMap<Class, PropertyEditor> customEditors = new HashMap<>();
customEditors.put(Date.class, customDateEditor);
// Bean Wrapper Field SetMapper
BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(Student.class);
fieldSetMapper.setCustomEditors(customEditors);
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(fieldSetMapper);
return lineMapper;
}
@Bean
public StudentWritter studentWritter() {
return new StudentWritter();
}
}
Student.java
@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
@RedisHash("student")
public class Student {
@Id @Indexed
private String id;
private String firstName;
private String lastName;
private Date dateOfBirth;
}
But dates are not storing correctly.. seem like formatting issue..
NOTE: If I open this file in CSV editor of Microsoft it shows me dd-MM-yyyy HH:mm
But If I edit this file in Notepad++, then it shows
id,firstName,lastName,dateOfBirth
1,John,Doe,25-11-2018 14:48:10
2,Sameer,Kumbhare,25-11-2018 14:48:10
Answrer posted here: https://jira.spring.io/browse/BATCH-2778
or
I used something like below. Looks like Redis doesn't worked well with the Java7 Date API, but it works well with the Java 8 API. But there is no out of the box support for the same.
Your issue is not related to JSON, the problem according to the stacktrace is that the
BeanWrapperFieldSetMapper
does not know how to convert05-12-1988 12:34:45
to ajava.util.Date
. You need to configure it with custom date editor like this:CustomDateEditor
is from Spring Framework:org.springframework.beans.propertyeditors.CustomDateEditor
.Hope this helps.