I have following spring configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd">
<bean id="downloadLogger" class="com.thomsonreuters.oa.sdi.camel.DownloadLogger" />
<bean id="fileFilter" class="com.thomsonreuters.oa.sdi.camel.IgnoreReadyFilesFilter" />
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="ftp://url_to_ftp?password=*******&noop=true&stepwise=false&binary=true&consumer.delay=10s&recursive=true&filter=#fileFilter" />
<process ref="downloadLogger" />
<to uri="file:data/outbox" />
</route>
</camelContext>
</beans>
At the ftp side I have 3 folders with files which I want to download. I want to achieve following scenario:
- On ftp is fixed amount of files (for isntance 5) at the first data pull consumer loads these files to the destination folder
- At the second attempt to load files, ftp state still the same (5 files) and camel ftp consumer just does nothing (except check for new files)
- To ftp arrives new 2 files, and at this data pull consumer downloads only these new two files
At the moment my current solutions downloads all files each time when I run dataload process, how I can manage information about downloaded files to prevent downloads of duplicates (I mean already copied files from ftp), I can write my own filter which will filter out already downloaded files but I belive there should be built in feature which will give me controle of this (maybe idempotentRepository, actually I am not sure)...
Maybe @endryha answer work well in 2011, but not with Camel 2.20.1
In Camel 2.20.1, these code will create two idempotentRepository
So the correct way to use idempotentRepository is (I remove most parameter for readability)
and a Bean
Finally I end up with following solution:
You need to use a persistent idempotent repository if you want Camel to be able to remember which files it previously have downloaded, between restarts.
You need to set this option on the ftp endpoint: idempotentRepository
See more details here: http://camel.apache.org/file2 (Note: The FTP component inherits the options from the file component.)
There are some examples on the wiki page how to use different stores. And you can also build you custom store.