Prevent duplicates across restarts in spring integ

Question

I have to poll a directory and write entries to rdbms. I wired up a redis metadatstore for duplicates check. I see that the framework updates the redis store with entries for all files in the folder [~ 140 files], much before the rdbms entries gets written. At the time of application termination, rdbms has logged only 90 files. On application restart no more files are picked from folder.

Properties: msgs.per.poll=10, polling.interval=2000 How can I ensure entries to redis are made after writing to db, so that both are in sync and I don't miss any files.

<code>
    <task:executor id="executor" pool-size="5" />
    <int-file:inbound-channel-adapter channel="filesIn" directory="${input.Dir}" scanner="dirScanner"    filter="compositeFileFilter" prevent-duplicates="true">
        <int:poller fixed-delay="${polling.interval}" max-messages-per-poll="${msgs.per.poll}" task-executor="executor">
        </int:poller>
    </int-file:inbound-channel-adapter>
    <int:channel id="filesIn" />

    <bean id="dirScanner" class="org.springframework.integration.file.RecursiveLeafOnlyDirectoryScanner" />

    <bean id="compositeFileFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
        <constructor-arg ref="persistentFilter" />
    </bean>

    <bean id="persistentFilter" class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
        <constructor-arg ref="metadataStore" />
    </bean>

    <bean name="metadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
        <constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
    </bean>
    <bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory" p:hostName="localhost" p:port="6379" />

    <int-jdbc:outbound-channel-adapter channel="filesIn" data-source="dataSource" query="insert into files values (:path,:name,:size,:crDT,:mdDT,:id)"
        sql-parameter-source-factory="spelSource">
    </int-jdbc:outbound-channel-adapter>

        ....
</code>

Answer 1

How can I ensure entries to redis are made after writing to db

It's isn't possible, because FileSystemPersistentAcceptOnceFileListFilter works before any message sending and only once, when FileReadingMessageSource.toBeReceived is empty. Of course, it tries to refetch files on the next application restart, but it can't do that because your RedisMetadataStore already contains entries for those files.

I think we don't have in your case any choice unless use some custom JdbcFileListFilter based on your files table. Fortunately you logic ends up with file entry anyway.

Answer 2

Artem is correct, you might as well extend the RedisMetadataStore and flush the entries that are not in your database on initialization time, this way you could use Redis and be in sync with the DB. But this kind of couples things a little.