Read and download from a paginated REST-Services w

Question

Currently I am working on a Spring Integration application which has a following scenario:

1. An int-http:outbound-gateway read from a REST-Services a list of paginated elements: about in
2. Each page content splitted and stored in a folder to be processed later by a spring batch job.

I'm quite new with spring-integration and I don't know if it's possibile to create a kind of loop with `int-http:outbound-gateway' to read all pages until the last one.

We're talking about 66254 elements splitted in 2651 pages. What I'm looking for is a best practice to read and download all pages and collecting data without have any memory issue.

Any suggestion will be appreciated

Thanks

Answer 1

Yes, it is possible, although a bit tricky.

Assume your REST service require page as request param, so, you would like to make a request from the page #1 and loop (increment page param) until the service returns empty result.

So, you may have configuration for the REST service like:

<int-http:outbound-gateway url="http://service/elements?page={page}">
    <int-http:uri-variable name="page" expression="headers['page']"/>
</int-http:outbound-gateway>

Pay attention to that <int-http:uri-variable> definition. From the beginning you have to send the message to this <int-http:outbound-gateway> with the page header as a 1.

The reply from this gateway you should send to something like <recipient-list-router>, or <publish-subscribe-channel>, where one of the subscriber is still your splitter to to store items into the folder.

Another subscriber is a bit smart. It starts from <filter> to check if the payload (a result from the REST call) is empty, meaning that we have done and no more pages on the service to retrieve. Otherwise you use <header-enricher> to increment and replace the page header and send the result into that our first <int-http:outbound-gateway>.