Currently I am working on a Spring Integration application which has a following scenario:
- An
int-http:outbound-gateway read from a REST-Services a list of paginated elements: about in
- Each page content splitted and stored in a folder to be processed later by a spring batch job.
I'm quite new with spring-integration and I don't know if it's possibile to create a kind of loop with `int-http:outbound-gateway' to read all pages until the last one.
We're talking about 66254 elements splitted in 2651 pages. What I'm looking for is a best practice to read and download all pages and collecting data without have any memory issue.
Any suggestion will be appreciated
Thanks
Yes, it is possible, although a bit tricky.
Assume your REST service require page as request param, so, you would like to make a request from the page #1 and loop (increment page param) until the service returns empty result.
So, you may have configuration for the REST service like:
<int-http:outbound-gateway url="http://service/elements?page={page}">
<int-http:uri-variable name="page" expression="headers['page']"/>
</int-http:outbound-gateway>
Pay attention to that <int-http:uri-variable> definition. From the beginning you have to send the message to this <int-http:outbound-gateway> with the page header as a 1.
The reply from this gateway you should send to something like <recipient-list-router>, or <publish-subscribe-channel>, where one of the subscriber is still your splitter to to store items into the folder.
Another subscriber is a bit smart. It starts from <filter> to check if the payload (a result from the REST call) is empty, meaning that we have done and no more pages on the service to retrieve. Otherwise you use <header-enricher> to increment and replace the page header and send the result into that our first <int-http:outbound-gateway>.
