Spring RestTemplate streaming response into anothe

Question

I am trying to stream the result of a file download directly into another post using spring's RestTemplate

My current approach is the following:

   ResponseEntity<InputStreamResource> downloadResponse = restTemplate.getForEntity(fileToDownloadUri, InputStreamResource.class);

   InputStreamResource imageInputStreamResource = downloadResponse.getBody();

   ResponseEntity<String> response = restTemplate.exchange(storageUri, POST, new HttpEntity<>(imageInputStreamResource), String.class);

However, I get the following exception running the code above:

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://host:port/path/some.jpg": stream is closed; nested exception is java.io.IOException: stream is closed

    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:6
...
Caused by: java.io.IOException: stream is closed
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.ensureOpen(HttpURLConnection.java:3348)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3373)

It seems that the response is always closed as the final step of processing. With the response, the HttpURLConnection is closed, and the stream is no longer processable.

I would like to be able to implement this scenario without having to hold the file completely in memory or writing it to a file (as described here).

Any hints are highly appreciated.

Answer 1

Since you tell RestTemplate to expect InputStreamResource it will try and use an appropriate converter to convert your message to a InputStreamResource. ( I'm guessing there is none that handles this as you want )

You should be able to let it expect a Resource from where you can get an input stream and read that.

 import org.springframework.core.io.Resource;

 ResponseEntity<Resource> exchange = RestTemplate.exchange(url, HttpMethod.GET, new HttpEntity(httpHeaders), Resource.class);
 InputStream inputStream = exchange.getBody().getInputStream();

using this you can write the response to somewhere else. Files.write(inputStream, new File("./test.json")); wrote the file for me, so I assume the inputstream can also be used somewhere else. ( I used Spring 4.3.5 )

edit:

As the OP states, this will still load the file in memory. Behind the scene the InputStream is a ByteArrayInputStream.

The default RestTemplate and MessageConverters are not made for streaming content at all. You could write your own implementation of a org.springframework.web.client.ResponseExtractor and maybe a MessageConverter. In ResponseExtractor you have access to the org.springframework.http.client.ClientHttpResponse

imho for your use case, you might be better of using Apache Httpcomponents HttpClient where you find HttpEntity#writeTo(OutputStream).

Answer 2

If you want to forward the response directly without ever holding it in memory, you have to directly write to the response:

@RequestMapping(value = "/yourEndPoint")
public void processRequest(HttpServletResponse response) {
    RestTemplate restTemplate = new RestTemplate();

    response.setStatus(HttpStatus.OK.value());

    restTemplate.execute(
        fileToDownloadUri,
        HttpMethod.GET,
        (ClientHttpRequest requestCallback) -> {},
        responseExtractor -> {
            IOUtils.copy(responseExtractor.getBody(), response.getOutputStream());
            return null;
        });
}