Download Large file from server using REST templat

Question

I have a REST service which sends me a large ISO file ,there are no issues in the REST service . Now I have written a Web application which calls the rest service to get the file ,on the client(web app) side I receive a Out Of memory Exception.Below is my code

HttpHeaders headers = new HttpHeaders();//1 Line

    headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
    headers.set("Content-Type","application/json");//3 Line
    headers.set("Cookie", "session=abc");//4 Line
    HttpEntity statusEntity=new HttpEntity(headers);//5 Line
    String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line

    ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line

I receive out of memory exception at 7 line ,I guess i will have to buffer and get in parts ,but dont know how can i get this file from the server ,the size of the file is around 500 to 700 MB . Can anyone please assist .

Exception Stack:

  org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause

java.lang.OutOfMemoryError: Java heap space
    java.util.Arrays.copyOf(Arrays.java:3236)
    java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
    org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
    org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
    org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
    org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
    org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
    org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
    org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
    org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
    org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
    org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
    com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)

My Server Side REST Service Code which is working fine is

@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{



    byte[] reportBytes = null;
    File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);

    if(result.exists()){
        InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName); 
        String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
        response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
        response.setHeader("Content-Type",type);

        reportBytes=new byte[100];//New change
        OutputStream os=response.getOutputStream();//New change
        int read=0;
        while((read=inputStream.read(reportBytes))!=-1){
            os.write(reportBytes,0,read);
        }
        os.flush();
        os.close();






    }

Answer 1

This prevents loading the entire request into memory.

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate rest = new RestTemplate(requestFactory);

For java.lang.OutOfMemoryError: Java heap space can be solved adding more memory to the JVM:

-Xmxn Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration.

For server deployments, -Xms and -Xmx are often set to the same value. See Garbage Collector Ergonomics at http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html

Examples:

-Xmx83886080
-Xmx81920k
-Xmx80m

Probably the problem you have is not strictly related to the request you are trying to execute (download large file) but the memory allocated for the process is not enough.

Answer 2

Here is how I do it. Based on hints from this Spring Jira issue.

RestTemplate restTemplate // = ...;

// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

From the aforementioned Jira issue:

Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.

Update for Spring 5

Spring 5 introduced the WebClient class which allows asynchronous (e.g. non-blocking) http requests. From the doc:

By comparison to the RestTemplate, the WebClient is:

• non-blocking, reactive, and supports higher concurrency with less hardware resources.
• provides a functional API that takes advantage of Java 8 lambdas.
• supports both synchronous and asynchronous scenarios.
• supports streaming up or down from a server.

Answer 3

You should use multipart file attachment, so the file stream isn't load into memory. In this example, I use a rest service implemented with Apache CXF.

...
import org.apache.cxf.jaxrs.ext.multipart.Attachment;
...

@Override
@Path("/put")
@Consumes("multipart/form-data")
@Produces({ "application/json" })
@POST
public SyncResponseDTO put( List<Attachment> attachments) {
    SyncResponseDTO response = new SyncResponseDTO();
    try {
        for (Attachment attr : attachments) {
            log.debug("get input filestream: " + new Date());
            InputStream is = attr.getDataHandler().getInputStream();