How to zip- compress HTTP request with Spring Rest

Question

How to gzip HTTP request, created by org.springframework.web.client.RestTemplate?

I am using Spring 4.2.6 with Spring Boot 1.3.5 (Java SE, not Android or Javascript in the web browser).

I am making some really big POST requests, and I want request body to be compressed.

Answer 1

The main idea is to create requestCallback, which will copy data you want to send from gzipOutputStream directly to request stream.

RequestCallback requestCallback = new RequestCallback() {
    @Override
    public void doWithRequest(ClientHttpRequest request) throws IOException {
        GZIPOutputStream gzipOutputStream;
        try {
            gzipOutputStream = new GZIPOutputStream(request.getBody());
        } catch (IOException ignored) {
            return;
        }

        request.getHeaders().add("Content-Type", "application/octet-stream");
        request.getHeaders().add("Content-Encoding", "gzip");

        try {
            String data = "Test data.";
            gzipOutputStream.write(data.getBytes(StandardCharsets.UTF_8));
            gzipOutputStream.flush(); // Optional in this example.
            gzipOutputStream.finish();
        } catch (IOException ignored) {
        }
    }
};

Now you can use it in the next way:

RestTemplate restTemplate = new RestTemplate();
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);

ResponseExtractor<String> responseExtractor = new HttpMessageConverterExtractor<>(String.class,
        restTemplate.getMessageConverters());
String response = restTemplate.execute("http://localhost:8080/gzip.php", HttpMethod.POST, requestCallback,
        responseExtractor);

System.out.println(response);

Links:

1. Simple example of compressing HTTP request via gzip with using Spring RestTemplate
2. How to forward large files with RestTemplate?

Answer 2

I propose two solutions, one simpler without streaming and one that supports streaming.

If you don't require streaming, use a custom ClientHttpRequestInterceptor, a Spring feature.

RestTemplate rt = new RestTemplate();
rt.setInterceptors(Collections.singletonList(interceptor));

Where interceptor could be:

ClientHttpRequestInterceptor interceptor = new ClientHttpRequestInterceptor() {

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution)
            throws IOException {
        request.getHeaders().add("Content-Encoding", "gzip");
        byte[] gzipped = getGzip(body);
        return execution.execute(request, gzipped);
    } 
 }

getGzip I copied

    private byte[] getGzip(byte[] body) throws IOException {

        ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
        try {
            GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);
            try {
                zipStream.write(body);
            } finally {
                zipStream.close();
            }
        } finally {
            byteStream.close();
        }

        byte[] compressedData = byteStream.toByteArray();
        return compressedData;

    }

After configuring the interceptor all requests will be zipped.

The disadvantage of this approach is that it does not support streaming as the ClientHttpRequestInterceptor receives the content as a byte[]

If you require streaming create a custom ClientHttpRequestFactory, say GZipClientHttpRequestFactory, and use it like this:

    SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    ClientHttpRequestFactory gzipRequestFactory = new GZipClientHttpRequestFactory(requestFactory);
    RestTemplate rt = new RestTemplate(gzipRequestFactory);

Where GZipClientHttpRequestFactory is:

public class GZipClientHttpRequestFactory extends AbstractClientHttpRequestFactoryWrapper {

    public GZipClientHttpRequestFactory(ClientHttpRequestFactory requestFactory) {
        super(requestFactory);
    }

    @Override
    protected ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod, ClientHttpRequestFactory requestFactory)
            throws IOException {
        ClientHttpRequest delegate = requestFactory.createRequest(uri, httpMethod);
        return new ZippedClientHttpRequest(delegate);
    }

}

And ZippedClientHttpRequest is:

public class ZippedClientHttpRequest extends WrapperClientHttpRequest
{
    private GZIPOutputStream zip;

    public ZippedClientHttpRequest(ClientHttpRequest delegate) {
        super(delegate);
        delegate.getHeaders().add("Content-Encoding", "gzip");
        // here or in getBody could add content-length to avoid chunking
        // but is it available ? 
        // delegate.getHeaders().add("Content-Length", "39");

    }

    @Override
    public OutputStream getBody() throws IOException {
        final OutputStream body = super.getBody();
        zip = new GZIPOutputStream(body);
        return zip;
    }

    @Override
    public ClientHttpResponse execute() throws IOException {
        if (zip!=null) zip.close();
        return super.execute();
    }

}

And finally WrapperClientHttpRequest is:

public class WrapperClientHttpRequest implements ClientHttpRequest {

    private final ClientHttpRequest delegate;

    protected WrapperClientHttpRequest(ClientHttpRequest delegate) {
        super();
        if (delegate==null)
            throw new IllegalArgumentException("null delegate");
        this.delegate = delegate;
    }

    protected final ClientHttpRequest getDelegate() {
        return delegate;
    }

    @Override
    public OutputStream getBody() throws IOException {
        return delegate.getBody();
    }

    @Override
    public HttpHeaders getHeaders() {
        return delegate.getHeaders();
    }

    @Override
    public URI getURI() {
        return delegate.getURI();
    }

    @Override
    public HttpMethod getMethod() {
        return delegate.getMethod();
    }

    @Override
    public ClientHttpResponse execute() throws IOException {
        return delegate.execute();
    }
}

This approach creates a request with chunked transfer encoding, this can be changed setting the content length header, if size is known.

The advantage of the ClientHttpRequestInterceptor and/or custom ClientHttpRequestFactory approach is that it works with any method of RestTemplate. An alternate approach, passing a RequestCallback is possible only with execute methods, this because the other methods of RestTemplate internally create their own RequestCallback(s) that produce the content.

BTW it seems that there is little support to decompress gzip request on the server. Also related: Sending gzipped data in WebRequest? that points to the Zip Bomb issue. I think you will have to write some code for it.

Answer 3

Further to the above answer from @TestoTestini, if we take advantage of Java 7+'s 'try-with-resources' syntax since both ByteArrayOutputStream and GZIPOutputStream implement closeable(), we can shrink the getGzip function into the following:

private byte[] getGzip(byte[] body) throws IOException {

    try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream()) {
        try (GZIPOutputStream zipStream = new GZIPOutputStream(byteStream)) {
            zipStream.write(body);
        }
        byte[] compressedData = byteStream.toByteArray();
        return compressedData;
    }

}

(I couldn't find a way of commenting on @TestoTestini's original answer and retaining the above code format, hence this Answer).