HTTP request compression

Question

General Use-Case

Imagine a client that is uploading large amounts of JSON. The Content-Type should remain application/json because that describes the actual data. Accept-Encoding and Transfer-Encoding seem to be for telling the server how it should format the response. It appears that responses use the Content-Encoding header explicitly for this purpose, but it is not a valid request header.

Is there something I am missing? Has anyone found an elegant solution?

Specific Use-Case

My use-case is that I have a mobile app that is generating large amounts of JSON (and some binary data in some cases but to a lesser extent) and compressing the requests saves a large amount of bandwidth. I am using Tomcat as my Servlet container. I am using Spring for it's MVC annotations primarily just to abstract away some of the JEE stuff into a much cleaner, annotation-based interface. I also use Jackson for auto (de)serialization.

I also use nginx, but I am not sure if thats where I want the decompression to take place. The nginx nodes simply balance the requests which are then distributed through the data center. It would be just as nice to keep it compressed until it actually got to the node that was going to process it.

Thanks in advance,

John

EDIT:

The discussion between myself and @DaSourcerer was really helpful for those that are curious about the state of things at the time of writing this.

I ended up implementing a solution of my own. Note that this specifies the branch "ohmage-3.0", but it will soon be merged into the master branch. You might want to check there to see if I have made any updates/fixes.

https://github.com/ohmage/server/blob/ohmage-3.0/src/org/ohmage/servlet/filter/DecompressionFilter.java

Answer 1

Add to your header when you are sending:

JSON : "Accept-Encoding" : "gzip, deflate"

Client code :

HttpUriRequest request = new HttpGet(url);
request.addHeader("Accept-Encoding", "gzip");

@JulianReschke pointed out that there can be a case of:

"Content-Encoding" : "gzip, gzip"

so extended server code will be:

InputStream in = response.getEntity().getContent();
Header encodingHeader = response.getFirstHeader("Content-Encoding");

String gzip = "gzip";
if (encodingHeader != null) {
    String encoding = encodingHeader.getValue().toLowerCase();
    int firstGzip = encoding.indexOf(gzip);
    if (firstGzip > -1) {
      in = new GZIPInputStream(in);
      int secondGzip = encoding.indexOf(gzip, firstGzip + gzip.length());
      if (secondGzip > -1) {
        in = new GZIPInputStream(in);
      }
    }
}

I suppose that nginx is used as load balancer or proxy, so you need to set tomcat to do decompression.

Add following attributes to the Connector in server.xml on Tomcat,

<Connector 
compression="on"
compressionMinSize="2048"
compressableMimeType="text/html,application/json"
... />

Accepting gziped requests in tomcat is a different story. You'll have to put a filter in front of your servlets to enable request decompression. You can find more about that here.

Answer 2

Because the original code is not available any more. In case someone come here need it. I use "Content-Encoding: gzip" to identify the filter need to decompression or not.

Here's the codes.

 @Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException
{
    HttpServletRequest httpServletRequest = (HttpServletRequest) request;

    String contentEncoding = httpServletRequest.getHeader("Content-Encoding");
    if (contentEncoding != null && contentEncoding.indexOf("gzip") > -1)
    {
        try
        {
            final InputStream decompressStream = StreamHelper.decompressStream(httpServletRequest.getInputStream());

            httpServletRequest = new HttpServletRequestWrapper(httpServletRequest)
            {

                @Override
                public ServletInputStream getInputStream() throws IOException
                {
                    return new DecompressServletInputStream(decompressStream);
                }

                @Override
                public BufferedReader getReader() throws IOException
                {
                    return new BufferedReader(new InputStreamReader(decompressStream));
                }
            };
        }
        catch (IOException e)
        {
            mLogger.error("error while handling the request", e);
        }
    }

    chain.doFilter(httpServletRequest, response);
}

Simple ServletInputStream wrapper class

public static class DecompressServletInputStream extends ServletInputStream
{
    private InputStream inputStream;

    public DecompressServletInputStream(InputStream input)
    {
        inputStream = input;

    }

    @Override
    public int read() throws IOException
    {
        return inputStream.read();
    }

}

Decompression stream code

public class StreamHelper
{

    /**
     * Gzip magic number, fixed values in the beginning to identify the gzip
     * format <br>
     * http://www.gzip.org/zlib/rfc-gzip.html#file-format
     */
    private static final byte GZIP_ID1 = 0x1f;
    /**
     * Gzip magic number, fixed values in the beginning to identify the gzip
     * format <br>
     * http://www.gzip.org/zlib/rfc-gzip.html#file-format
     */
    private static final byte GZIP_ID2 = (byte) 0x8b;

    /**
     * Return decompression input stream if needed.
     * 
     * @param input
     *            original stream
     * @return decompression stream
     * @throws IOException
     *             exception while reading the input
     */
    public static InputStream decompressStream(InputStream input) throws IOException
    {
        PushbackInputStream pushbackInput = new PushbackInputStream(input, 2);

        byte[] signature = new byte[2];
        pushbackInput.read(signature);
        pushbackInput.unread(signature);

        if (signature[0] == GZIP_ID1 && signature[1] == GZIP_ID2)
        {
            return new GZIPInputStream(pushbackInput);
        }
        return pushbackInput;
    }
}

Answer 3

It appears [Content-Encoding] is not a valid request header.

That is actually not quite true. As per RFC 2616, sec 14.11, Content-Encoding is an entity header which means it can be applied on the entities of both, http responses and requests. Through the powers of multipart MIME messages, even selected parts of a request (or response) can be compressed.

However, webserver support for compressed request bodies is rather slim. Apache supports it to a degree via the mod_deflate module. It's not entirely clear to me if nginx can handle compressed requests.