Gzip Byte Array not the same for C# and R

Question

I'm using RserveCLI2 on C#. I tried to pass gzip byte array generated by R to C# so that C# can decompress it. However I'm not able to get it work. I did some comparison of the byte array generated by both R and C# when doing gzip compression of string "ABCDEF". Here are the results.

    # R gzip compression command and result
    > as.numeric(memCompress(charToRaw("ABCDEF"),"gzip"))
    [1] 120 156 115 116 114 118 113 117   3   0   5 126   1 150

    # C-sharp gzipstream compress result
    byte[] data1 = Encoding.ASCII.GetBytes("ABCDEF");
    var memoryStream = new MemoryStream();
    using (var gz = new GZipStream(memoryStream, CompressionMode.Compress))
    {
        gz.Write(data1, 0, data1.Length);
    }

    memoryStream.ToArray()
  {byte[26]}
  [0]: 31
  [1]: 139
  [2]: 8
  [3]: 0
  [4]: 0
  [5]: 0
  [6]: 0
  [7]: 0
  [8]: 4
  [9]: 0
  [10]: 115
  [11]: 116
  [12]: 114
  [13]: 118
  [14]: 113
  [15]: 117
  [16]: 3
  [17]: 0
  [18]: 105
  [19]: 254
  [20]: 118
  [21]: 187
  [22]: 6
  [23]: 0
  [24]: 0
  [25]: 0

I did some study and gzip byte array should starts with 31 and 139 (Hexadecimal 1F and 8B). In this case seems like C# byte array is correct. So I wonder why is it R byte array so different compared to C#? Is there any way that make R generate byte array similar to C#? Thanks.

Answer 1

This should provide you with the results you need. It's a modified version of a function in the now defunct Rcompression package. I've found the built in mem* functions to be nigh useless:

library(inline)

gz_compress <- cfunction(
  sig=c(r_content="raw"),
  body="
  int status, numProtects = 0, level = 1, method = Z_DEFLATED, 
      windowBits = 15+16, memLevel = 9, strategy = 0;
  uLongf destLen = 0;
  z_stream strm;  

  strm.zalloc = NULL;
  strm.zfree = NULL;
  strm.opaque = NULL;
  strm.total_out = 0;
  strm.next_in = RAW(r_content);
  strm.avail_in = GET_LENGTH(r_content);

  SEXP r_result = Rf_allocVector(RAWSXP, strm.avail_in * 1.01 + 12);

  status = deflateInit2(&strm, level, method, windowBits, memLevel, strategy);
  if(status != Z_OK) return(r_content);

  destLen = GET_LENGTH(r_result);

  do {
    strm.next_out = RAW(r_result) + strm.total_out;
    strm.avail_out = destLen - strm.total_out;

    status = deflate(&strm, Z_FINISH);
    if (status == Z_STREAM_END) 
      break;
    else if (status == Z_OK) {
      SET_LENGTH(r_result, 2*destLen);
      PROTECT(r_result); numProtects++;
      destLen *= 2;
    } else if (status == Z_MEM_ERROR) {
      return(r_content);
    }
  } while(1);

  SET_LENGTH(r_result, strm.total_out);

  deflateEnd(&strm);

  if (numProtects) UNPROTECT(numProtects);

  return(r_result);
  ",
  includes=c("#include <zlib.h>", "#include <Rdefines.h>", "#include <Rinternals.h>", '#include "R_ext/Memory.h"', '#include "R_ext/Utils.h"'),
  libargs="-lz"
)

You can see that it produces saner results:

gz_compress(charToRaw("ABCDEF"))
## [1] 1f 8b 08 00 00 00 00 00 04 03 73 74 72 76 71 75 03 00 69 fe 76 bb 06 00 00 00

as.integer(gz_compress(charToRaw("ABCDEF")))
## [1]  31 139   8   0   0   0   0   0   4   3 115 116 114 118 113 117   3   0 105 254 118 187   6   0   0   0

If there's sufficient demand, I can resurrect and make a modern version of the Rcompression package (I've had to resurrect it for myself and work projects but would be glad to have a more publicly available version that would be easier to use than inline C code).

---

I threw it into a tiny package that you can install with:

devtools::install_github("hrbrmstr/gzmem")

Now, all you have to do is:

library(gzmem)

mem_compress(charToRaw("ABCDEF"))
## [1] 1f 8b 08 00 00 00 00 00 04 03 73 74 72 76 71 75 03 00 69 fe 76 bb 06 00 00 00

rawToChar(mem_inflate(mem_compress(charToRaw("ABCDEF")), 32))
## [1] 41 42 43 44 45 46

There will be a much better version of mem_inflate() later this week.