How to dump a compressed object for given key from

2019-08-28 05:37发布

问题:

I'm using the following command to dump the compressed object for given key from memcached:

cat <(printf "\x1f\x8b\x08\x00\x00\x00\x00\x00") <(memccat CACHE-KEY) | gunzip

It prints the value (a JSON), but with the warning at the end:

gzip: stdin: unexpected end of file

I believe it may be missing last 4 bytes of checksum (ADLER32), but I'm not sure.

What would be the proper way of dumping a compressed key value in a plain text format from Memcached caching service?

回答1:

It looks like the result of memccat CACHE-KEY is a zlib (RFC 1950) stream. You are providing eight of the ten bytes of a gzip header for a gzip stream (RFC 1952) as a prefix, which results in gunzip eating the two-byte zlib header as the missing two-bytes at the end of the gzip header. gunzip is then expecting a deflate stream (RFC 1951), which it is getting, followed by a gzip trailer which it is not getting. A gzip trailer is eight bytes, consisting of the CRC-32 and length of the uncompressed data. Instead it is getting the zlib trailer, which is a four-byte Adler-32 of the uncompressed data.

To satsify gunzip, you would need to not give it the last four bytes of the zlib stream, replacing it with eight bytes consisting of the CRC-32 of the uncompressed data, which you would need to calculate, and the length of the uncompressed data. That's kind of silly though, since the point of those things is as an integrity check on the uncompressed data.

What you really want to do is interpret that zlib stream as a zlib stream, which will make use of the Adler-32 at the end as an integrity check. You can use pigz to decode the zlib stream with pigz -dz, or you can write your own zlib decoder easily with zlib. An example is provided in the examples directory called zpipe.