How to unpack a msgpack file?

Question

I am writing msgpack-encoded data to a file. On writing, I am just using the fbuffer of the C API. As in (I striped all error handling for the example):

FILE *fp = fopen(filename, "ab");
msgpack_packer pk;
msgpack_packer_init(pk, fp, msgpack_fbuffer_write);
msgpack_pack_int(pk, 42);
// more data ...

How do I read this file back in? All the example I found assume that the data is in memory, however, my files are up to 5GB, it is not exactly a good idea to hold this in memory completely. Also I do not want to read in chunks myself. After all, I do not know how long the msgpack objects are, so chances are I end up with half an integer in my buffer.

Can msgpack's unpack somehow read from disk directly? Or is there some standard pattern to do this?

Answer 1

Okay, I managed to do it.

Here is how to write:

#include <stdlib.h>
#include <msgpack.h>
#include <msgpack/fbuffer.h>

int main(int argc, char **argv) {
    if(2 != argc) {
            fprintf(stderr, "Call all writeFile <file>");
            return;
    }

    FILE *fp = fopen(argv[1], "ab");
    msgpack_packer pk;
    msgpack_packer_init(&pk, fp, msgpack_fbuffer_write);

    for(int i=0;i<2048;i++) {
            msgpack_pack_int(&pk, i);
    }
    fclose(fp);
}

And this is what the read looks like:

#include <stdlib.h>
#include <msgpack.h>

static const int BUFFERSIZE = 2048;

int main(int argc, char **argv) {
    if(2 != argc) {
            fprintf(stderr, "Call with readFile <file>");
            return 1;
    }

    char *inbuffer = (char *) malloc(BUFFERSIZE);
    if(NULL == inbuffer) {
            fprintf(stderr, "Out of memory!");
            return 1;
    }

    FILE *fp = fopen(argv[1], "rb");
    size_t off = 0;
    size_t read = 0;
    msgpack_unpacked unpacked;
    msgpack_unpacked_init(&unpacked);
    do {
            read = fread(inbuffer, sizeof(char), BUFFERSIZE - off, fp);
            off = 0;
            while(msgpack_unpack_next(&unpacked, inbuffer, read, &off)) {
                    msgpack_object_print(stdout, unpacked.data);
                    puts("");
            }
            memcpy(inbuffer, &(inbuffer[off]), read-off);
            off = read - off;
    } while(read != 0);
    free(inbuffer);
    fclose(fp);
    msgpack_unpacked_destroy(&unpacked);
    return 0;
}

I did not try, but I think it will work with larger objects (arrays, maps etc.) as well.

Answer 2

You might consider using "msgpack_unpacker" for that instead, which seems to be the official way that MessagePack implements a 'streaming' deserializer. Have a look at msgpack-c/example/c/lib_buffer_unpack.c

Regards, NiteHawk