Get pointer to and length of element in msgpack ar

Question

I have some data that was packed with msgpack using the C/C++ api as follows:

msgpack::sbuffer sbuf;
msgpack::packer<msgpack::sbuffer> pk(&sbuf);

int var1 = 10;
std::string var2 = "test";
double var3 = 3.14159; // could be any type

pk.pack_array(3);
pk.pack(var1);
pk.pack(var2);
pk.pack(var3);

At a later point I need to unpack that array, but need direct access to the third element so I can persist to file/db/redis/memcached/etc. While the first two elements of the array are fixed types, the third can be any type acceptable to msgpack (int, string, vector, map, etc).

size_t off = 0;
msgpack::unpacked result = msgpack::unpack(sbuf.data(), sbuf.len(), off);
msgpack::object obj = result.get();

int var1;
obj.via.array.ptr[0].convert(&var1);

std::string var2;
obj.via.array.ptr[1].convert(&var2);

// now here I want to get a pointer & len to the 3rd item so I can persist
// this value that is already msgpack'd.

const char* dataptr = reinterpret_cast<const char*>(&obj.via.array.ptr[2]);
// now what is the length of the data pointed to by dataptr?

Potentially I could do a reinterpret_cast on obj.via.array.ptr[2] as shown above, but in the case of binary data, or a msgpack'd struct, a simple strlen() wouldn't get me the length and I can't see where to get the item's length. I know there is a size variable in many of the types but don't believe this is accurate when that item is an array or map.

Answer 1

Here is a memory model of the msgpack-c: https://github.com/msgpack/msgpack-c/wiki/v1_1_cpp_unpacker#memory-management

I know there is a size variable in many of the types but don't believe this is accurate when that item is an array or map.

True. obj has already been unpacked. It is not suitable for persistent. I think that storing msgpack format binary data directly is better approach. First, separate the msgpack to the first two and the thrid one. Then, pack the first two as an array. Finally, simply pack the third value. That is packing process.

    pk.pack_array(2); // for the first two
    pk.pack(var1);
    pk.pack(var2);
    pk.pack(var3);    // for the thrid one

When unpacking, unpack the first two data with offset.

    // Unpacking
    size_t off = 0;
    msgpack::unpacked result = msgpack::unpack(sbuf.data(), sbuf.size(), off);
    // off has been set

After unpacking them, offset has been set. So you can get the start point of the third data. Then, store the msgpack format binary data.

    std::string store; // file/db/redis/memcached/etc
    std::copy(sbuf.data() + off, sbuf.data() + sbuf.size(), std::back_inserter(store));

That is storing process.

When you get the msgpack format binary data from storage, unpack them.

Here is a whole code example:

#include <msgpack.hpp>

#include <iostream>
#include <string>
#include <algorithm>

int main() {
    msgpack::sbuffer sbuf;
    msgpack::packer<msgpack::sbuffer> pk(&sbuf);

    int var1 = 10;
    std::string var2 = "test";
    double var3 = 3.14159; // could be any type

    // Separate the data into the two msgpacks
    pk.pack_array(2); // for the first two
    pk.pack(var1);
    pk.pack(var2);
    pk.pack(var3);    // for the thrid one

    // Unpacking
    size_t off = 0;
    msgpack::unpacked result = msgpack::unpack(sbuf.data(), sbuf.size(), off);
    msgpack::object obj = result.get();

    auto converted = obj.as<std::tuple<int, std::string>>();
    std::cout << std::get<0>(converted) << std::endl;
    std::cout << std::get<1>(converted) << std::endl;

    // Storing the thrid one
    std::cout << "off: " << off << std::endl;
    std::string store; // file/db/redis/memcached/etc
    std::copy(sbuf.data() + off, sbuf.data() + sbuf.size(), std::back_inserter(store));

    {
        // Unpack the thrid one from store
        msgpack::unpacked result = msgpack::unpack(store.data(), store.size());
        msgpack::object obj = result.get();
        if (obj.type == msgpack::type::FLOAT) {
            auto f = obj.as<float>();
            std::cout << f << std::endl;
        }
    }
}

You can check the behavior of the code above here: http://melpon.org/wandbox/permlink/uFfRGKQLqnIIiDrv