Google protocol buffers and use of std::string for

Question

Related Question: vector <unsigned char> vs string for binary data.

My code uses vector<unsigned char> for arbitrary binary data. However, a lot of my code has to interface to Google's protocol buffers code. Protocol buffers uses std::string for arbitrary binary data. This makes for a lot of ugly allocate/copy/free cycles just to move data between Google protocol buffers and my code. It also makes for a lot of cases where I need two constructors (one which takes a vector and one a string) or two functions to convert a function to binary wire format.

The code deals with raw structures a lot internally because structures are content-addressable (stored and retrieved by hash), signed, and so on. So it's not just a matter of the interface to Google's protocol buffers. Objects are handled in raw forms in other parts of the code as well.

One thing I could do is just cut all my code over to use std::string for arbitrary binary data. Another thing I could do is try to work out more efficient ways to store and retrieve my vectors into Google protocol buffer objects. I guess my other choice would be to create standard, simple, but slow conversion functions to strings and always use them. This would avoid the rampant code duplication, but would be worst from a performance standpoint.

Any suggestions? Any better choices I'm missing?

This is what I'm trying to avoid:

if(SomeCase)
{
    std::vector<unsigned char> rawObject(objectdata().size());
    memcpy(&rawObject.front(), objectdata().data(), objectdata().size());
    DoSometingWith(rawObject);
}

The allocate, copy, process, free is completely senseless when the raw data is already sitting there.

Answer 1

To avoid this code (which you present),

if(SomeCase)
{
    std::vector<unsigned char> rawObject(objectdata().size());
    memcpy(&rawObject.front(), objectdata().data(), objectdata().size());
    DoSometingWith(rawObject);
}

where presumably objectData is a std::string, consider

typedef unsigned char      Byte;
typedef std::vector<Byte>  ByteVector;

and then e.g.

if( someCase )
{
    auto const& s = objectData;
    doSomethingWith( ByteVector( s.begin(), s.end() ) );
}

Answer 2

There are two ways to avoid copying that I know of and have seen in use.

The traditional way is indeed to pass a pointer/reference to a known entity. While this works fine and with a minimum of fuss, the issue is that it ties you up to a given representation, which entails conversions (as you experienced) when necessary.

The other way I discovered with LLVM:

• ArrayRef
• StringRef

The idea is amazingly simple: both hold a T* pointing to the start of an array of T and a size_t indicating the number of elements.

What is magical is that they completely hide the actual storage, be it a string, a vector, a dynamically or statically allocated C-array... it does not matter. The interface presented is completely uniform and no copy is involved.

The only caveat is that they do not take ownership of the memory (Ref!) so subtle bugs might creep in if you do not take care. Still, it is usually fine if you only use them in transient operations (within a function, for example) and do not store them for later use.

I have found them incredibly handy in buffer manipulations, especially thanks to the free slicing operations. Ranges are just so much easier to manipulate than pairs of iterators.

---

There is also a third way I have experienced, but never used in serious code up until now. The idea is that a vector<unsigned char> is a very low-level representation. By raising the abstraction layer and use, say, a Buffer class, you can completely encapsulate the exact way the memory is stored so that it becomes a non-issue, as far as your code is concerned.

And then, feel free to choose the one memory representation that requires the less conversion.