Struct padding in C++

Question

If I have a struct in C++, is there no way to safely read/write it to a file that is cross-platform/compiler compatible?

Because if I understand correctly, every compiler 'pads' differently based on the target platform.

Answer 1

No. That is not possible. It's because of lack of standardization of C++ at the binary level.

Don Box writes (quoting from his book Essential COM, chapter COM As A Better C++)

C++ and Portability

Once the decision is made to distribute a C++ class as a DLL, one is faced with one of the fundamental weaknesses of C++, that is, lack of standardization at the binary level. Although the ISO/ANSI C++ Draft Working Paper attempts to codify which programs will compile and what the semantic effects of running them will be, it makes no attempt to standardize the binary runtime model of C++. The first time this problem will become evident is when a client tries to link against the FastString DLL's import library from a C++ developement environment other than the one used to build the FastString DLL.

Struct padding is done differently by different compilers. Even if you use the same compiler, the packing alignment for structs can be different based on what pragma pack you're using.

Not only that if you write two structs whose members are exactly same, the only difference is that the order in which they're declared is different, then the size of each struct can be (and often is) different.

For example, see this,

struct A
{
   char c;
   char d;
   int i;
};

struct B
{
   char c;
   int i;
   char d;
};

int main() {
        cout << sizeof(A) << endl;
        cout << sizeof(B) << endl;
}

Compile it with gcc-4.3.4, and you get this output:

8
12

That is, sizes are different even though both structs have the same members!

Code at Ideone: http://ideone.com/HGGVl

The bottom line is that the standard doesn't talk about how padding should be done, and so the compilers are free to make any decision and you cannot assume all compilers make the same decision.

Answer 2

If you have the opportunity to design the struct yourself, it should be possible. The basic idea is that you should design it so that there would be no need to insert pad bytes into it. the second trick is that you must handle differences in endianess.

I'll describe how to construct the struct using scalars, but the you should be able to use nested structs, as long as you would apply the same design for each included struct.

First, a basic fact in C and C++ is that the alignment of a type can not exceed the size of the type. If it would, then it would not be possible to allocate memory using malloc(N*sizeof(the_type)).

Layout the struct, starting with the largest types.

 struct
 {
   uint64_t alpha;
   uint32_t beta;
   uint32_t gamma;
   uint8_t  delta;

Next, pad out the struct manually, so that in the end you will match up the largest type:

   uint8_t  pad8[3];    // Match uint32_t
   uint32_t pad32;      // Even number of uint32_t
 }

Next step is to decide if the struct should be stored in little or big endian format. The best way is to "swap" all the element in situ before writing or after reading the struct, if the storage format does not match the endianess of the host system.

Answer 3

You could use something like boost::serialization.

Answer 4

No, there's no safe way. In addition to padding, you have to deal with different byte ordering, and different sizes of builtin types.

You need to define a file format, and convert your struct to and from that format. Serialization libraries (e.g. boost::serialization, or google's protocolbuffers) can help with this.

Answer 5

Long story short, no. There is no platform-independent, Standard-conformant way to deal with padding.

Padding is called "alignment" in the Standard, and it begins discussing it in 3.9/5:

Object types have alignment requirements (3.9.1, 3.9.2). The alignment of a complete object type is an implementation-defined integer value representing a number of bytes; an object is allocated at an address that meets the alignment requirements of its object type.

But it goes on from there and winds off to many dark corners of the Standard. Alignment is "implementation-defined" meaning it can be different across different compilers, or even across address models (ie 32-bit/64-bit) under the same compiler.

Unless you have truly harsh performance requirements, you might consider storing your data to disc in a different format, like char strings. Many high-performance protocols send everything using strings when the natural format might be something else. For example, a low-latency exchange feed I recently worked on sends dates as strings formatted like this: "20110321" and times are sent similarly: "141055.200". Even though this exchange feed sends 5 million messages per second all day long, they still use strings for everything because that way they can avoid endian-ness and other issues.