FIFOs implementation

Question

Consider the following code:

writer.c

mkfifo("/tmp/myfifo", 0660);

int fd = open("/tmp/myfifo", O_WRONLY);

char *foo, *bar;

...

write(fd, foo, strlen(foo)*sizeof(char));
write(fd, bar, strlen(bar)*sizeof(char));

reader.c

int fd = open("/tmp/myfifo", O_RDONLY);

char buf[100];
read(fd, buf, ??);

My question is:

Since it's not know before hand how many bytes will foo and bar have, how can I know how many bytes to read from reader.c?
Because if I, for example, read 10 bytes in reader and foo and bar are together less than 10 bytes, I will have them both in the same variable and that I do not want.
Ideally I would have one read function for every variable, but again I don't know before hand how many bytes will the data have.
I thought about adding another write instruction in writer.c between the write for foo and bar with a separator and then I would have no problem decoding it from reader.c. Is this the way to go about it?

Thanks.

Answer 1

You'll have to define some kind wire protocol or serialization/deserialization format so that your reader knows how to interpret the data that it's reading from the fifo. Using a separator is the simplest way to go about this but you'll run into problems if your separator ever appears as part of the data output of your writer.

Slightly farther along the complexity scale, your protocol might define both a separator and a way of indicating the length of each "piece" or "message" of data that you're sending.

Finally, this problem is more thoroughly solved by writing serialized messages which your writer will then deserialize after receiving. You may be interested in using something like Protocol Buffers or Thrift to achieve this (with the added bonus that you can implement your reader or writer in a number of different programming languages without modifying your protocol).

Answer 2

A separator is one way to go about it, and this will work fine, as long as you know the order of your data, and you use the separator as only a separator, and never as part of your data.

Another way is to precede each write to the pipe with the number of bytes to follow, in a fixed width. Thus, you will know how much data is about to come down the pipe. Use a fixed width, so you know exactly how long the width field will be, so you know both when to start and stop reading each chunk of data.

Answer 3

To generalize WhirlWind's answer slightly, you've got to establish a protocol of SOME variety. There has to be order to what you are sending or else you don't know top from bottom, as you point out.

Both of WhirlWind's suggestions will work. You can also go so far as to implement a custom (or standard) protocol on top of a pipe or FIFO to make porting your code to a more distributed environment with disparate systems and easier task later. The crux of the issue, though, is that you've got to set RULES for communicating before you're able to actually communicate.

Answer 4

Many of the other answers mention using some sort of protocol for your data and I believe this is the correct approach. This protocol can be as simple or complex as necessary. I have provided a couple of examples that you may find useful¹.

---

In a simple case, you may only have a length byte followed by the data byte(s) (i.e. C string).

+--------------+
| length byte  |
+--------------+
| data byte(s) |
+--------------+

Writer:

uint8_t foo[UCHAR_MAX+1];
uint8_t len;
int fd;

mkfifo("/tmp/myfifo", 0660);
fd = open("/tmp/myfifo", O_WRONLY);

memset(foo, UCHAR_MAX+1, 0);
len = (uint8_t)snprintf((char *)foo, UCHAR_MAX, "Hello World!");

/* The length byte is written first followed by the data. */
write(fd, len, 1);
write(fd, foo, strlen(foo));

Reader:

uint8_t buf[UCHAR_MAX+1];
uint8_t len;
int fd;

fd = open("/tmp/myfifo", O_RDONLY);

memset(buf, UCHAR_MAX+1, 0);

/* The length byte is read first followed by a read 
 * for the specified number of data bytes.
 */
read(fd, len, 1);
read(fd, buf, len);

---

In a more complex case, you may have a length byte followed by data bytes containing more than a simple C string.

+----------------+
|  length byte   |
+----------------+
| data type byte |
+----------------+
|  data byte(s)  |
+----------------+

Common Header:

#define FOO_TYPE 100
#define BAR_TYPE 200

typedef struct {
    uint8_t type;
    uint32_t flags;
    int8_t msg[20];
} __attribute__((aligned, packed)) foo_t;

typedef struct {
    uint8_t type;
    uint16_t flags;
    int32_t value;
} __attribute__((aligned, packed)) bar_t;

Writer:

foo_t foo;
unsigned char len;
int fd;

mkfifo("/tmp/myfifo", 0660);
fd = open("/tmp/myfifo", O_WRONLY);

memset(&foo, sizeof(foo), 0);
foo.type = FOO_TYPE;
foo.flags = 0xDEADBEEF;
snprintf(foo.msg, 20-1, "Hello World!");

/* The length byte is written first followed by the data. */
len = sizeof(foo);
write(fd, len, 1);
write(fd, foo, sizeof(foo));

Reader:

uint8_t buf[UCHAR_MAX+1];
uint8_t len;
uint16_t type;
union data {
    foo_t * foo;
    bar_t * bar;
}
int fd;

fd = open("/tmp/myfifo", O_RDONLY);

memset(buf, UCHAR_MAX+1, 0);

/* The length byte is read first followed by a read 
 * for the specified number of data bytes.
 */
read(fd, len, 1);
read(fd, buf, len);

/* Retrieve the message type from the beginning of the buffer. */
memcpy(&type, buf, sizeof(type));

/* Process the data depending on the type. */
switch(type) {
    case FOO_TYPE:
        data.foo = (foo_t)buf;
        printf("0x%08X: %s\n", data.foo.flags, data.foo.msg); 
        break;
    case BAR_TYPE:
        data.bar = (bar_t)buf;
        printf("0x%04X: %d\n", data.bar.flags, data.bar.value); 
        break;
    default:
        printf("unrecognized type\n");
}

_{1 - This code was written from memory and is untested.}

Answer 5

A separator is indeed one way to do this - and conveniently enough, C strings come with such a separator - the nul-terminator at the end of the string.

If you change your write() calls so that they also write out the nul-terminator (note that sizeof(char) is defined to be 1, so it can be left out):

write(fd, foo, strlen(foo) + 1);
write(fd, bar, strlen(bar) + 1);

You can then pick apart the strings after you read them in (you will still need to read them into one buffer and then break them apart, unless you read them a character at a time).