I have a child process which generates some output of variable length and then sends it to the parent using a half duplex pipe. In the parent, how do I use the read() function?
Since the data can be of different length each time, how can I at run time know the size of the data to do any malloc() for a buffer? Can the fstat() function be used on a pipe file descriptor?
I know that the read() function will read a specified number of bytes but will return 0 if the end of file (not the EOF character) is reached before the bytes requested have been read.
I am specifically running Ubuntu GNU/Linux with a 2.6.27-9 Kernel.
All examples in Richard Stevens' Advanced Programming in the UNIX Environment have specified either the length of data while writing into the pipe or have relied on fgets() stdio.h function. Since I am concerned with speed, I want to stay away from using stdio.h as much as possible.
Will this be necessarily faster with shared memory?
Thanks,
-Dhruv
Since it seems that you intend to make a single read of all the data from the pipe, I think the following will serve you better than delimiter+encoding or miniheader techniques suggested in other answers:
From the pipe (7) manpage:
If all file descriptors referring to
the write end of a pipe have been
closed, then an attempt to read(2)
from the pipe will see end-of-file
(read(2) will return 0).
The following example was taken from the pipe (2) manpage and reversed so that the child does the writing, the parent the reading (just to be sure). I also added a variable size buffer. The child will sleep for 5 seconds. The delay will ensure that the exit() of the child can have nothing to do with pipeio (the parent will print a complete line before the child exits).
#include <sys/wait.h>
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
char *
slurpfd(int fd)
{
const int bytes_at_a_time = 2;
char *read_buffer = NULL;
int buffer_size = 0;
int buffer_offset = 0;
int chars_io;
while (1) {
if (buffer_offset + bytes_at_a_time > buffer_size) {
buffer_size = bytes_at_a_time + buffer_size * 2;
read_buffer = realloc(read_buffer, buffer_size);
if (!read_buffer) {
perror("memory");
exit(EXIT_FAILURE);
}
}
chars_io = read(fd,
read_buffer + buffer_offset,
bytes_at_a_time);
if (chars_io <= 0) break;
buffer_offset += chars_io;
}
if (chars_io < 0) {
perror("read");
exit(EXIT_FAILURE);
}
return read_buffer; /* caller gets to free it */
}
int
main(int argc, char *argv[])
{
int pipefd[2];
pid_t cpid;
assert(argc == 2);
if (pipe(pipefd) == -1) {
perror("pipe");
exit(EXIT_FAILURE);
}
cpid = fork();
if (cpid == -1) {
perror("fork");
exit(EXIT_FAILURE);
}
if (cpid == 0) { /* Child writes argv[1] to pipe */
close(pipefd[0]); /* Close unused read end */
write(pipefd[1], argv[1], strlen(argv[1]) + 1);
close(pipefd[1]); /* Reader will see EOF */
/* sleep before exit to make sure that there
will be a delay after the parent prints it's
output */
sleep(5);
exit(EXIT_SUCCESS);
} else { /* Parent reads from pipe */
close(pipefd[1]); /* Close unused write end */
puts(slurpfd(pipefd[0]));
close(pipefd[0]);
wait(NULL); /* Wait for child */
_exit(EXIT_SUCCESS);
}
}
From your comment I see now that you may want to read the data as it becomes available, to update the UI or whatever, to reflect your system's status. To do that open the pipe in non-blocking (O_NONBLOCK) mode. Read repeatedly whatever is available until -1 is returnd and errno == EAGAIN and do your parsing. Repeat unil read returns 0, which indicates that the child has closed the pipe.
To use an in-memory buffer for File* functions you can use fmemopen() in the GNU C library.
Since the writing end can always write more data to the pipe, there's no way to know the size of the data in it. You can have the sender write the length first, or you can allocate a largish buffer, read as much as you can, then resize the buffer if it's not large enough.
Shared memory will be faster as it avoids copies and may avoid some syscalls, but the locking protocols needed to transfer data across shmem are more complex and prone to error, so it's generally best to avoid shared memory unless you absolutely need it. Additionally, with shared memory you must set a fixed maximum size to the data to be transferred when you allocate the buffer.
You can't get any size information from a pipe, since there is no size.
You need to either use a defined size, or a delimiter.
In other words, in the child, output the size of the upcoming output as an int, then write the actual output; in the parent you you read the size (it's an int, so it's always the same size), then read that many bytes.
Or: define an end character and until you see that, assume you need to keep reading. This may require some sort of escaping/encoding mechanism, however, and probably won't be as fast. I think this is basically what fgets does.
Why not write the length into the pipe as (say) the first 'n' bytes ? Then at the other end you can read those bytes, determine the length, and then read that number of bytes (i.e. you have a very simple protocol)
Other posters are correct: you must have a way to specify the length of the packets yourself. One concrete, practical way to do this is with netstrings. It's simple to create and parse, and it's supported by some common frameworks such as Twisted.
You might try using IPC message queues if your messages are not too big.
