Race condition in starting up sub processes causes

Question

I have two threads that start up a child process each. The first application is a binary that runs quite long. The second exits quite quickly.

There is a race condition that sometimes causes this to fail. Below I have a minimum viable code example.

It uses Boost Process 0.5, which uses the standard fork / execve / dup2 system. There are some hacks regarding how Boost Process works, but in general it works quite well.

The parent process starts up a lot more processes, and in general it works.

I can't readily boot processes one at a time, especially since I don't know which parts can't interleave.

Any ideas on why this would hang?

Expected output:

/etc/init.d/led restart: Creating child
Creating child1
Reading STDOUT
/etc/init.d/led restart: Waiting for it to exit
Reading std_err_pipe


wait_for_exit(pullapp);
Reading std_out_pipe
< file list>

Done

However, often, but not always, it stops at std_err_pipe.

#include <iostream>
#include <string>
#include <vector>

#include <boost/iostreams/stream.hpp>
#include <boost/process.hpp>
#include <boost/thread.hpp>

void run_sleep()
{
    int      exit_code;
    std::string   str;

    std::vector< std::string >  args;

    boost::shared_ptr<boost::process::child> child;

    args.push_back(boost::process::search_path("sleep"));
    args.push_back("20");

    boost::iostreams::stream< boost::iostreams::file_descriptor_source >
        out_stream;

    boost::process::pipe out_pipe = boost::process::create_pipe();

    {
        //MUST BE IN SEPARATE SCOPE SO SINK AND SOURCE ARE DESTROYED
        // See http://stackoverflow.com/a/12469478/5151127
        boost::iostreams::file_descriptor_sink    out_sink
            (out_pipe.sink,   boost::iostreams::close_handle);
        boost::iostreams::file_descriptor_source  out_source
            (out_pipe.source, boost::iostreams::close_handle);

        std::cout << "Creating child1" << std::endl;
        child.reset(new boost::process::child(
            boost::process::execute(
                boost::process::initializers::run_exe(args[0]),
                boost::process::initializers::set_args(args),
                boost::process::initializers::bind_stdout(out_sink),
                boost::process::initializers::bind_stderr(out_sink)
            )
        ));

        out_stream.open(out_source);
    }

    std::cout << "Reading STDOUT" << std::endl;
    while( out_stream ) {
        std::string line;

        std::getline(out_stream, line);

        std::cout << line << std::endl;
    }

    std::cout << "wait_for_exit(pullapp);" << std::endl;
    exit_code = wait_for_exit(*child);

    child.reset();

    return;
}


void run_ls()
{
    int      exit_code;
    std::string   str;

    std::vector< std::string >  args ;

    args.push_back(boost::process::search_path("ls"));
    args.push_back("/lib");

    boost::process::pipe std_out_pipe = boost::process::create_pipe();
    boost::process::pipe std_err_pipe = boost::process::create_pipe();

    std::cout << "/etc/init.d/led restart: Creating child" << std::endl;

    {
        boost::process::child child = boost::process::execute(
            boost::process::initializers::set_args(args),
            boost::process::initializers::bind_stdout(
                boost::iostreams::file_descriptor_sink(
                    std_out_pipe.sink,
                    boost::iostreams::close_handle
                )
            ),
            boost::process::initializers::bind_stderr(
                boost::iostreams::file_descriptor_sink(
                    std_err_pipe.sink,
                    boost::iostreams::close_handle
                )
            ),
            boost::process::initializers::throw_on_error()
        );

        std::cout << "/etc/init.d/led restart: Waiting for it to exit" << std::endl;
        exit_code = wait_for_exit(child);
    }

    { //with std_err_stream, istream
        boost::iostreams::stream< boost::iostreams::file_descriptor_source >
            std_err_stream(
                boost::iostreams::file_descriptor_source(
                    std_err_pipe.source, boost::iostreams::close_handle
                )
            );

        std::cout << "Reading std_err_pipe" << std::endl;
        std::istream istream(std_err_stream.rdbuf());
        while( istream ) {
            getline(istream, str);
            std::cout << str << std::endl;
        }
    }

    { //with std_out_stream, istream
        boost::iostreams::stream< boost::iostreams::file_descriptor_source >
            std_out_stream(
                boost::iostreams::file_descriptor_source(
                    std_out_pipe.source, boost::iostreams::close_handle
                )
            );

        std::cout << "Reading std_out_pipe" << std::endl;
        std::istream istream(std_out_stream.rdbuf());
        while( istream ) {
            getline(istream, str);
            std::cout << str << std::endl;
        }
    }

    std::cout << "Done" << std::endl;
}

int main()
{
    boost::thread  run_sleep_tr(run_sleep);
    boost::thread  run_ls_tr(run_ls);

    run_sleep_tr.join();
    run_ls_tr.join();

    return 0;
}

(Save as process-test.cpp and compile with g++ process-test.cpp -o process-test -lboost_iostreams -lboost_filesystem -lboost_thread -lboost_system)

Answer 1

I'm not sure if "use boost.process 0.6" counts as an answer, but that does that for you. After a few bug-reports that is. On windows closing the sink in the father process should suffice.

Answer 2

Apparently this is because file handles end up in multiple processes. Those processes don't close those handles, so the parent remains waiting.

For Linux the fix is relatively easy; the pipe should be created with O_CLOEXEC in create_pipe. The dup2 call in the bind_* methods clears this flag, which is enough for the pipe to work properly.

On Windows, I haven't really found a solution yet. You have to mark the handle as inheritable. It may be possible to do this in the executor() method, but maybe this requires a global mutex. I haven't had the time to properly look into it yet.