Ruby—Open3.popen3 / how to print the output

Question

I have a little ruby script which does a mysql import in the way: mysql -u <user> -p<pass> -h <host> <db> < file.sql, but utilizes Open3.popen3 to do so. That is what I have so far:

mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"

Open3.popen3(mysqlimp) do |stdin, stdout, stderr, wthr|
  stdin.write "DROP DATABASE IF EXISTS #{mysqllocal['db']};\n"
  stdin.write "CREATE DATABASE #{mysqllocal['db']};\n"
  stdin.write "USE #{mysqllocal['db']};\n"

  stdin.write mysqldump #a string containing the database data
  stdin.close

  stdout.each_line { |line| puts line }
  stdout.close

  stderr.each_line { |line| puts line }
  stderr.close
end

That is actually doing the Job, but there is one thing that bothers me, concerned to the output I would like to see.

If I change the first line to:

mysqlimp = "mysql -v -u #{mysqllocal['user']} " #note the -v

then the whole script hangs forever.

I guess, that happens because the read- and write-stream block each other and I also guess that the stdout needs to be flushed regularly so that stdin will go on to be consumed. In other words, as long as the buffer of the stdout is full, the process will wait until its flushed, but since this is done at the very bottom first, that never happens.

I hope someone can verify my theory? How could I write code that does prints out everything from the stdout and writes everything to the stdin as well?

Thanks in ahead!

Answer 1

Whenever you start a process from the command line or via fork, the process inherits stdin, stdout and stderr from the father process. This means, if your command line runs in a terminal, stdin, stdout and stderr of the new process are connected to the terminal.

Open3.popen3, on the other hand, does not connect stdin, stdout and stderr to the terminal, because you do not want direct user interaction. So we need something else.

For stdin, we need something with two abilities:

1. The father process needs something to enqueue data that the subprocess is supposed to get when it reads from stdin.
2. The subprocess needs something that offers a read function like stdin does.

For stdout and stderr, we need something similar:

1. The subprocess needs something to write to. puts and print should enqueue the data, that the father process is supposed to read.
2. The father process needs something that offers a read function in order to get the stdout and stderr data of the subprocess.

This means, for stdin, stdout and stderr, we need three queues (FIFO) for communication between father process and subprocess. These queues have to act a little bit like files as they have to provide read, write (for puts and print), close and select (is data available?). Therefore, both Linux and Windows provide anonymous pipes. This is one of the conventional (local) interprocess communication mechanisms. And, well, Open3.popen3 really wants to do communication between two different processes. This is why Open3.popen3 connects stdin, stdout and stderr to anonymous pipes.

Each pipe, be it anonymous or named, does have a buffer of limited size. This size depends on operation system. The catch is: If the buffer is full and a processes tries to write to the pipe, the operating system suspends the process until another processes reads from the pipe.

This may be your problem:

1. You keep feeding data to your subprocess, but you do not read what your subprocess writes to stdout.
2. Consequently, the output of our subprocess keeps accumulating in a buffer until the buffer is full.
3. This is when the operation system suspends your subprocess (puts or print blocks).
4. Now you can still feed data to the anonymous pipe that is connected to the stdin of your subprocesses until too much of stdin data has accumulated. The buffer of the stdin pipe got full. Then the operating system will suspend the father processes (stdin.write will block).

I advise you to use Open3.capture2e or a similar wrapper around Open3.popen3. You can pass data to the subprocess with the keyword argument :stdin_data.

If you insist on communicating with your subprocess "interactively", you need to learn about IO.select or using multi-threading. Both of them are quite a challenge. Better use Open3.capture*.

Answer 2

• Since you are only writing to stdout, you can simply use Open3#popen2e which consolidates stdout and stderr into a single stream.
• To write newline terminated strings to a stream, you can use puts as you would with $stdout in a simple hello world program.
• You must use waith_thread.join or wait_thread.value to wait until the child process terminates.
• In any case, you will have to start a separate thread for reading from the stream, if you want to see the results immediately.

Example:

require 'open3'

cmd = 'sh'

Open3.popen2e(cmd) do |stdin, stdout_stderr, wait_thread|
  Thread.new do
    stdout_stderr.each {|l| puts l }
  end

  stdin.puts 'ls'
  stdin.close

  wait_thread.value
end

Your code, fixed:

require 'open3'

mysqldump = # ...

mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"

Open3.popen2e(mysqlimp) do |stdin, stdout_stderr, wait_thread|
  Thread.new do
    stdout_stderr.each {|l| puts l }
  end

  stdin.puts "DROP DATABASE IF EXISTS #{mysqllocal['db']};"
  stdin.puts "CREATE DATABASE #{mysqllocal['db']};"
  stdin.puts "USE #{mysqllocal['db']};"
  stdin.close

  wait_thread.value
end