Ruby pipes: How do I tie the output of two subproc

问题:

Is there an automated way to do shell piping in Ruby? I'm trying to convert the following shell code to Ruby:

a | b | c... > ...

but the only solution I have found so far is to do the buffer management myself (simplified, untested, hope it gets my meaning across):

a = IO.popen('a')
b = IO.popen('b', 'w+')
Thread.new(a, b) { |in, out|
    out.write(in.readpartial(4096)) until in.eof?
    out.close_write
}
# deal with b.read...

I guess what I'm looking for is a way to tell popen to use an existing stream, instead of creating a new one? Or alternatively, an IO#merge method to connect a's output to b's input? My current approach becomes rather unwieldly when the number of filters grows.

I know about Kernel#system('a | b') obviously, but I need to mix Ruby filters with external program filters in a generic way.

回答1:

Old question, but since its one of the first result on Google, here is the answer : http://devver.wordpress.com/2009/10/12/ruby-subprocesses-part_3/ (method 8)

In short :

sh = Shell.new
sh.system("a") | sh.system("b") | sh.system("c")

And you can do more complicated things like

sh.echo(my_string) | sh.system("wc") > "file_path"
xml = (sh.echo(html) | sh.system("tidy", "-q")).to_s

回答2:

If a, b, and c are commands normally accessed from the command-line then you could use:

captured_output = `a | b | c`

Ruby will run the commands in a sub-shell, and capture STDOUT.

If you need to route the output to a file for some reason, then you can add the redirection to the command too. STDOUT will not be returned to you in that case, but you could open the file and manually process it:

`a | b | c > captured_output`
File.foreach('captured_output') do |li|
  print li
end

It doesn't offer as much control as using system or popen3 but it's convenient:

>> sin, sout, serr = Open3.popen3('ls -al | tail -1') #=> [#<IO:fd 4>, #<IO:fd 5>, #<IO:fd 7>, #<Thread:0x00000100bb8798 run>]
>> sout.read #=> "drwxr-xr-x   3 greg  staff    102 Nov  2 21:01 python\n"

回答3:

Using plain ruby, spawn has redirection options that you can use to connect processes with pipes.

1) Create a pipe

r,w = IO.pipe

2) Use it to connect two spawned processes

spawn(*%w[echo hello world], out: w)
spawn(*%w[tr a-z A-Z], in: r)
# => HELLO WORLD

Of course, you can encapsulate this in something like sh.system from the mentioned Shell library, and create a |() method for doing the interconnecting.

The open3 module of the standard library has some really nice tools for this kind of stuff, including the creation of complete pipelines.

回答4:

Sadly, piping in shell is a serious matter and, indeed, it would require a fair amount of code. It's not necessary to spawn threads with read/write loops, but it would still require a lot of work.

The simplest example I've found is redirect implementation in dash (Debian Almquist Shell). Generally, if you'd want to do the same in Ruby, you'll need to replicate these fd manipulation tricks using Ruby's IO#dup, IO#fileno, IO#pipe, IO#reopen, etc. It might be easier to reuse shell (such as dash's) code in C .so library for Ruby interpreter than trying to combine the same thing using just Ruby primitives.

I don't know of any existing generalized Ruby API for complex interprocess piping / redirection. If you could suggest a good API you'd like to use, may be I could participate in implementation.