Streaming data from Sinatra/Rack application

Question

I am trying to stream textual data (XML/JSON) from a Ruby (1.9.1p378) Sinatra (1.0) Rack (1.2.1) application. The suggested solutions (e.g. Is there a way to flush html to the wire in Sinatra) do not seem to work - the server just blocks when I yield elements of some infinite stream (e.g. from %w(foo bar).cycle). I tried webrick and thin as servers.

Any suggestions on getting this done? Should I use http://sinatra.rubyforge.org/api/classes/Sinatra/Streaming.html and if so how would I use it in my application?

Answer 1

As Colin mentioned, Goliath can stream response data, as well as incoming (large file uploads). There is an example in the repo for streaming data to the client: https://github.com/postrank-labs/goliath/blob/master/examples/stream.rb

Instead of a timer, you can easily hook up any other data stream to push data to the client. For example, you can connect an AMQP queue, or any other message queue directly to Goliath and let it act as an HTTP frontend to that data.

Answer 2

Neither Webrick nor Thin support streaming that way. You could try Mongrel or Unicorn. If you want to use Thin or Rainbows!, you have to hook into the event loop in order to achieve streaming:

require 'sinatra'

class Stream
  include EventMachine::Deferrable
  def initialize
    @counter = 0
  end

  def each(&block)
    if @counter > 10
      succeed
    else
      EM.next_tick do
        yield counter
        each(&block)
      end
    end
  end
end

get '/' do
  Stream.new
end

I recently wrote a EventSource implementation that way:

require 'sinatra'

class EventStream
  include EventMachine::Deferrable
  def each
    count = 0
    timer = EventMachine::PeriodicTimer.new(1) do
      yield "data: #{count += 1}\n\n"
    end
    errback { timer.cancel }
  end
end

get '/' do
  EventMachine.next_tick do
    request.env['async.callback'].call [
      200, {'Content-Type' => 'text/event-stream'},
      EventStream.new ]
  end
  [-1, {}, []]
end

If you want to use Webrick for Streaming: here is a patch.

Answer 3

Starting with Sinatra 1.3, you could also use the new streaming API:

get '/evented' do
  stream(:keep_open) do |out|
    EventMachine::PeriodicTimer.new(1) { out << "#{Time.now}\n" }
  end
end

Answer 4

You should definitely take a look at the rack-able Goliath web server. It supports streaming out-of-the-box. I am using it for a firehose style streaming api.

Goliath is both an app server and a lightweight framework designed to meet the following goals: fully asynchronous processing, middleware support, simple configuration, high-performance, and arguably most importantly, readable and maintainable code.