Ruby on Rails with IMAP IDLE for multiple accounts

Question

I'm currently building a Ruby on Rails app that allows users to sign in via Gmail and it have a constant IDLE connection to their Inbox. Emails need to arrive in the app as soon as they come into their Gmail Inbox.

Currently I have the following in terms of implementation, and some issues that I really need some help figuring out.

At the moment, when the Rails app boots up, it creates a thread per user which authenticates and runs in a loop to keep the IDLE connection alive.

Every 10-15 minutes, the thread will "bounce IDLE", so that a little data is transferred to make sure the IDLE connection stays alive.

The major issue I think is in terms of scalability and how many connections the app has to Postgres. It seems that each thread requires a connection to Postgres, this will be heavily limited on Heroku by the number of max connections (20 for basic and 500 for any plans after that).

I really need help with the following:

• What's the best way to keep all these IDLE connections alive, but reducing the number of threads and connections needed to the database?
  • Note: user token refresh may happen if the refresh token to Gmail runs out, so this requires access to the database
• Are there any other suggestions for how this may be implemented?

EDIT:

I have implemented something similar to the OP in this question: Ruby IMAP IDLE concurrency - how to tackle?

Answer 1

Start an EventMachine in Rails

Since you are on Heroku, you are probably using thin, which already starts an EventMachine for you. However, should you ever move to another host and use some other web server (e.g. Phusion Passenger), you can start an EventMachine with a Rails initializer:

module IMAPManager
  def self.start
    if defined?(PhusionPassenger)
      PhusionPassenger.on_event(:starting_worker_process) do |forked|
      # for passenger, we need to avoid orphaned threads
        if forked && EM.reactor_running?
          EM.stop
        end
        Thread.new { EM.run }
        die_gracefully_on_signal
      end
    else
      # faciliates debugging
      Thread.abort_on_exception = true
      # just spawn a thread and start it up
      Thread.new { EM.run } unless defined?(Thin)
      # Thin is built on EventMachine, doesn't need this thread
    end
  end

  def self.die_gracefully_on_signal
    Signal.trap("INT")  { EM.stop }
    Signal.trap("TERM") { EM.stop }
  end
end

IMAPManager.start

(adapted from a blog post by Joshua Siler.)

Share 1 connection

What you have is a good start, but having O(n) threads with O(n) connections to the database is probably hard to scale. However, since most of these database connections are not doing anything most of the time, one might consider sharing one database connection.

As @Deepak Kumar mentioned, you can use the EM IMAP adapter to maintain the IMAP IDLE connections. In fact, since you are using EM within Rails, you might be able to simply use Rails' database connection pool by making your changes through the Rails models. More information on configuring the connection pool can be found here.

Answer 2

There is no need to spawn a new thread for each IMAP session. These can be done in a single thread.

Maintain an Array (or Hash) of all users and their IMAP sessions. Spawn a thread, in that thread, send IDLE keep-alive to each of the connections one after the other. Run the loop periodically. This will definitely give you far more concurrency than your current approach.

A long term approach will be to use EventMachine. That will allow using many IMAP connections in the same thread. If you are processing web requests in the same process, you should create a separate thread for Event Machine. This approach can provide you phenomenal concurrency. See https://github.com/ConradIrwin/em-imap for Eventmachine compatible IMAP library.