Heroku + Sidekiq: ActiveRecord::StatementInvalid:

Question

Hi we are running on Heroku's Cedar stack with Unicorn and Sidekiq. We intermittently get the following errors

BurnThis ActiveRecord::StatementInvalid: PG::UnableToSend: SSL SYSCALL error: EOF detected

ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() SSL SYSCALL error: Connection timed out

Does anyone have any insight what the direct cause of these errors? Is it too many connections to our database? We have our forking set up already in the following way:

unicorn.rb

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 30
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::

Base.connection.disconnect!
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  # other setup
  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['adapter'] = 'postgis'
    config['pool']              = ENV['DB_POOL'] || 5
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    ActiveRecord::Base.establish_connection(config)
  end
end

And sidekiq.rb

Sidekiq.configure_server do |config|
  config.redis = { :url => ENV['REDIS_URL'], :namespace => 'btsidekiq' }

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['adapter'] = 'postgis'
    config['pool']              = ENV['DB_POOL'] || 5
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    ActiveRecord::Base.establish_connection(config)
  end
end

Sidekiq.configure_client do |config|
  config.redis = { :url => ENV['REDIS_URL'], :namespace => 'btsidekiq' }
end

Our database pool size is pretty large DB_POOL=100 and we are on a PG database that apparently supports 500 connections concurrently.

Answer 1

This error is caused by your postgis adapter trying to use a stale/dead connection from the ActiveRecord connection pool. There are two ways to address this issue:

1. Size your connection pool to match the number of threads/process
2. Lower connection reaping frequency (Reaper checks pool for dead connections, every N secs)

To implement #1, you need to set your pool size appropriate for Unicorn and for Sidekiq, which likely have different needs.

Unicorn is single threaded, so the default pool size of 5 connections per process is correct for you. This will allocate up to 5 connections for each of WEB_CONCURRENCY backend unicorn workers. You should reset the default pool size and use your existing unicorn.rb:

$> heroku config:set DB_POOL=5

Sidekiq however uses a very different model. By default, Sidekiq has a single process and N threads. You want a slightly larger DB pool size than the number of Sidekiq threads. You can implement this in your config/initializers/sidekiq.rb as follows:

Sidekiq.configure_server do |config|
  pool_size = Sidekiq.options[:concurrency] + 2

  config.redis = { :url => ENV['REDIS_URL'], :namespace => 'btsidekiq', :size => pool_size }

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['adapter'] = 'postgis'
    config['pool']              = pool_size
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    ActiveRecord::Base.establish_connection(config)
  end
end

My best guess is that using such a large 100 connection pool, you are more likely to accrue dead connections. Sizing the pool appropriately should fix this.

If this does not work, you should try decreasing your DB_REAP_FREQ to 5 seconds.