ruby resque without loading rails environment

Question

I have a resque worker which works great but is just too slow. The main reason for this is I'm using activerecord and having to load the entire environment which takes at least 10-20 seconds just to load up (I don't keep a running worker at all times as I'm using Heroku and pay for the time the worker runs). I'm using a resque worker to grab & parse data from an external website and then dumping the data into my database.

My question is whether I should rewrite the method to not use Rails and instead use DataMapper? Or something else which would load faster than activerecord.

Or If I should extract the code (using ActiveRecord) which figures out what to do with the external data and move it out of the worker and back into the app?

Hope that makes sense.

Answer 1

I have the same problem. you could setup your environment on the rake resque:setup rake task

I tried this. assuming my rake resque task is on lib/tasks/resque.rake

require "resque/tasks"

task "resque:setup" do
  root_path = "#{File.dirname(__FILE__)}/../.."

  db_config = YAML::load(File.open(File.join(root_path,'config','database.yml')))["development"]
  ActiveRecord::Base.establish_connection(db_config)

  require "#{root_path}/app/workers/photo_downloader.rb" #workers

  #Dir.glob("#{root_path}/app/models/*").each { |r| puts r; require r } #require all model

  require "#{root_path}/app/models/photo.rb" # require model individually
end

I haven't completely success beacuse I use the Paperclip gem which require rails environment

Answer 2

Use bundler to get active_record and other gem to you without rails application .

require 'rubygems'
require 'logger'
require 'active_record'
require 'bundler'

require "active_support"

require "spreadsheet"

require 'net/ping'

require 'net/http'

Bundler.setup

Bundler.require(:default) if defined?(Bundler)

$config_logger = Logger.new("./log/dev.log")



  class Dbconnect

     def  initialize
       @settings = YAML.load_file('./config/database.yml')["development"]
       @adapter = @settings["adapter"] if @settings["adapter"]
       @database = @settings["database"] if @settings["database"]
       @pool = @settings["pool"] if @settings["pool"]
       @timeout = @settings["timeout"] if @settings["timeout"]
    end

    def connect_to_db
      ActiveRecord::Base.establish_connection(
        :adapter => @adapter,
        :database => @database,
        :reconnect => @reconnect,
        :pool => @pool,
        :timeout => @timeout)

      $config_logger.info "\n  db Connected: to => #{@database}  "
    end
  end

end  

}


Example Gemfile : 


source "http://rubygems.org"

gem 'mail'

gem "escape_utils"

gem 'json',:require => "json"

gem 'json_pure'

gem 'resque'

gem 'resque-scheduler'

gem 'redis-namespace'

gem 'resque-status'

gem 'rake'

gem 'em-udns'

gem 'sqlite3'

gem 'spreadsheet'

gem 'activerecord', '3.2.1', :require => "active_record"

gem 'net-scp', :require => 'net/scp'

gem 'net-sftp', :require => 'net/sftp'

gem 'net-ssh', :require => 'net/ssh'

gem 'dir'

gem 'amatch'

gem 'haml'

gem 'net-ping'



gem install bundler

rest of the thing : bundle install .

Answer 3

Rails’ bootstrap is really slow; it is intended to be kept running, until certain time for restart (to eliminate some memory leaks there most likely is, any software is not bug-free), and is not intended to be used as a site that is launched for one request and then shut down.

That kind of usage more resembles a script. If you need to launch it with browser, you can easily use something like Erubis to write the page and use ActiveRecord in the script (I think it was useable outside of rails) or similar abstraction layer. Myself, for small tasks, I just use Mysql2.