How to write Rake task to import data to Rails app

Question

Goal: Using a CRON task (or other scheduled event) to update database with nightly export of data from an existing system.

All data is created/updated/deleted in an existing system. The website does no directly integrate with this system, so the rails app simply needs to reflect the updates that appear in the data export.

I have a .txt file of ~5,000 products that looks like this:

"1234":"product name":"attr 1":"attr 2":"ABC Manufacturing":"2222"
"A134":"another product":"attr 1":"attr 2":"Foobar World":"2447"
...

All values are strings enclosed in double quotes (") that are separated by colons (:)

Fields are:

• id: unique id; alphanumeric
• name: product name; any character
• attribute columns: strings; any character (e.g., size, weight, color, dimension)
• vendor_name: string; any character
• vendor_id: unique vendor id; numeric

Vendor information is not normalized in the current system.

What are best practices here? Is it okay to delete the products and vendors tables and rewrite with the new data on every cycle? Or is it better to only add new rows and update existing ones?

Notes:

1. This data will be used to generate Orders that will persist through nightly database imports. OrderItems will need to be connected to the product ids that are specified in the data file, so we can't rely on an auto-incrementing primary key to be the same for each import; the unique alphanumeric id will need to be used to join products to order_items.
2. Ideally, I'd like the importer to normalize the Vendor data
3. I cannot use vanilla SQL statements, so I imagine I'll need to write a rake task in order to use Product.create(...) and Vendor.create(...) style syntax.
4. This will be implemented on EngineYard

Answer 1

Since Products don't really change that often, the best way I would see is to update only the records that change.

1. Get all the deltas
2. Mass update using a single SQL statement

If you are having your normalization code in the models, you could use Product.create and Vendor.create or else it would be just a overkill. Also, Look into inserting multiple records in a single SQL transaction, its much faster.

Answer 2

• Create an importer rake task that is cronned
• Parse the file line by line using Faster CSV or via vanilla ruby like:

file.each do |line| products_array = line.split(":") end

• Split each line on the ":" and push in into a hash

• Use a find_or_initialize to populate your db such as:

  Product.find_or_initialize_by_name_and_vendor_id("foo", 111)

Answer 3

I wouldn't delete the products and vendors tables on every cycle. Is this a rails app? If so there are some really nice ActiveRecord helpers that would come in handy for you.

If you have a Product active record model, you can do:

p = Product.find_or_initialize_by_identifier(<id you get from file>)
p.name = <name from file>
p.size = <size from file>
etc...
p.save!

The find_or_initialize will lookup the product in the database by the id you specify, and if it can't find it, it will create a new one. The really handy thing about doing it this way, is that ActiveRecord will only save to the database if any of the data has changed, and it will automatically update any timestamp fields you have in the table (updated_at) accordingly. One more thing, since you would be looking up records by the identifier (id from the file), I would make sure to add an index on that field in the database.

To make a rake task to accomplish this, I would add a rake file to the lib/tasks directory of your rails app. We'll call it data.rake.

Inside data.rake, it would look something like this:

namespace :data do
  desc "import data from files to database"
  task :import => :environment do
    file = File.open(<file to import>)
    file.each do |line|
      attrs = line.split(":")
      p = Product.find_or_initialize_by_identifier(attrs[0])
      p.name = attrs[1]
      etc...
      p.save!
    end
  end
end

Than to call the rake task, use "rake data:import" from the command line.