Dealing with API rate limits?

问题:

I've an app that's set up to make scheduled calls to a number of APIs once a day. This works very nicely but i'm aware that some of the APIs i'm calling (Twitter for example) have a rate limit. As the number of calls i'm making is set to continually grow, can anyone recommend a way to throttle my calls so I can send in bursts of x per hour/minute etc?

I've found the Glutton Ratelimit gem, is anyone using this and is it any good? Are there others I should be looking at?

回答1:

If you're using some kind of background worker to perform your API calls, you could reschedule the task to be reperformed in the next time slot, when the rate limits have been reset.

class TwitterWorker
  include Sidekiq::Worker

  def perform(status_id)
    status = Twitter.status(status_id)
    # ...

  rescue Twitter::Error::TooManyRequests
    # Reschedule the query to be performed in the next time slot
    TwitterWorker.perform_in(15.minutes, status_id)
  end
end

No scientific solution though, there's e.g. the risk that a query might be rescheduled each time if you try to perform much more API calls in a day than the rate limit allows for. But until then, something easy might do the trick!

回答2:

Another solution is to buy proxies which allow you to send request with different IP addresses

Use standard http lib http://ruby-doc.org/stdlib-2.0/libdoc/net/http/rdoc/Net/HTTP.html#method-c-Proxy

I am not sure that you will not be blocked but maybe it is worth to try. Randomly choosen IP should increase your limits

回答3:

Unless you're making concurrent requests there's not much to it.

• Figure out how much delay you need per request
• Check the time before the request, subtract from the time after the request and sleep the rest.

With concurrent requests you can be more accurate, I once blogged about that here

回答4:

I know this is an old question, but wanted to mention something in case it helps others with the same question.

If the work can be queued to jobs using resque, then you could use the gem I've just released which pauses a queue when you hit a rate_limit - and unpauses it some time later.

https://github.com/pavoni/resque-rate_limited_queue