Ruby Eventmachine & HTTP requests via proxy

2019-08-08 21:00发布

问题:

I'm coding on a high scalable web harvester running on top of Eventmachine. All works fine and fast. Recently I'm trying to fire the requests through a bunch of proxies which also works fine, e.g.:

EventMachine.run do
  connect_opts = { :proxy => { :host => '11.12.13.14', :port => 3128 } }
  request_opts = { :proxy => { :authorization => ['jdoe', 'mysecretpass'] } }
  req = EventMachine::HttpRequest.new('http://www.example.com/', connect_opts).get request_opts
  req.callback { }
end

I'm iterating over hundreds of proxies and firing several hundred requests per second. What I need now is to know which proxy was used for which request to store this metadata in a db. How would you fetch this information from the req object?

I hoped there's is some method like req.proxy_foo (equivalent to req.response, req.response_header etc.) to get this information but didn't find any appropriate way yet.