mysterious rails error with almost no trace

Question

We're having a strange problem with one crawler. Occasionally it will throw a Rails FATAL error on some request, but the trace is very limited and looks something like this

[2014-07-01 18:16:37] FATAL Rails :
ArgumentError (invalid %-encoding (c ^   FK+ 9u$_    t  Kl
ΥE!   =k \  ̕* ߚ>c+<O   یo ʘ> C     R! 2 D  (5      x q#!` 4 p      |8 I   E
:+   H^9`^ #    Vo{   >

  =[z     )):
  lib/locale_middleware.rb:14:in `call'

The crawler user-agent is

Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)

We can ask it to stop crawling us via robots.txt, but it would be better to deal with the root cause and not fail with 500 on those requests if possible.

We can't really reproduce this kind of request either, so any suggestions on how to generate a similar request would be of great help.

We're using Rails 3.2.19, Unicorn on Ubuntu 12.04. Here's our locale_middleware.rb

Answer 1

Special thanks to Benjamin Sinclaire for pointing to the right issue on github.

The solution was described on this comment:

• Install the rack-robustness gem
• add this to application.rb:

config.middleware.use ::Rack::Robustness do |g|
  g.no_catch_all
  g.on(ArgumentError) { |ex| 400 }
  g.content_type 'text/plain'
  g.body{ |ex| ex.message }
  g.ensure(true) { |ex| env['rack.errors'].write(ex.message) }
end