nginx + passenger + rails 3.1 = 502 bad gateway?

I have the latest Nginx running with Passenger, SQLite and Rails 3.1. Somehow, when I have Passenger running for a while, I start getting "502 bad gateway" errors when visiting my website.

Here is a snippet from my Nginx error log:

2011/06/27 08:55:33 [error] 20331#0: *11270 upstream prematurely closed connection while reading response header from upstream, client: xxx.xxx.xx.x, server: www.example.com, request: "GET / HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "example.com"
2011/06/27 08:55:47 [info] 20331#0: *11273 client closed prematurely connection, so upstream connection is closed too while sending request to upstream, client: xxx.xxx.xx.x, server: www.example.com, request: "GET / HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "example.com"

Here is my passenger-status --show=backtraces output:

Thread 'Client thread 7':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 10':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 11':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 12':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 13':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 14':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 15':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 16':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 17':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 18':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 19':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 20':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 21':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 22':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 23':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'Client thread 24':
 in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
 in 'void Client::threadMain()' (HelperAgent.cpp:603)

Thread 'MessageServer thread':
 in 'void Passenger::MessageServer::mainLoop()' (MessageServer.h:537)

Thread 'MessageServer client thread 35':
 in 'virtual bool Passenger::BacktracesServer::processMessage(Passenger::MessageServer::CommonClientContext&, boost::shared_ptr<Passenger::MessageServer::ClientContext>&, const std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)' (BacktracesServer.h:47)
 in 'void Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:470)

This is what my passenger-memory-stats shows:

---------- Nginx processes ----------
PID    PPID   VMSize   Private  Name
-------------------------------------
16291  1      35.4 MB  0.1 MB   nginx: master process /home/apps/.nginx/sbin/nginx
16292  16291  36.0 MB  0.8 MB   nginx: worker process
16293  16291  35.8 MB  0.5 MB   nginx: worker process
16294  16291  35.8 MB  0.5 MB   nginx: worker process
16295  16291  35.8 MB  0.5 MB   nginx: worker process
### Processes: 5
### Total private dirty RSS: 2.46 MB


----- Passenger processes ------
PID    VMSize    Private   Name
--------------------------------
16251  87.0 MB   0.3 MB    PassengerWatchdog
16254  100.4 MB  1.3 MB    PassengerHelperAgent
16256  41.6 MB   5.7 MB    Passenger spawn server
16259  134.8 MB  0.8 MB    PassengerLoggingAgent
18390  770.4 MB  17.1 MB   Passenger ApplicationSpawner: /home/apps/manager/current
18415  853.3 MB  147.7 MB  Rack: /home/apps/manager/current
18424  790.5 MB  57.2 MB   Rack: /home/apps/manager/current
18431  774.7 MB  18.7 MB   Rack: /home/apps/manager/current
### Processes: 8
### Total private dirty RSS: 248.85 MB

It seems there is an issue with my the communication between Passenger and Nginx?

Also, looking at the Rails logs, it is clear that the request never reaches Rails at all, as there are no log entries for visits that get the 502 error. So my initial thought of something being wrong with any Rack middleware should not be possible.

回答1:

The "V" in VM is for Virtual. See also answers on other SO questions, e.g. Virtual Memory Usage from Java under Linux, too much memory used.

That top 147 MB does not hint of anything unusual whatsoever. Your 502 errors mean something else is wrong with the worker processes from Passenger's point of view. You should check your Rails & Nginx log files for clues, and perhaps passenger-status --show=backtraces.

回答2:

Try setting passenger_spawn_method conservative -- apparently there are issues with Passenger default forking settings and Rails 3.1

回答3:

I just meet such deadly "502 Bad Gateway error" reported by nginx, web stack is Ubuntu 12.04 + Rails 3.2.9 + Passenger 3.0.18 + nginx 1.2.4, it spent me 2 hours to found the root cause:

My rails application no need database support, so I just remove the gem 'sqlite3' in the Gemfile, it works fine in development mode, but will lead 502 Bad Gateway in production mode.

So after add back gem 'sqlite3' in Gemfile, such 502 Bad Gateway error disappear....

回答4:

I had the same problem and in my case it helped to increase the passenger_max_pool_size setting in the Nginx configuration file.

Maybe you can also take a look on the following postings which also helped me finding this solution:

http://whowish-programming.blogspot.com/2011/10/nginx-502-bad-gateway.html
https://groups.google.com/forum/?fromgroups#!topic/phusion-passenger/fgj6_4VdlLo and
https://groups.google.com/forum/?fromgroups#!topic/phusion-passenger/wAQDCrFHHgE

回答5:

It was the same for me in Rails 4, but I have added a "SECRETKEYBASE" in /confirg/secrets.yml

production:
secretkeybase: # add yours here