Os x terminal, ssh , and too many open files

问题:

Hi guys : I have some code that runs several rest queries over a connection that is ssh forwarded to an AWS machine (fyi: these queries are hitting a Solr server running on this machine), and the queries run against my localhost (which is forwarded to the AWS instance).

The code initially runs great getting data as necessary, but after running for a while, the code stalls (in eclipse).

At this exact moment, the terminal (i.e. where I have started my ssh tunnel) goes completely frozen, filling up with the String :

"accept : too many open files"

Because this infinite print is not associated with a bash terminal (i.e. I can't tell wether the ssh connection is still alive or not, and there is no text indicating which shell im in... just unpridled, relentless print statements) I cant tell wether it's coming from amazon, or from my client terminal.

I want to find the cause of this behavior and pinpoint the machine which is causing my terminal to explode

To test which of the two machines was causing the infinite print outs of the error, I ran the ulimit command on the server... and found that the max number of open files allowed (on the aws server) was well above the amount of open files (also determined using ulimit) at any given time while the client program (running from my ide) is executing.

I did the same test on my client , and found no significant increase in the number of open files.

Some side details : I'm running several hundreds of queries into a SOLR server that has over 100GB of data in a short time period.

Any hints on how to determine why my sshd mac os x terminal is dying and infinitely printing this message would be potentially very useful to me. Of course, wether or not they were specific to solr. That said, any insights into why this would happen when using a solr service may also help to solve this problem.

回答1:

You could try looking at ulimit (via Terminal type):

ulimit -a

Specifically, check the value for open files. On my machine (OS X) it reports 256. You might want to try increasing it to 512:

ulimit -n 512

回答2:

This seems to have worked for me:

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 4096' | sudo tee -a /etc/profile

Then restart OS X.

https://superuser.com/questions/302754/increase-the-maximum-number-of-open-file-descriptors-in-snow-leopard

回答3:

There isn't enough information here to be certain, but it sounds like ssh is hitting its per-process file descriptor limit while trying to accept connections to the forwarded socket locally, which in turn suggests that already open connections are not being closed in a timely fashion. You can run ssh with -d to see details of connections and disconnections; you might want to capture its stderr and use a script to track the socket operations as they'll be buried in a lot of other debug information.

Note that ssh itself may limit its open file descriptors beyond the default to avoid denial of service attacks on its forwarded port; I don't see any indication of this in the documentation, but it's the kind of thing I would expect.

回答4:

sudo launchctl limit maxfiles 1000000 unlimited

or

sudo sysctl -w kern.maxfilesperproc=1000000
sudo sysctl -w kern.maxfilesperproc=18000

To make the change permanent, use sudo to put your settings in /etc/sysctl.conf (which you may have to create), like this:

kern.maxfiles=20480 
kern.maxfilesperproc=18000

Note - select the number at your own risk

回答5:

The following command helped me,

launchctl limit maxfiles