How to increase the limit of “maximum open files”

Question

The default limit for the max open files on Mac OS X is 256 (ulimit -n) and my application needs about 400 file handlers.

I tried to change the limit with setrlimit() but even if the function executes correctly, i'm still limited to 256.

Here is the test program I use:

#include <stdio.h>
#include <sys/resource.h>

main()
{
  struct rlimit rlp;

  FILE *fp[10000];
  int i;

  getrlimit(RLIMIT_NOFILE, &rlp);
  printf("before %d %d\n", rlp.rlim_cur, rlp.rlim_max);

  rlp.rlim_cur = 10000;
  setrlimit(RLIMIT_NOFILE, &rlp);

  getrlimit(RLIMIT_NOFILE, &rlp);
  printf("after %d %d\n", rlp.rlim_cur, rlp.rlim_max);

  for(i=0;i<10000;i++) {
    fp[i] = fopen("a.out", "r");
    if(fp[i]==0) { printf("failed after %d\n", i); break; }
  }

}

and the output is:

before 256 -1
after 10000 -1
failed after 253

I cannot ask the people who use my application to poke inside a /etc file or something. I need the application to do it by itself.

Answer 1

For some reason (perhaps binary compatibility), you have to define _DARWIN_UNLIMITED_STREAMS before including <stdio.h>:

#define _DARWIN_UNLIMITED_STREAMS

#include <stdio.h>
#include <sys/resource.h>

main()
{
  struct rlimit rlp;

  FILE *fp[10000];
  int i;

  getrlimit(RLIMIT_NOFILE, &rlp);
  printf("before %d %d\n", rlp.rlim_cur, rlp.rlim_max);

  rlp.rlim_cur = 10000;
  setrlimit(RLIMIT_NOFILE, &rlp);

  getrlimit(RLIMIT_NOFILE, &rlp);
  printf("after %d %d\n", rlp.rlim_cur, rlp.rlim_max);

  for(i=0;i<10000;i++) {
    fp[i] = fopen("a.out", "r");
    if(fp[i]==0) { printf("failed after %d\n", i); break; }
  }

}

prints

before 256 -1
after 10000 -1
failed after 9997

This feature appears to have been introduced in Mac OS X 10.6.

Answer 2

This may be a hard limitation of your libc. Some versions of solaris have a similar limitation because they store the fd as an unsigned char in the FILE struct. If this is the case for your libc as well, you may not be able to do what you want.

As far as I know, things like setrlimit only effect how many file you can open with open (fopen is almost certainly implemented in terms on open). So if this limitation is on the libc level, you will need an alternate solution.

Of course you could always not use fopen and instead use the open system call available on just about every variant of unix.

The downside is that you have to use write and read instead of fwrite and fread, which don't do things like buffering (that's all done in your libc, not by the OS itself). So it could end up be a performance bottleneck.

Can you describe the scenario that requires 400 files open ** simultaneously**? I am not saying that there is no case where that is needed. But, if you describe your use case more clearly, then perhaps we can recommend a better solution.

Answer 3

etresoft found the answer on the apple discussion board:

The whole problem here is your printf() function. When you call printf(), you are initializing internal data structures to a certain size. Then, you call setrlimit() to try to adjust those sizes. That function fails because you have already been using those internal structures with your printf(). If you use two rlimit structures (one for before and one for after), and don't print them until after calling setrlimit, you will find that you can change the limits of the current process even in a command line program. The maximum value is 10240.

Answer 4

Mac OS doesn't allow us to easily change the limit as in many of the unix based operating system. We have to create two files

/Library/LaunchDaemons/limit.maxfiles.plist /Library/LaunchDaemons/limit.maxproc.plist describing the max proc and max file limit. The ownership of the file need to be changed to 'root:wheel'

This alone doesn't solve the problem, by default latest version of mac OSX uses 'csrutil', we need to disable it. To disable it we need to reboot our mac in recovery mode and from there disable csrutil using terminal.

Now we can easily change the max open file handle limit easily from terminal itself (even in normal boot mode).

This method is explained in detail in the following link. http://blog.dekstroza.io/ulimit-shenanigans-on-osx-el-capitan/

works for OSX-elcapitan and OSX-Seirra.

Answer 5

I know that's sound a silly question, but you really need 400 files opened at the same time? By the way, are you running this code as root are you?

Answer 6

rlp.rlim_cur = 10000;

Two things.

1st. LOL. Apparently you have found a bug in the Mac OS X' stdio. If I fix your program up/add error handling/etc and also replace fopen() with open() syscall, I can easily reach the limit of 10000 (which is 240 fds below my 10.6.3' OPEN_MAX limit 10240)

2nd. RTFM: man setrlimit. Case of max open files has to be treated specifically regarding OPEN_MAX.