Why is the subprocess.Popen argument length limit

Question

I am running Python 3.4.3 on Linux 3.16.0. I want to use subprocess.Popen to run a command with a long single argument (a complex Bash invocation), roughly 200KiB.

According to getconf and xargs, this should be well within my limits:

$ getconf ARG_MAX
2097152
$ xargs --show-limits < /dev/null
Your environment variables take up 3364 bytes
POSIX upper limit on argument length (this system): 2091740
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088376
Size of command buffer we are actually using: 131072

However, Python fails with quite smaller limits:

>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (131072-4096)), shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x7f4613b58410>
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (262144-4096)), shell=True, executable='/bin/bash')
Traceback (most recent call last):
  [...]
OSError: [Errno 7] Argument list too long

Note that the Python limit is roughly the same as the "actually using" command buffer xargs reports. This suggests that xargs is somehow smart enough to start with a smaller limit and increase it as needed, but Python is not.

Questions:

1. Why is the Python limit smaller than the OS limit of 2MiB?
2. Can I increase the Python limit?
3. If so, how?

Answer 1

This other question is similar to yours, but for Windows. As in that scenario, you can bypass any shell limitations by avoiding the shell=True option.

Otherwise, you can provide a list of files to subprocess.Popen() as done in that scenario, and as suggested by @Aaron Digulla.

Answer 2

The maximum size for a single string argument is limited to 131072. It has nothing to do with python:

~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null
~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null
bash: /bin/echo: Argument list too long

It is actually the MAX_ARG_STRLEN that decides the max size for a single string:

And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'". (pointed out by Xan Lopez and Ralf Wildenhues)

See this discussion of ARG_MAX, under "Number of arguments and maximum length of one argument", and this question on unix.stackexchange.

You can see it in binfmts.h:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

~$ echo $(( $(getconf PAGE_SIZE)*32 )) 
131072

You can pass multiple strings of length 131071:

subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))

But a single string arg cannot be longer than 131071 bytes.