Python argparse Optional argument only works when

Question

In my script I have 3 positional arguments and 1 optional argument. One of the three positional arguments is required and the rest is optional (as specified using narg='?').

My optional argument doesn't pass any other arguments (action ='store_true') and is just there to enable sorting which will be implemented at a later time.

However, my problem is that my optional argument only works when it is the first or last argument in the script call.

Below is my script so far:

parser = argparse.ArgumentParser()
parser.add_argument("-s", "--sort", help="option to sort the ip addresses", action='store_true')
parser.add_argument("file_name", help="Name of the file containing the tcpdump")
parser.add_argument("source_IP", nargs='?', type=str, help="Source ip to search") 
parser.add_argument("dest_IP", nargs='?', type=str, help="Destination ip to search")

args = parser.parse_args()    

If I enter my -s between any of the other positional arguments I get an error.

 Ex: ./my_script file.txt -s 192.168.152.32 192.168.0.25

usage: 1 [-h] [-s] file_name [source_IP] [dest_IP]
1: error: unrecognized arguments: 192.168.152.32 192.168.0.25

My goal is to be able to enter my optional argument (-s) anywhere in the script call and have it working.

Answer 1

You have three positional arguments, but two of them are also optional thanks to nargs='?'. argparse is getting screwed up because it sees the positional filename, and then has to choose arbitrarily between interpreting the -s as the optional positional source, or as the switch. Either interpretation is valid (it's not doing complicated backtracking parsing to try to find some legal interpretation of the arguments that would allow it to complete; doing so with some argument types could lead to very bad behavior, like opening a file, then backtracking, closing it, and opening something else).

Short answer: In general, optional arguments should be either all positional, or all switches. Mixing and matching introduces complications that would make parsing a complicated recursive process that could only heuristically guess at the correct parsing (particularly with nargs='*' and nargs='+', but even '?' causes problems as you see). Removing the nargs qualifier from source and dest, or leaving them optional and converting to switches will allow -s to be passed in whatever order you like.

Answer 2

This problem is a subtile one, and requires a good understanding of how argparse parses postionals and optionals.

If all the positionals took one argument (the default nargs), then the -s could occur anywhere - start, end, or between any positional.

What happens with:

./my_script file.txt -s 192.168.152.32 192.168.0.25

is that source_IP,dest_IP are both consumed (and set to their defaults) when file_name is parsed. It then handles -s. Now there are 2 strings left, but no positionals to consume them, hence the unrecognized arguments error. Note that ./my_script file.txt runs fine, as does ./my_script file.txt -s. In all 3 cases, the 2 IP arguments are consumed at the same time as file_name.

parse_args alternates between consuming positionals and optionals. It will consume as many positionals at time as there strings (think of a greedy regex expression). Since source_IP and dest_IP are ok with 0 args, all three are consumed the first time it handles positionals.

There isn't a neat fix for the user, except to be wary of using nargs='?' positionals.

There is a bug/issue that tries to fix this. But the fix isn't trivial. The parser has to 'look ahead', noting that it could delay parsing these '?' arguments.

http://bugs.python.org/issue15112
argparse: nargs='*' positional argument doesn't accept any items
   if preceded by an option and another positional

Your parser runs fine with the argparse patched as proposed in that issue. But there's quite a backlog of potential patches for argparse.