How to parse multiple nested sub-commands using py

Question

I am implementing a command line program which has interface like this:

cmd [GLOBAL_OPTIONS] {command [COMMAND_OPTS]} [{command [COMMAND_OPTS]} ...]

I have gone through the argparse documentation. I can implement GLOBAL_OPTIONS as optional argument using add_argument in argparse. And the {command [COMMAND_OPTS]} using Sub-commands.

From the documentation it seems I can have only one sub-command. But as you can see I have to implement one or more sub-commands. What is the best way to parse such command line arguments useing argparse?

Answer 1

You could try arghandler. This is an extension to argparse with explicit support for subcommands.

Answer 2

The solution provide by @Vikas fails for subcommand-specific optional arguments, but the approach is valid. Here is an improved version:

import argparse

# create the top-level parser
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--foo', action='store_true', help='foo help')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')

# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')

# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')
parser_b.add_argument('--baz', choices='XYZ', help='baz help')

# parse some argument lists
argv = ['--foo', 'command_a', '12', 'command_b', '--baz', 'Z']
while argv:
    print(argv)
    options, argv = parser.parse_known_args(argv)
    print(options)
    if not options.subparser_name:
        break

This uses parse_known_args instead of parse_args. parse_args aborts as soon as a argument unknown to the current subparser is encountered, parse_known_args returns them as a second value in the returned tuple. In this approach, the remaining arguments are fed again to the parser. So for each command, a new Namespace is created.

Note that in this basic example, all global options are added to the first options Namespace only, not to the subsequent Namespaces.

This approach works fine for most situations, but has three important limitations:

• It is not possible to use the same optional argument for different subcommands, like myprog.py command_a --foo=bar command_b --foo=bar.
• It is not possible to use any variable length positional arguments with subcommands (nargs='?' or nargs='+' or nargs='*').
• Any known argument is parsed, without 'breaking' at the new command. E.g. in PROG --foo command_b command_a --baz Z 12 with the above code, --baz Z will be consumed by command_b, not by command_a.

These limitations are a direct limitation of argparse. Here is a simple example that shows the limitations of argparse -even when using a single subcommand-:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('spam', nargs='?')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')

# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')

# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')

options = parser.parse_args('command_a 42'.split())
print(options)

This will raise the error: argument subparser_name: invalid choice: '42' (choose from 'command_a', 'command_b').

The cause is that the internal method argparse.ArgParser._parse_known_args() it is too greedy and assumes that command_a is the value of the optional spam argument. In particular, when 'splitting' up optional and positional arguments, _parse_known_args() does not look at the names of the arugments (like command_a or command_b), but merely where they occur in the argument list. It also assumes that any subcommand will consume all remaining arguments. This limitation of argparse also prevents a proper implementation of multi-command subparsers. This unfortunately means that a proper implementation requires a full rewrite of the argparse.ArgParser._parse_known_args() method, which is 200+ lines of code.

Given these limitation, it may be an options to simply revert to a single multiple-choice argument instead of subcommands:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--bar', type=int, help='bar help')
parser.add_argument('commands', nargs='*', metavar='COMMAND',
                 choices=['command_a', 'command_b'])

options = parser.parse_args('--bar 2 command_a command_b'.split())
print(options)
#options = parser.parse_args(['--help'])

It is even possible to list the different commands in the usage information, see my answer https://stackoverflow.com/a/49999185/428542

Answer 3

You can always split up the command-line yourself (split sys.argv on your command names), and then only pass the portion corresponding to the particular command to parse_args -- You can even use the same Namespace using the namespace keyword if you want.

Grouping the commandline is easy with itertools.groupby:

import sys
import itertools
import argparse    

mycommands=['cmd1','cmd2','cmd3']

def groupargs(arg,currentarg=[None]):
    if(arg in mycommands):currentarg[0]=arg
    return currentarg[0]

commandlines=[list(args) for cmd,args in intertools.groupby(sys.argv,groupargs)]

#setup parser here...
parser=argparse.ArgumentParser()
#...

namespace=argparse.Namespace()
for cmdline in commandlines:
    parser.parse_args(cmdline,namespace=namespace)

#Now do something with namespace...

untested