Argparse: mixing parent parser with subparsers

Question

I want to write a simple tool that takes an arbitrary number of input files and performs one operation on each of them. The syntax is stupidly simple:

mytool operation input1 input2 ... inputN

Some of these operations may require an extra argument

mytool operation op_argument input1 input2 ... inputN 

In addition to this I'd like the users to be able to specify whether the operations should be performed in place, and to specify the target directory of the output.

mytool -t $TARGET --in-place operation op_argument input1 input2 input3

And as a very last requirement, I'd like users to be able to get help on each operation individually, as well as on the usage of the tool as a whole.

Here's my attempt at designing an Argument Parser for said tool, together with a Minimal, Complete, Verifiable Example:

#!/bin/env python

import argparse
from collections import namedtuple

Operations = namedtuple('Ops', 'name, argument, description')
IMPLEMENTED_OPERATIONS = {'echo': Operations('echo',
                                             None,
                                             'Echo inputs'),
                          'fancy': Operations('fancy',
                                              'fancyarg',
                                              'Do fancy stuff')}


if __name__ == "__main__":

    # Parent parser with common stuff.
    parent = argparse.ArgumentParser(add_help=False)
    parent.add_argument('-t', '--target-directory', type=str, default='.',
                        help="An output directory to store output files.")
    parent.add_argument('-i', '--in-place', action='store_true',
                        help="After succesful execution, delete the inputs.")
    # The inputfiles should be the very last positional argument.
    parent.add_argument('inputfiles', nargs='*', type=argparse.FileType('r'),
                        help="A list of input files to operate on.")

    # Top level parser.
    top_description = "This is mytool. It does stuff"
    parser = argparse.ArgumentParser(prog="mytool",
                                     description=top_description,
                                     parents=[parent])

    # Operation parsers.
    subparsers = parser.add_subparsers(help='Sub-command help', dest='chosen_op')

    op_parsers = {}
    for op_name, op in IMPLEMENTED_OPERATIONS.items():
        op_parsers[op_name] = subparsers.add_parser(op_name,
                                                    description=op.description,
                                                    parents=[parent])
        if op.argument is not None:
            op_parsers[op_name].add_argument(op.argument)

    args = parser.parse_args()
    op_args = {}
    for key, subparser in op_parsers.items():
        op_args[key] = subparser.parse_args()


    print(args.chosen_op)

The problem I have is that the order of the positional arguments is wrong. Somehow, the way I implemented this makes Argparse think that the operation (and its op_argument) should come after the input files, which is obviously not the case.

How can I have the parent positional argument, in my case the inputfiles, as the last positional argument?

Answer 1

To the main parser, subparsers is just another positional argument, but with a unique nargs ('+...'). So it will look for the inputfiles arguments first, and then allocate any left overs to subparsers.

Mixing positionals with subparsers is tricky. It is best to define inputfiles as an argument for each subparser.

parents can make it easy to add the same set of arguments to several subparsers- however those arguments will added first.

So I think you want:

for op_name, op in IMPLEMENTED_OPERATIONS.items():
        op_parsers[op_name] = subparsers.add_parser(op_name,
                                                    description=op.description,
                                                    parents=[parent])
        if op.argument is not None:
            op_parsers[op_name].add_argument(op.argument)

        op_parsers[op_name].add_argument('inputfiles', nargs='*', type=argparse.FileType('r'),
                        help="A list of input files to operate on.")

As for the help, the normal behavior is to get help for the main parser, or for each subparser. Combining those into one display has been the topic of several SO questions. It's possible but not easy.

---

The main parser handles input strings in order - flags, positionals etc. When it handles the subparsers positional, it hands the task of to the name subparser, along with all remaining commandline strings. The subparser then acts like a new independent parser, and returns a namespace to the main parser to be incorporated into the main namespace. The main parser does not resume parsing the commandline. So the subparser action is always last.