argparse - Combining parent parser, subparsers and

Question

I wanted to define different subparsers in a script, with both inheriting options from a common parent, but with different defaults. It doesn't work as expected, though.

Here's what I did:

import argparse

# this is the top level parser
parser = argparse.ArgumentParser(description='bla bla')

# this serves as a parent parser
base_parser = argparse.ArgumentParser(add_help=False)
base_parser.add_argument('-n', help='number', type=int)


# subparsers
subparsers = parser.add_subparsers()
subparser1= subparsers.add_parser('a', help='subparser 1', 
                                   parents=[base_parser])
subparser1.set_defaults(n=50)
subparser2 = subparsers.add_parser('b', help='subparser 2',
                                   parents=[base_parser])
subparser2.set_defaults(n=20)

args = parser.parse_args()
print args

When I run the script from the command line, this is what I get:

$ python subparse.py b
Namespace(n=20)

$ python subparse.py a
Namespace(n=20)

Apparently, the second set_defaults overwrites the first one in the parent. Since there wasn't anything about it in the argparse documentation (which is pretty detailed), I thought this might be a bug.

Is there some simple solution for this? I could check the args variable afterwards and replace None values with the intended defaults for each subparser, but that's what I expected argparse to do for me.

This is Python 2.7, by the way.

Answer 1

What's happening

The problem here is that parser arguments are objects, and when a parser inherits from it's parents, it adds a reference to the parent's action to it's own list. When you call set_default, it sets the default on this object, which is shared across the subparsers.

You can examine the subparsers to see this:

>>> a1 = [ action for action in subparser1._actions if action.dest=='n' ].pop()
>>> a2 = [ action for action in subparser2._actions if action.dest=='n' ].pop()
>>> a1 is a2 # same object in memory
True
>>> a1.default
20
>>> type(a1)
<class 'argparse._StoreAction'>

First solution: Explicitly add this argument to each subparser

You can fix this by adding the argument to each subparser separately rather than adding it to the base class.

subparser1= subparsers.add_parser('a', help='subparser 1', 
                               parents=[base_parser])
subparser1.add_argument('-n', help='number', type=int, default=50)
subparser2= subparsers.add_parser('b', help='subparser 2', 
                               parents=[base_parser])
subparser2.add_argument('-n', help='number', type=int, default=20)
...

Second solution: multiple base classes

If there are many subparsers which share the same default value, and you want to avoid this, you can create different base classes for each default. Since parents is a list of base classes, you can still group the common parts into another base class, and pass the subparser multiple base classes to inherit from. This is probably unnecessarily complicated.

import argparse

# this is the top level parser
parser = argparse.ArgumentParser(description='bla bla')

# this serves as a parent parser
base_parser = argparse.ArgumentParser(add_help=False)
# add common args

# for group with 50 default
base_parser_50 = argparse.ArgumentParser(add_help=False)
base_parser_50.add_argument('-n', help='number', type=int, default=50)

# for group with 50 default
base_parser_20 = argparse.ArgumentParser(add_help=False)
base_parser_20.add_argument('-n', help='number', type=int, default=20)

# subparsers
subparsers = parser.add_subparsers()
subparser1= subparsers.add_parser('a', help='subparser 1', 
                                   parents=[base_parser, base_parser_50])

subparser2 = subparsers.add_parser('b', help='subparser 2',
                                   parents=[base_parser, base_parser_20])

args = parser.parse_args()
print args

First solution with shared args

You can also share a dictionary for the arguments and use unpacking to avoid repeating all the arguments:

import argparse

# this is the top level parser
parser = argparse.ArgumentParser(description='bla bla')

n_args = '-n',
n_kwargs = {'help': 'number', 'type': int}

# subparsers
subparsers = parser.add_subparsers()
subparser1= subparsers.add_parser('a', help='subparser 1')
subparser1.add_argument(*n_args, default=50, **n_kwargs)

subparser2 = subparsers.add_parser('b', help='subparser 2')
subparser2.add_argument(*n_args, default=20, **n_kwargs)

args = parser.parse_args()
print args

Answer 2

set_defaults loops through the actions of the parser, and sets each default attribute:

   def set_defaults(self, **kwargs):
        ...
        for action in self._actions:
            if action.dest in kwargs:
                action.default = kwargs[action.dest]

Your -n argument (an action object) was created when you defined the base_parser. When each subparser is created using parents, that action is added to the ._actions list of each subparser. It doesn't define new actions; it just copies pointers.

So when you use set_defaults on subparser2, you modify the default for this shared action.

This Action is probably the 2nd item in the subparser1._action list (h is the first).

 subparser1._actions[1].dest  # 'n'
 subparser1._actions[1] is subparser2._actions[1]  # true

If that 2nd statement is True, that means the same action is in both lists.

If you had defined -n individually for each subparser, you would not see this. They would have different action objects.

I'm working from my knowledge of the code, not anything in the documentation. It was pointed out recently in Cause Python's argparse to execute action for default that the documentation says nothing about add_argument returning an Action object. Those objects are an important part of the code organization, but they don't get much attention in the documentation.

---

Copying parent actions by reference also creates problems if the 'resolve' conflict handler is used, and the parent needs to be reused. This issue was raised in

argparse conflict resolver for options in subcommands turns keyword argument into positional argument

and Python bug issue:

http://bugs.python.org/issue22401

A possible solution, for both this issue and that, is to (optionally) make a copy of the action, rather than share the reference. That way the option_strings and defaults can be modified in the children without affecting the parent.