Create variable key/value pairs with argparse (pyt

问题:

I'm using argparse module to set my command line options. I'm also using a dict as a config in my application. Simple key/value store.

What I'm looking for is a possibility to override JSON options using command line arguments, without defining all possible arguments in advance. Something like --conf-key-1 value1 --conf-key-2 value2, which would create a dict {'key_1': 'value1','key_2': 'value2'} ('-' in the argument is replaced by '_' in the dict). Then I can combine this dict with my JSON config (dict).

So basically I would like to define --conf-* as an argument, where * can be any key and what comes after is the value.

I did find configargparse module, but as far as I can see I start with a dict I already use.

Any ideas how I could approach this?

回答1:

The first thing I'd try is use parse_known_args to handle other arguments, and handle the list of extras with my on routine. Adding the '--conf-' handling to argparse would be more work.

argv = '--conf-key-1 value1 --conf-key-2 value2'.split()
p = argparse.ArgumentParser()
args, extras = p.parse_known_args(argv)

def foo(astr):
    if astr.startswith('--conf-'):
        astr = astr[7:]
    astr = astr.replace('-','_')
    return astr

d = {foo(k):v for k,v in zip(extras[::2],extras[1::2])}
# {'key_1': 'value1', 'key_2': 'value2'}

The extras parsing could be more robust - making sure that there are proper pairs, rejecting badly formed keys, handling =.

Another approach would be to scan sys.argv for --conf- strings, and use those to construct add_argument statements.

keys = [k for k in argv if k.startswith('--conf-')]
p = argparse.ArgumentParser()
for k in keys:
    p.add_argument(k, dest=foo(k))
print vars(p.parse_args(argv))

---

If you would accept '--conf key1 value1 --conf key2 value2 ...' as the input, you could define

parser.add_argument('--conf', nargs=2, action='append')

which would produce:

namespace('conf': [['key1','value1'],['key2','value2']])

which could easily be turned into a dictionary. Or a custom Action could use setattr(namespace, values[0], values[1]) to enter the key/value pairs directly into the namespace.

I believe there have been SO question(s) about accepting '"key1:value" "key2:value2"' inputs.

回答2:

I had a similar issue and found a very workable pattern that works well with argparse (here three key-pairs: foo, bar and baz:

mycommand par1 --set foo=hello bar="hello world" baz=5

1. Defining the optional, multivalued argument

The set argument must be defined so:

import argparse
parser = argparse.ArgumentParser(description="...")
...
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+',
                        help="Set a number of key-value pairs "
                             "(do not put spaces before or after the = sign). "
                             "If a value contains spaces, you should define "
                             "it with double quotes: "
                             'foo="this is a sentence". Note that '
                             "values are always treated as strings.")
args = parser.parse_args()

The argument is optional and multivalued, with a minimum of one occurrence (nargs='+').

The result is a list of strings e.g. ["foo=hello", "bar=hello world", "baz=5"] in args.set, which we now need to parse (note how the shell has processed and removed the quotes!).

2. Parsing the result

For this we need 2 helper functions:

def parse_var(s):
    """
    Parse a key, value pair, separated by '='
    That's the reverse of ShellArgs.

    On the command line (argparse) a declaration will typically look like:
        foo=hello
    or
        foo="hello world"
    """
    items = s.split('=')
    key = items[0].strip() # we remove blanks around keys, as is logical
    if len(items) > 1:
        # rejoin the rest:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    """
    Parse a series of key-value pairs and return a dictionary
    """
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

At this point it is very simple:

# parse the key-value pairs
values = parse_vars(args.set)

You now have a dictionary:

values = {'foo':'hello', 'bar':'hello world', 'baz':'5'}

Note how the values are always returned as strings.

This method is also documented as a git gist.

回答3:

This can all be done much more simply using str.split(delim, limit):

class kvdictAppendAction(argparse.Action):
    """
    argparse action to split an argument into KEY=VALUE form
    on the first = and append to a dictionary.
    """
    def __call__(self, parser, args, values, option_string=None):
        assert(len(values) == 1)
        try:
            (k, v) = values[0].split("=", 2)
        except ValueError as ex:
            raise argparse.ArgumentError(self, f"could not parse argument \"{values[0]}\" as k=v format")
        d = getattr(args, self.dest) or {}
        d[k] = v
        setattr(args, self.dest, d)

...


myparser.add_argument("--keyvalue",
                      nargs=1,
                      action=kvdictAppendAction,
                      metavar="KEY=VALUE",
                      help="Add key/value params. May appear multiple times.")

回答4:

To simplify slightly on fralaus answer, the 2 methods can be combined into one easily.

Note: My docstring etc differ as I was using it for ansible extra_vars, but the core logic for string splitting came from fralaus' answer.

 def parse_vars(extra_vars):
     """
     Take a list of comma seperated key value pair strings, seperated
     by comma strings like 'foo=bar' and return as dict.
     :param extra_vars: list[str] ['foo=bar, 'key2=value2']

     :return: dict[str, str] {'foo': 'bar', 'key2': 'value2'}
     """
     vars_list = []
     if extra_vars:
         for i in extra_vars:
            items = i.split('=')
            key = items[0].strip()
            if len(items) > 1:
                value = '='.join(items[1:])
                vars_list.append((key, value))
     return dict(vars_list)

print parse_vars(args.set)

 $ test.py --set blah=gar one=too
>> {"blah": "gar", "one": "too"}