Which is the best way to allow configuration optio

Question

I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.

Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.

I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.

Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all optinos to the appropriate values...

Answer 1

I can't say it's the best way, but I have an OptionParser class that I made that does just that - acts like optparse.OptionParser with defaults coming from a config file section. You can have it...

class OptionParser(optparse.OptionParser):
    def __init__(self, **kwargs):
        import sys
        import os
        config_file = kwargs.pop('config_file',
                                 os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
        self.config_section = kwargs.pop('config_section', 'OPTIONS')

        self.configParser = ConfigParser()
        self.configParser.read(config_file)

        optparse.OptionParser.__init__(self, **kwargs)

    def add_option(self, *args, **kwargs):
        option = optparse.OptionParser.add_option(self, *args, **kwargs)
        name = option.get_opt_string()
        if name.startswith('--'):
            name = name[2:]
            if self.configParser.has_option(self.config_section, name):
                self.set_default(name, self.configParser.get(self.config_section, name))

Feel free to browse the source. Tests are in a sibling directory.

Answer 2

Try to this way

# encoding: utf-8
import imp
import argparse


class LoadConfigAction(argparse._StoreAction):
    NIL = object()

    def __init__(self, option_strings, dest, **kwargs):
        super(self.__class__, self).__init__(option_strings, dest)
        self.help = "Load configuration from file"

    def __call__(self, parser, namespace, values, option_string=None):
        super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)

        config = imp.load_source('config', values)

        for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
            setattr(namespace, key, getattr(config, key))

Use it:

parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")

And create example config:

# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")

Answer 3

Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.

Answer 4

fromfile_prefix_chars

Maybe not the perfect API, but worth knowing about. main.py:

#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())

Then:

$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py @opts.txt
Namespace(a='1', b='2')
$ ./main.py @opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 @opts.txt
Namespace(a='1', b='2')

Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars

Tested on Python 3.6.5, Ubuntu 18.04.

Answer 5

I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:

subp.set_defaults(**dict(conffile.items(subn)))

This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.

A more complete example is below:

####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost

import ConfigParser
import argparse

parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()

parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')

parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')

conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')

for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
    subp.set_defaults(**dict(conffile.items(subn)))

print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')

Answer 6

I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:

Default with no user input:

$ ./argparse-partial.py
Option is "default"

Default from configuration file:

$ cat argparse-partial.config 
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config 
Option is "Hello world!"

Default from configuration file, overridden by commandline:

$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"

argprase-partial.py follows. It is slightly complicated to handle -h for help properly.

import argparse
import ConfigParser
import sys

def main(argv=None):
    # Do argv default this way, as doing it in the functional
    # declaration sets it at compile time.
    if argv is None:
        argv = sys.argv

    # Parse any conf_file specification
    # We make this parser with add_help=False so that
    # it doesn't parse -h and print help.
    conf_parser = argparse.ArgumentParser(
        description=__doc__, # printed with -h/--help
        # Don't mess with format of description
        formatter_class=argparse.RawDescriptionHelpFormatter,
        # Turn off help, so we print all options in response to -h
        add_help=False
        )
    conf_parser.add_argument("-c", "--conf_file",
                        help="Specify config file", metavar="FILE")
    args, remaining_argv = conf_parser.parse_known_args()

    defaults = { "option":"default" }

    if args.conf_file:
        config = ConfigParser.SafeConfigParser()
        config.read([args.conf_file])
        defaults.update(dict(config.items("Defaults")))

    # Parse rest of arguments
    # Don't suppress add_help here so it will handle -h
    parser = argparse.ArgumentParser(
        # Inherit options from config_parser
        parents=[conf_parser]
        )
    parser.set_defaults(**defaults)
    parser.add_argument("--option")
    args = parser.parse_args(remaining_argv)
    print "Option is \"{}\"".format(args.option)
    return(0)

if __name__ == "__main__":
    sys.exit(main())