Parse config files, environment, and command-line

Question

Python's standard library has modules for configuration file parsing (configparser), environment variable reading (os.environ), and command-line argument parsing (argparse). I want to write a program that does all those, and also:

• Has a cascade of option values:

  • default option values, overridden by
  • config file options, overridden by
  • environment variables, overridden by
  • command-line options.

• Allows one or more configuration file locations specified on the command line with e.g. --config-file foo.conf, and reads that (either instead of, or additional to, the usual configuration file). This must still obey the above cascade.

• Allows option definitions in a single place to determine the parsing behaviour for configuration files and the command line.

• Unifies the parsed options into a single collection of option values for the rest of the program to access without caring where they came from.

Everything I need is apparently in the Python standard library, but they don't work together smoothly.

How can I achieve this with minimum deviation from the Python standard library?

Answer 1

The library confect I built is precisely to meet most of your needs.

• It can load configuration file multiple times through given file paths or module name.
• It loads configurations from environment variables with a given prefix.

• It can attach command line options to some click commands

  (sorry, it's not argparse, but click is better and much more advanced. confect might support argparse in the future release).

• Most importantly, confect loads Python configuration files not JSON/YMAL/TOML/INI. Just like IPython profile file or DJANGO settings file, Python configuration file is flexible and easier to maintain.

For more information, please check the README.rst in the project repository. Be aware of that it supports only Python3.6 up.

Examples

Attaching command line options

import click
from proj_X.core import conf

@click.command()
@conf.click_options
def cli():
    click.echo(f'cache_expire = {conf.api.cache_expire}')

if __name__ == '__main__':
    cli()

It automatically creates a comprehensive help message with all properties and default values declared.

$ python -m proj_X.cli --help
Usage: cli.py [OPTIONS]

Options:
  --api-cache_expire INTEGER  [default: 86400]
  --api-cache_prefix TEXT     [default: proj_X_cache]
  --api-url_base_path TEXT    [default: api/v2/]
  --db-db_name TEXT           [default: proj_x]
  --db-username TEXT          [default: proj_x_admin]
  --db-password TEXT          [default: your_password]
  --db-host TEXT              [default: 127.0.0.1]
  --help                      Show this message and exit.

Loading environment variables

It only needs one line to load environment variables

conf.load_envvars('proj_X')

Answer 2

It seems the standard library doesn't address this, leaving each programmer to cobble configparser and argparse and os.environ all together in clunky ways.

Answer 3

The argparse module makes this not nuts, as long as you're happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @, makes it so that,

my_prog --foo=bar

is equivalent to

my_prog @baz.conf

if @baz.conf is,

--foo
bar

You can even have your code look for foo.conf automatically by modifying argv

if os.path.exists('foo.conf'):
    argv = ['@foo.conf'] + argv
args = argparser.parse_args(argv)

The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.

Answer 4

To hit all those requirements, I would recommend writing your own library that uses both [opt|arg]parse and configparser for the underlying functionality.

Given the first two and the last requirement, I'd say you want:

Step one: Do a command line parser pass that only looks for the --config-file option.

Step two: Parse the config file.

Step three: set up a second command line parser pass using the output of the config file pass as the defaults.

The third requirement likely means you have to design your own option definition system to expose all the functionality of optparse and configparser that you care about, and write some plumbing to do conversions in between.