yaml and jinja2 reader

Question

I would like to be able to read in python a YAML jinja configuration file like using the PyYAML library but I'm receiving errors:

{% set name = "abawaca" %}
{% set version = "1.00" %}

package:
   name: {{ name }}
   version: {{ version }}

source:
   fn: {{ name }}-{{ version }}.tar.gz
   url: https://github.com/CK7/abawaca/archive/v{{ version }}.tar.gz
   sha256: 57465bb291c3a9af93605ffb11d704324079036205e5ac279601c9e98c467529

build:
   number: 0

requirements:
   build:
        - gcc   # [not osx]
        - llvm  # [osx]

Answer 1

Your input is not valid YAML, as you can easily check, e.g. here You should first expand the {% %} constructs, and then process the YAML, or you should make your file into valid YAML.

This is a partly consequence of choosing jinja2 for which the macro sequences {% ... %} start with a character ({) that has special meaning in YAML.

If you need to change the YAML, and write it out again, you can define your own delimiters and choose them so that don't have special meaning in YAML.

The {% %} you should put in a YAML comment block as at the top-level you have a mapping and should only have key-value pairs. One way to achieve that is by redefining the start as #% %# (you don't necessarily have to change the end, but I prefer the symmetry).

Then after updating, run the correct YAML through a small script that processes the file and replaces the delimiters to those that jinja2 understands, or tweak the environment, to change the actual definitions used by jinja2.

corrected data.yaml:

#% set name = "abawaca" %#
#% set version = "1.00" %#

package:
   name: <{ name }>
   version: 42

source:
   fn: <{ name }>-<{ version }>.tar.gz
   url: https://github.com/CK7/abawaca/archive/v<{ version }>.tar.gz
   sha256: 57465bb291c3a9af93605ffb11d704324079036205e5ac279601c9e98c467529

build:
   number: 0

requirements:
   build:
      - gcc   # [not osx]
      - llvm  # [osx]

This can be processed by:

import jinja2
from ruamel import yaml

yaml_file = 'data.yaml'
tmp_file = 'tmp.yaml'

data = yaml.round_trip_load(open(yaml_file))
data['package']['version'] = '<{ version }>'
with open(tmp_file, 'w') as fp:
    yaml.round_trip_dump(data, fp)

environment = jinja2.Environment(
    loader=jinja2.FileSystemLoader(searchpath='.'),
    trim_blocks=True,
    block_start_string='#%', block_end_string='%#',
    variable_start_string='<{', variable_end_string='}>')

    print(environment.get_template(tmp_file).render())

to give:

package:
  name: abawaca
  version: 1.00

source:
  fn: abawaca-1.00.tar.gz
  url: https://github.com/CK7/abawaca/archive/v1.00.tar.gz
  sha256: 57465bb291c3a9af93605ffb11d704324079036205e5ac279601c9e98c467529

build:
  number: 0

requirements:
  build:
  - gcc       # [not osx]
  - llvm      # [osx]

Please note that you have to use `ruamel.yaml (disclaimer: I am the author of that package), you cannot do this as easily with PyYAML as it throws away the comments on reading the YAML file. Since all of the jinja2 within comments occurs at the beginning of the file you can work around this with this particular example, but in general that will not be the case.