How to update yaml file using python

Question

I have a some.yaml file with the below contents.

    init_config: {}
    instances:
        - host: <IP>
          username: <username>
          password: <password>

The yaml file should be parsed and updated as below.

    init_config: {}
    instances:
        - host: 1.2.3.4
          username: Username
          password: Password

How do I parse the values and update them appropriately?

Answer 1

I don't know if you need YAML. Aside from using the YAML tag, it seems that you have no interest in the YAML document. So why not using Jinja2 or some template language?

from jinja2 import Template

tmpl = Template(u'''\
    init_config: {}
    instances:
         - host: {{ IP }}
           username: {{ username }}
           password: {{ password }}
''')

print tmpl.render(
     IP=u"1.2.3.4",
     username=u"Username",
     password=u"Password"
)

I don't know if it is a good idea, but if you only need to obtain a file with some fields changed, you don't need to actually parse the YAML document and can benefit from a Template language directly.

---

Bonus: Use case

I have worked with very complex YAML documents, for which there are tags unknown

...
  propertiesIDs: { 1, 2, 3, 4 }
  globalID: !myapplication.InterfaceID &primitiveID

replication: !myapplication.replication
  beginDate: 2012-09-10T20:00:03
  endDate: 2020-09-10T20:00:04
  replicant_uuid:
    ? 17169504-B6AB-11E4-8437-36E258BB2172
    ? 206B5842-B6AB-11E4-AAC3-36E258BB2172
...

Performing a valid parse of this document is difficult and time-consuming. I only need to populate some values, and the YAML is sent to a third-party application. So instead of parsing the YAML or trying to generate a valid document directly using pyyaml, is simpler (more time-efficient, less bug-prone) to generate it directly through templates. Moreover, template languages can easily be used with loops to populate dynamically sized fields.

Answer 2

The ruamel.yaml package was specifically enhanced (by me starting from PyYAML) to do this kind of round-trip, programmatic, updating.

If you start with (please note I removed the extra initial spaces):

init_config: {}
instances:
    - host: <IP>              # update with IP
      username: <username>    # update with user name
      password: <password>    # update with password

and run:

import ruamel.yaml

file_name = 'input.yml'
config, ind, bsi = load_yaml_guess_indent(open(file_name))

instances = config['instances']
instances[0]['host'] = '1.2.3.4'
instances[0]['username'] = 'Username'
instances[0]['password'] = 'Password'

with open('output.yml', 'w') as fp:
    yaml.dump(config, fp)

The output will be:

init_config: {}
instances:
    - host: 1.2.3.4           # update with IP
      username: Username      # update with user name
      password: Password      # update with password

The ordering of mapping keys (host, username and password), the style and the comments are preserved without any further specific action.

Instead of having the indent and block sequence indent guessed, you can do a manual traditional load, and set the indent values yourself:

yaml = ruamel.yaml.YAML()
yaml.indent(mapping=6, sequence=4)
with open(file_name) as fp:
    config = yaml.load(open(file_name))

---

If you look at the history of this answer, you can see how to do this with a more limited, PyYAML like, API.

Answer 3

Here's how i generate docker-crane templates for dev, production, stage, etc...

1. mkdir crane_templates
2. touch crane_templates/init.py
3. Add template content with nano crane_templates/some.yaml
4. Nano crane_gen.py

--- crane_gen.py ---

#!/usr/bin/env python
from jinja2 import Environment, PackageLoader

env = Environment(loader=PackageLoader('crane_templates', './'))
tmpl = env.get_template('crane.yaml.tmpl')

result = tmpl.render(
     IP=u"1.2.3.4",
     username=u"Username",
     password=u"Password"
)

5. python crane_gen.py > result.yaml

Answer inspired by @MariusSiuram

Answer 4

This is how i can read from the above file i mentioned, parse and update as needed.

import yaml

fname = "some.yaml"

stream = open(fname, 'r')
data = yaml.load(stream)

data['instances'][0]['host'] = '1.2.3.4'
data['instances'][0]['username'] = 'Username'
data['instances'][0]['password'] = 'Password'

with open(fname, 'w') as yaml_file:
    yaml_file.write( yaml.dump(data, default_flow_style=False))

Answer 5

Here are sample using PyYaml. As I understand you have something like template in yaml format, and you have to substitute places in angle brackets with actual values.

import yaml

s = """
    init_config: {}
    instances:
        - host: <IP>
          username: <username>
          password: <password>
"""

dict_obj = yaml.load(s) # loads string in internal data structure - dict
dict_obj['instances'][0]['host'] = 'localhost' # change values
print yaml.dump(dict_obj) # dumps dict to yaml format back