Ansible playbook fails to lock apt

Question

I took over a project that is running on Ansible for server provisioning and management. I'm fairly new to Ansible but thanks to the good documentation I'm getting my head around it. Still I'm having an error which has the following output:

failed: [build] (item=[u'software-properties-common', u'python-pycurl', u'openssh-server', u'ufw', u'unattended-upgrades', u'vim', u'curl', u'git', u'ntp']) => {"failed": true, "item": ["software-properties-common", "python-pycurl", "openssh-server", "ufw", "unattended-upgrades", "vim", "curl", "git", "ntp"], "msg": "Failed to lock apt for exclusive operation"}

The playbook is run with sudo: yes so I don't understand why I'm getting this error (which looks like a permission error). Any idea how to trace this down?

- name: "Install very important packages"
  apt: pkg={{ item }} update_cache=yes state=present
  with_items:
    - software-properties-common # for apt repository management
    - python-pycurl # for apt repository management (Ansible support)
    - openssh-server
    - ufw
    - unattended-upgrades
    - vim
    - curl
    - git
    - ntp

playbook:

- hosts: build.url.com
  sudo: yes
  roles:
    - { role: postgresql, tags: postgresql }
    - { role: ruby, tags: ruby }
    - { role: build, tags: build }

Answer 1

I just had the same issue on a new VM. I tried many approaches, including retrying the apt commands, but in the end the only way to do this was by removing unattended upgrades.

I'm using raw commands here, since at this point the VM doesn't have Python installed, so I need to install that first, but I need a reliable apt for that.

Since it is a VM and I was testing the playbook by resetting it to a Snapshot, the system date was off, which forced me to use the date -s command in order to not have problems with the SSL certificate during apt commands. This date -s triggered an unattended upgrade.

So this snippet of a playbook is basically the part relevant to disabling unattended upgrades in a new system. They are the first commands I'm issuing on a new system.

- name: Disable timers for unattended upgrade, so that none will be triggered by the `date -s` call.
  raw: systemctl disable --now {{item}}
  with_items:
    - 'apt-daily.timer'
    - 'apt-daily-upgrade.timer'

- name: Reload systemctl daemon to apply the new changes
  raw: systemctl daemon-reload

# Syncing time is only relevant for testing, because of the VM's outdated date.
#- name: Sync time
#  raw: date -s "{{ lookup('pipe', 'date') }}"

- name: Wait for any possibly running unattended upgrade to finish
  raw: systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true

- name: Purge unattended upgrades
  raw: apt-get -y purge unattended-upgrades    

- name: Update apt cache
  raw: apt-get -y update

- name: If needed, install Python
  raw: test -e /usr/bin/python || apt-get -y install python

Anything else would cause apt commands to randomly fail because of locking issues caused by unattended upgrades.

Answer 2

This is a very common situation when provisioning Ubuntu (and likely some other distributions). You try to run Ansible while automatic updates are running in background (which is what happens right after setting up a new machine). As APT uses semaphore, Ansible gets kicked out.

The playbook is ok and the easiest way to verify is to run it later (after automatic update process finishes).

For a permanent resolution, you might want to:

• use an OS image with automatic updates disabled
• add an explicit loop in the Ansible playbook to repeat the failed task until it succeeds