How are people testing opsworks cookbooks?

Question

I have a fairly complex deployment of dynamically configured instances of a rail apps on nginx using passenger that I currently use Capistrano for. I am trying to automate the entire process on Opsworks from creating the stack, layers, instances and dynamic app deployment. None of the existing Opsworks recipes come close to what I need so I started my own custom cookbooks. I am trying to test them using test-kitchen and vagrant but the Opsworks cookbooks have too many custom dependancies to be used locally. So it seems like I am stuck either re-inventing the wheel by using only my own recipes that don't depend on Opsworks cookbooks or trying to test my cookbooks on AWS which is slow and expensive.

Is anybody doing any custom cookbook development on Opsworks and if so how are you testing it? Are you using any community cookbooks and if so how painful have they been to get working on Opsworks?

Answer 1

Opsworks cookbooks build upon their custom boxes, so getting a local VM set up where you can run their cookbooks alongside yours is a challenge,

Fortunately their Ubuntu box has been recreated by this project. https://github.com/wwestenbrink/vagrant-opsworks

Building on top of this, I've set up a local vagrant environment that runs the opsworks cookbooks and one of my own. https://github.com/erupenkman/opsworks-example

With this you can actually test in an almost identical local environment before deployment.

Answer 2

Opsworks now supports Chef 11.10 and Berkshelf (http://berkshelf.com/) to manage dependencies which makes using custom cookbooks so. much. easier.

I've been pretty happy with Test Kitchen (https://github.com/test-kitchen/test-kitchen) with Vagrant & Chef Solo for local testing.

Additionally you can manually run particular recipes on an Opsworks stack from the stack settings page. Have a look here: http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-manual.html.

You can also run cookbook tests during an Opsworks deployment with some custom JSON: http://docs.aws.amazon.com/opsworks/latest/userguide/troubleshoot-debug-test.html

Answer 3

Checkout Travis-ci and sous-chef you may find this suitable for testing cookbooks etc and they have a deploy configuration for opsworks too.

https://github.com/michaelklishin/sous-chef http://docs.travis-ci.com/user/deployment/opsworks/

Answer 4

I was able to solve this is issue using Chef+Kitchen+Vagrant

Here's my wrapper: https://github.com/elitechance/opsworks-cookbooks-wrapper

Below is how to test

git clone https://github.com/elitechance/opsworks-cookbooks-wrapper.git

git clone https://github.com/aws/opsworks-cookbooks.git

cd opsworks-cookbooks-wrapper

cp .kitchen.sample.yml .kitchen.yml

cp Berksfile.sample Berksfile

cp metadata.sample.rb metadata.rb

The last command will simulate OpsWorks NodeJs Setup Recipes, see .kitchen.sample.yml for details.

$ chef exec kitchen setup

KNOWN BUGS

The first time you run $ chef exec kitchen setup, you will encounter errors saying:

Running handlers: [2016-04-08T17:08:34+00:00] ERROR: Running exception handlers Running handlers complete [2016-04-08T17:08:34+00:00] ERROR: Exception handlers complete Chef Client failed. 43 resources updated in 01 minutes 02 seconds [2016-04-08T17:08:34+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out [2016-04-08T17:08:34+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report [2016-04-08T17:08:34+00:00] ERROR: ruby_blockFallback for remote_file[/tmp/rubygems-2.2.2.tgz] had an error: NoMethodError: remote_file/tmp/rubygems-2.2.2.tgz had an error: NoMethodError: undefined method `to_sym' for [:create]:Array [2016-04-08T17:08:34+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Just run again:

$ chef exec kitchen setup

Answer 5

We are using Opsworks with a lot of custom cookbooks, and vagrant as well. While the test-coverage is far from perfect, it works pretty smooth, adapting community cookbooks as well. Our cookbook repository is public: https://github.com/till/easybib-cookbooks

The interesting bits of this repo are:

• We use a role-recipe as the entry point for each layer instead of specifiying multiple recipes with the layer in opsworks: https://github.com/till/easybib-cookbooks/blob/master/easybib/recipes/role-nginxapp-api.rb
• We check if the recipe run is in AWS or in Vagrant using a custom library function: https://github.com/till/easybib-cookbooks/blob/master/easybib/libraries/easybib.rb#L208-L216
• All opsworks-specific cookbooks/providers are usually only used in the deploy-cookbook, which we only include when is_aws is true, otherwise we fall back for a default application deployment.

Browsing our cookbooks should give you some more hints for your questions. We test our cookbooks with Travis, and do not test the opsworks cookbooks at all. Although I have to note that I am currently playing around there with some ideas how to integrate them in our test runs, since without their providers, testing our deploy cookbook is pretty much impossible.

Answer 6

There are a lot of useful suggestions on this page but I would HIGHLY recommend everyone check out Mike Greiling's blog post Simplify OpsWorks Development With Packer and his github repo opsworks-vm which help you to mock the entire opsworks stack including the install of the opsworks agent so you can also test app deploy recipes, multiple layers, multiple instances at the same time, etc . It is extremely impressive.

Quick Start on Ubuntu 14.04

NOTE: This can NOT be done from an ubuntu virtual machine because virtualbox does not support nested virtualization of 64-bit machines.

1. Install ChefDK
   1. mkdir /tmp/packages && cd /tmp/packages
   2. wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chefdk_0.8.1-1_amd64.deb
   3. sudo dpkg -i chefdk_0.8.0-1_amd64.deb
   4. cd /opt/chefdk/
   5. chef verify
   6. which ruby
   7. echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile && source ~/.bash_profile
2. Install VirtualBox
   1. echo 'deb http://download.virtualbox.org/virtualbox/debian vivid contrib' > /etc/apt/sources.list.d/virtualbox.list
   2. wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -
   3. sudo apt-get update -qqy
   4. sudo apt-get install virtualbox-5.0 dkms
3. Install Vagrant
   1. cd /tmp/packages
   2. wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.4_x86_64.deb
   3. sudo dpkg -i vagrant_1.7.4_x86_64.deb
   4. vagrant plugin install vagrant-berkshelf
   5. vagrant plugin install vagrant-omnibus
   6. vagrant plugin list
4. Install Packer
   1. mkdir /opt/packer && cd /opt/packer
   2. wget https://dl.bintray.com/mitchellh/packer/packer_0.8.6_linux_amd64.zip
   3. unzip packer_0.8.6_linux_amd64.zip
   4. echo 'PATH=$PATH:/opt/packer' >> ~/.bash_profile && source ~/.bash_profile
5. Build Mike Greiling's opsworks-vm virtualbox image using Packer
   1. mkdir ~/packer && cd ~/packer
   2. git clone https://github.com/pixelcog/opsworks-vm.git
   3. cd opsworks-vm
   4. rake build install
   5. This will install a new virtualbox vm to ~/.vagrant.d/boxes/ubuntu1404-opsworks/

To mock a single opsworks instance, create a new Vagrantfile like so:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu1404-opsworks"
  config.vm.provision :opsworks, type: 'shell', args: 'path/to/dna.json'
end

The dna.json file path is set relative to the Vagrantfile and should contain any JSON data you wish to send to OpsWorks Chef.

For example:

{
  "deploy": {
    "my-app": {
      "application_type": "php",
      "scm": {
        "scm_type": "git",
        "repository": "path/to/my-app"
      }
    }
  },
  "opsworks_custom_cookbooks": {
    "enabled": true,
    "scm": {
      "repository": "path/to/my-cookbooks"
    },
    "recipes": [
      "recipe[opsworks_initial_setup]",
      "recipe[dependencies]",
      "recipe[mod_php5_apache2]",
      "recipe[deploy::default]",
      "recipe[deploy::php]",
      "recipe[my_custom_cookbook::configure]"
    ]
  }
}

To mock multiple opsworks instances and include layers see his AWS OpsWorks "Getting Started" Example which includes the stack.json below.

Vagrantfile (for multiple instances)

Vagrant.configure("2") do |config|

  config.vm.box = "ubuntu1404-opsworks"

  # Create the php-app layer
  config.vm.define "app" do |layer|

    layer.vm.provision "opsworks", type:"shell", args:[
      'ops/dna/stack.json',
      'ops/dna/php-app.json'
    ]

    # Forward port 80 so we can see our work
    layer.vm.network "forwarded_port", guest: 80, host: 8080
    layer.vm.network "private_network", ip: "10.10.10.10"
  end

  # Create the db-master layer
  config.vm.define "db" do |layer|

    layer.vm.provision "opsworks", type:"shell", args:[
      'ops/dna/stack.json',
      'ops/dna/db-master.json'
    ]

    layer.vm.network "private_network", ip: "10.10.10.20"
  end
end

stack.json

{
  "opsworks": {
    "layers": {
      "php-app": {
        "instances": {
          "php-app1": {"private-ip": "10.10.10.10"}
        }
      },
      "db-master": {
        "instances": {
          "db-master1": {"private-ip": "10.10.10.20"}
        }
      }
    }
  },
  "deploy": {
    "simple-php": {
      "application_type": "php",
      "document_root": "web",
      "scm": {
        "scm_type": "git",
        "repository": "dev/simple-php"
      },
      "memcached": {},
      "database": {
        "host": "10.10.10.20",
        "database": "simple-php",
        "username": "root",
        "password": "correcthorsebatterystaple",
        "reconnect": true
      }
    }
  },
  "mysql": {
    "server_root_password": "correcthorsebatterystaple",
    "tunable": {"innodb_buffer_pool_size": "256M"}
  },
  "opsworks_custom_cookbooks": {
    "enabled": true,
    "scm": {
      "repository": "ops/cookbooks"
    }
  }
}

For those not familiar with vagrant you just do a vagrant up to start the instance(s). Then you can modify your cookbook locally and any changes can be applied by re-running chef against the existing instance(s) with vagrant provision. You can do a vagrant destroy and vagrant up to start from scratch.