userData are not getting executed after launching

Question

I created a AWS cloudformation, which creates a launch configuration and an autoscaling group. In the user Data in the launch Config I have configured the file system mount target, and I installed the cloudwatch agent:

Code EDITED

  "LaunchConfig":{
    "Type":"AWS::AutoScaling::LaunchConfiguration",
    "Metadata" : {
      "AWS::CloudFormation::Init" : {
        "config" : {
          "files" : {
            "/etc/cwlogs.cfg": {
              "content": { "Fn::Join" : ["", [
                "[general]",
                "state_file = /var/awslogs/state/agent-state",
                "[/var/log/syslog]",
                "file = /tmp/",
                "log_group_name = ecs-dataloader",
                "log_stream_name = ECS-loader",
                "datetime_format = %b %d %H:%M:%S"
                ]]},
                "mode": "000755",
                "owner": "root",
                "group": "root"
            },
            "/etc/ecs/ecs.config": {
              "content": { "Fn::Join" : ["", [
                "ECS_CLUSTER=", { "Ref" : "ClusterName" }
              ]]},
              "mode": "000755",
              "owner": "root",
              "group": "root"
            }
          },
          "commands": {
            "Update": {
              "command": "yum -y update"
            },
            "InstallNfs":{
              "command": "yum -y install nfs-utils"
            },
            "CreatFolder": {
              "command": "mkdir -p /efs-mount-point/"
            },
            "EditPerms": {
              "command": "chown ec2-user:ec2-user /efs-mount-point/"
            },
            "MountPoint": {
              "command": { "Fn::Join" : ["", [
                "AZ=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)\n",
                "echo LABEL=cloudimg-rootfs / ext4 defaults,discard    0 0\n",
                "$AZ.",{ "Fn::FindInMap" : [ "FileSystemMap", {"Ref" : "EnvParam"}, "FileSystemID"] },
                ".efs.",{ "Ref" : "AWS::Region" },".amazonaws.com:/  /efs-script-import-tmp nfs4 nfsvers=4.1 0 0 >> /etc/fstab"
              ]]}
            },
            "Mount": {
              "command": "mount -a -t nfs4"
            },
            "CloudWatchAgent": {
              "command": { "Fn::Join" : ["", [
                "curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O\n",
                "python ./awslogs-agent-setup.py --region ",{"Ref" : "AWS::Region"},"\n",
                "chmod +x ./awslogs-agent-setup.py ./awslogs-agent-setup.py -n -r",
                {"Ref" : "AWS::Region"}," -c /etc/cwlogs.cfg"
              ]]}
            }
          },
          "services" : {
            "sysvinit" : {
              "awslogs"  : { "enabled" : "true", "ensureRunning" : "true" }
            }
          }
        }
      }
    },
    "Properties":{
      "ImageId":{ "Fn::FindInMap":[ "AWSRegionToAMI", { "Ref":"AWS::Region" }, "AMIID" ] },
      "SecurityGroups":[ { "Ref":"EcsSecurityGroup" } ],
      "InstanceType": {"Ref":"InstanceType" },
      "IamInstanceProfile":{ "Ref":"EC2InstanceProfile" },
      "KeyName":{ "Fn::FindInMap" : [ "KeyPairMapping", {"Ref" : "EnvParam"}, "Key"] },
      "UserData":{  "Fn::Base64" : {
        "Fn::Join" : ["", [
          "#!/bin/bash -xe\n",
          "/opt/aws/bin/cfn-init -v ",
          "         --stack ", { "Ref": "AWS::StackName" },
          "         --resource LaunchConfig",
          "         --region ", { "Ref" : "AWS::Region" },"\n"
        ]]}
      }
    }
  }

The image details : "eu-west-1": { "AMIID":"ami-ba346ec9" },

After running the template, the resources got created successfuly. So I connected to my instance that got created by the autoscaling group via SSH to see if the userData was properly run and set. Unfortunately, After checking, this is what I found in the /etc/fstab file:

   $ cat /etc/fstab
   LABEL=cloudimg-rootfs    /    ext4   defaults,discard    0 0


   $ cat /etc/ecs/ecs.config
   cat: /etc/ecs/ecs.config: No such file or directory

The instance is not connected to the file system, the file that I tried to create in the cloudformation::init /etc/cwlogs.cfg does not exist either (it's the cloudwatch agent config file) . Can any one tell me what is wrong in the user data that it didn't get executed?

I tried to check the log files but :

$ cat /var/log/cfn-init.log 
cat: /var/log/cfn-init.log: No such file or directory

What is the problem here ?

EDIT

$ cat /var/log/cloud-init-ouput.log
...

Cloud-init v. 0.7.6 running 'modules:final' at Fri, 17 Feb 2017 11:43:42 +0000. Up 44.66 seconds.
+ yum install -y aws-cfn-bootstrap/opt/aws/bin/cfn-init -v --stack Mystack --resource LaunchConfig --region eu-west-1
Loading "priorities" plugin
Loading "update-motd" plugin
Config time: 0.009
Command line error: no such option: --stack
Feb 17 11:43:43 cloud-init[2814]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Feb 17 11:43:43 cloud-init[2814]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Feb 17 11:43:43 cloud-init[2814]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.6 finished at Fri, 17 Feb 2017 11:43:43 +0000. Datasource DataSourceEc2.  Up 45.18 seconds

Answer 1

User Data log files are located at:

• Linux cloud-init: /var/log/cloud-init.log
• Windows EC2Config: C:\cfn\log\cloud-init.log

Check to see whether anything is in the log file. If not, then something's wrong with passing the User Data script from the template. (Why do you have the initial empty quotes in the Join?)

Answer 2

cfn-init is only installed by default on Amazon Linux AMI, so if you're using any other Image ID to launch your EC2 instance you need to ensure that it's installed correctly before invoking it. See my previous answer to the question, "Installing packages using apt-get in CloudFormation file" for more info.

Answer 3

Here is how I resolved the problem: I Update the cloud-init in the user data before calling the meta-data and instead of installing the cloudwatch agent in the metadata, I did in the userdata.

          "UserData":{  "Fn::Base64" : {
            "Fn::Join" : ["", [
              "#!/bin/bash -xe\n",
              "yum -y install aws-cfn-bootstrap\n",
              "/opt/aws/bin/cfn-init -v",
              "         --stack ", { "Ref": "AWS::StackName" },
              "         --resource LaunchConfig",
              "         --region ", { "Ref" : "AWS::Region" },"\n",
              "# Get the CloudWatch Logs agent\n",
              "wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py\n",
              "# Install the CloudWatch Logs agent\n",
              "python ./awslogs-agent-setup.py -n -r ", { "Ref" : "AWS::Region" }, " -c /etc/cwlogs.cfg || error_exit 'Failed to run CloudWatch Logs agent setup'\n",
              "service awslogs start"
          ]]}