Terraform: configuring cloudwatch log subscription

I need to ship my cloudwatch logs to a log analysis service.

I've followed along with these articles here and here and got it working by hand, no worries.

Now I'm trying to automate all this with Terraform (roles/policies, security groups, cloudwatch log group, lambda, and triggering the lambda from the log group).

But I can't figure out how to use TF to configure AWS to trigger the lambda from the cloudwatch logs.

I can link the two TF resources together by hand by doing the following (in the Lambda web console UI):

go into the lambda function's "Triggers" section
click "Add Trigger"
select "cloudwatch logs" from the list of trigger types
select the log group I want to trigger the lambda
enter a filter name
leave the filter pattern empty (implying trigger on all log streams)
make sure "enable trigger" is selected
click the submit button

Once that's done, the lambda shows up on the cloudwatch logs console in the subscriptions column - displays as "Lambda (cloudwatch-sumologic-lambda)".

I tried to create the subscription with the following TF resource:

resource "aws_cloudwatch_log_subscription_filter" "cloudwatch-sumologic-lambda-subscription" {
  name = "cloudwatch-sumologic-lambda-subscription"
  role_arn = "${aws_iam_role.jordi-waf-cloudwatch-lambda-role.arn}"
  log_group_name = "${aws_cloudwatch_log_group.jordi-waf-int-app-loggroup.name}"
  filter_pattern = "logtype test"
  destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}

But it fails with:

* aws_cloudwatch_log_subscription_filter.cloudwatch-sumologic-lambda-subscription: InvalidParameterException: destinationArn for vendor lambda cannot be used with roleArn

I found this answer about setting up a similar thing for a scheduled event, but that doesn't seem to be equivalent to what the console actions I described above do (the console UI method doesn't create an event/rule that I can see).

Can someone give me a pointer on what I'm doing wrong please?

I had the "aws_cloudwatch_log_subscription_filter" resource defined incorrectly - you should not provide the "role_arn" argument in this situation.

You also need to add an aws_lambda_permission resource (with a "depends_on" relationship defined on the filter or TF may do it in the wrong order).

Note that the AWS lambda console UI adds the lambda permission for you invisibly, so beware that the "aws_cloudwatch_log_subscription_filter" will work without the permission resource if you happen to have done the same action before in the console UI.

The necessary TF config looks like this (the last two resources are the relevant ones for configuring the actual cloudwatch->lambda trigger):

// intended for application logs (access logs, modsec, etc.)
resource "aws_cloudwatch_log_group" "test-app-loggroup" {
  name = "test-app"
  retention_in_days = 90
}


resource "aws_security_group" "cloudwatch-sumologic-lambda-sg" {
  name = "cloudwatch-sumologic-lambda-sg"
  tags {
    Name = "cloudwatch-sumologic-lambda-sg"
  }
  description = "Security group for lambda to move logs from CWL to SumoLogic"
  vpc_id = "${aws_vpc.dev-vpc.id}"
}

resource "aws_security_group_rule" "https-egress-cloudwatch-sumologic-to-internet" {
  type = "egress"
  from_port = 443
  to_port = 443
  protocol = "tcp"
  security_group_id = "${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"
  cidr_blocks = ["0.0.0.0/0"]
}

resource "aws_iam_role" "test-cloudwatch-lambda-role" {
  name = "test-cloudwatch-lambda-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "test-cloudwatch-lambda-policy" {
  name = "test-cloudwatch-lambda-policy"
  role = "${aws_iam_role.test-cloudwatch-lambda-role.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole1",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole2",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "arn:aws:ec2:ap-southeast-2:${var.dev_vpc_account_id}:network-interface/*"
    },

    {
      "Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole1",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:*"
    },
    {
      "Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole2",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
    "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:log-group:/aws/lambda/*"
      ]
    },

    {
      "Sid": "CopiedFromTemplateAWSLambdaAMIExecutionRole",
      "Effect": "Allow",
      "Action": [
         "ec2:DescribeImages"
      ],
      "Resource": "*"
    }


  ]
}
EOF
}

resource "aws_lambda_function" "cloudwatch-sumologic-lambda" {
  function_name = "cloudwatch-sumologic-lambda"
  filename = "${var.lambda_dir}/cloudwatchSumologicLambda.zip"
  source_code_hash = "${base64sha256(file("${var.lambda_dir}/cloudwatchSumologicLambda.zip"))}"
  handler = "cloudwatchSumologic.handler"

  role = "${aws_iam_role.test-cloudwatch-lambda-role.arn}"
  memory_size = "128"
  runtime = "nodejs4.3"
  // set low because I'm concerned about cost-blowout in the case of mis-configuration
  timeout = "15"
  vpc_config = {
    subnet_ids = ["${aws_subnet.dev-private-subnet.id}"]
    security_group_ids = ["${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"]
  }
}

resource "aws_lambda_permission" "test-app-allow-cloudwatch" {
  statement_id = "test-app-allow-cloudwatch"
  action = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
  principal = "logs.ap-southeast-2.amazonaws.com"
  source_arn = "${aws_cloudwatch_log_group.test-app-loggroup.arn}"
}

resource "aws_cloudwatch_log_subscription_filter" "test-app-cloudwatch-sumologic-lambda-subscription" {
  depends_on = ["aws_lambda_permission.test-app-allow-cloudwatch"]
  name = "cloudwatch-sumologic-lambda-subscription"
  log_group_name = "${aws_cloudwatch_log_group.test-app-loggroup.name}"
  filter_pattern = ""
  destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}