How to autoscale Servers in ECS?

I recently started using ECS. I was able to deploy a container image in ECR and create task definition for my container with CPU/Memory limits. My use case is that each container will be a long running app (no webserver, no port mapping needed). The containers will be spawned on demand 1 at a time and deleted on demand 1 at a time.

I am able to create a cluster with N server instances. But I'd like to be able for the server instances to automatically scale up/down. For example if there isn't enough CPU/Memory in the cluster, I'd like a new instance to be created.

And if there is an instance with no containers running in it, I'd like that specific instance to be scaled down / deleted. This is to avoid auto scale down termination of a server instance that has running tasks in it.

What steps are needed to be able to achieve this?

Considering that you already have an ECS Cluster created, AWS provides instructions on Scaling cluster instances with CloudWatch Alarms.

Assuming that you want to scale the cluster based on the memory reservation, at a high level, you would need to do the following:

Create an Launch Configuration for your Auto Scaling Group. This
Create an Auto Scaling Group, so that the size of the cluster can be scaled up and down.
Create a CloudWatch Alarm to scale the cluster up if the memory reservation is over 70%
Create a CloudWatch Alarm to scale the cluster down if the memory reservation is under 30%

Because it's more of my specialty I wrote up an example CloudFormation template that should get you started for most of this:

Parameters:
  MinInstances:
    Type: Number
  MaxInstances:
    Type: Number
  InstanceType:
    Type: String
    AllowedValues:
      - t2.nano
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
  VpcSubnetIds:
    Type: String

Mappings:
  EcsInstanceAmis:
    us-east-2:
      Ami: ami-1c002379
    us-east-1:
      Ami: ami-9eb4b1e5
    us-west-2:
      Ami: ami-1d668865
    us-west-1:
      Ami: ami-4a2c192a
    eu-west-2:
      Ami: ami-cb1101af
    eu-west-1:
      Ami: ami-8fcc32f6
    eu-central-1:
      Ami: ami-0460cb6b
    ap-northeast-1:
      Ami: ami-b743bed1
    ap-southeast-2:
      Ami: ami-c1a6bda2
    ap-southeast-1:
      Ami: ami-9d1f7efe
    ca-central-1:
      Ami: ami-b677c9d2

Resources:
  Cluster:
    Type: AWS::ECS::Cluster
  Role:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - ec2.amazonaws.com    
  InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref Role    
  LaunchConfiguration:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ImageId: !FindInMap [EcsInstanceAmis, !Ref "AWS::Region", Ami]
      InstanceType: !Ref InstanceType
      IamInstanceProfile: !Ref InstanceProfile
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          echo ECS_CLUSTER=${Cluster} >> /etc/ecs/ecs.config  
  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MinSize: !Ref MinInstances
      MaxSize: !Ref MaxInstances
      LaunchConfigurationName: !Ref LaunchConfiguration
      HealthCheckGracePeriod: 300
      HealthCheckType: EC2
      VPCZoneIdentifier: !Split [",", !Ref VpcSubnetIds]
    ScaleUpPolicy:
      Type: AWS::AutoScaling::ScalingPolicy
      Properties:
        AdjustmentType: ChangeInCapacity
        AutoScalingGroupName: !Ref AutoScalingGroup
        Cooldown: '1'
        ScalingAdjustment: '1'
    MemoryReservationAlarmHigh:
      Type: AWS::CloudWatch::Alarm
      Properties:
        EvaluationPeriods: '2'
        Statistic: Average
        Threshold: '70'
        AlarmDescription: Alarm if Cluster Memory Reservation is to high
        Period: '60'
        AlarmActions:
        - Ref: ScaleUpPolicy
        Namespace: AWS/ECS
        Dimensions:
        - Name: ClusterName
          Value: !Ref Cluster
        ComparisonOperator: GreaterThanThreshold
        MetricName: MemoryReservation
    ScaleDownPolicy:
      Type: AWS::AutoScaling::ScalingPolicy
      Properties:
        AdjustmentType: ChangeInCapacity
        AutoScalingGroupName: !Ref AutoScalingGroup
        Cooldown: '1'
        ScalingAdjustment: '-1'
    MemoryReservationAlarmLow:
      Type: AWS::CloudWatch::Alarm
      Properties:
        EvaluationPeriods: '2'
        Statistic: Average
        Threshold: '30'
        AlarmDescription: Alarm if Cluster Memory Reservation is to Low
        Period: '60'
        AlarmActions:
        - Ref: ScaleDownPolicy
        Namespace: AWS/ECS
        Dimensions:
        - Name: ClusterName
          Value: !Ref Cluster
        ComparisonOperator: LessThanThreshold
        MetricName: MemoryReservation

This creates an ECS Cluster, a Launch Configuration, An AutoScaling Group, As well as the Alarms based on the ECS Memory Reservation.

Now we can get to the interesting discussions.

Why can't we scale up based on the CPU Utilization And Memory Reservation?

The short answer is you totally can But you're likely to pay a lot for it. EC2 has a known property that when you create an instance, you pay for a minimum of 1 hour, because partial instance hours are charged as full hours. Why that's relevant is, imagine you have multiple alarms. Say you have a bunch of services that are currently running idle, and you fill the cluster. Either the CPU Alarm scales down the cluster, or the Memory Alarm scales up the cluster. One of these will likely scale the cluster to the point that it's alarm is no longer triggered. After the cooldown, period, the other alarm will undo it's last action, After the next cooldown, the action will likely be redone. Thus instances are created then destroyed repeatedly on every other cooldown.

After giving a bunch of thought to this, the strategy that I came up with was to use Application Autoscaling for ECS Services based on CPU Utilization, and Memory Reservation based on the cluster. So if one service is running hot, an extra task will be added to share the load. This will slowly fill the cluster memory reservation capacity. When the memory gets full, the cluster scales up. When a service is cooling down, the services will start shutting down tasks. As the memory reservation on the cluster drops, the cluster will be scaled down.

The thresholds for the CloudWatch Alarms might need to be experimented with, based on your task definitions. The reason for this is that if you put the scale up threshold too high, it may not scale up as the memory gets consumed, and then when autoscaling goes to place another task, it will find that there isn't enough memory available on any instance in the cluster, and therefore be unable to place another task.