How to scale-in ECS hosts

September 2, 2017
Contact Us
Weekly Shorts are topics we discuss in our weekly remote meeting related to recent work we have done with our customers
How to scale-in ECS hosts

ECS provides the freedom of scaling different services through adding “tasks”. These operate and communicate independently, and are being hosted on a series of EC2 instances. Scaling the system’s services in and out, as well as scaling out EC2 hosts is a convenient task, achieved with the help of CloudWatch metrics and alarms. However, scaling-in EC2 hosts require some complexity in decision making.

ECS is a three-layer architecture:

  1. Cluster: a set of applications running on a shared group of resources.
  2. Service: a logical instance (e.g “application”)‍
  3. Task: a physical instance of a service‍

Scaling tasks in and out is relatively simple; based on a couple of metric thresholds such as CPU load percentage which determines whether new tasks should be loaded, or current ones should be removed. Scaling ECS hosts out, is about the same; for example when a certain level of CPU is crossed, or when the physical cluster doesn’t have the required amount of memory to hold the current load, new EC2 instances are triggered.

Removing resources is about cost reduction; these are the resources that are being paid for, and removing them when they are no longer required is about utilization which is translated to costs.
However, the scale-in task is somewhat challenging: applying the general method of crossing a metric threshold might result in cutting down required resources in other parameters, for example: removing resources due to lower levels of CPU while the level of memory reservation cannot “afford” to lose hosts.
Since AWS doesn’t provide the function of multiple-metric scale, and due to the fact that such a scale requires a few more calculations to make the decision of resource removal, an external tool is required for the task.

The problem

As said, scaling-in is not a straightforward task: assuming a host cluster was scaled up based on low memory available, but then scaled down to low CPU levels, removing instances may result again in low memory available which will trigger new hosts again. This can be a never-ending scaling loop.

Containers scale graph
Never ending loop — # of container instances scale up and down repeatedly as each scale triggers the opposite action

A multi-metric scale trigger is required. Ignoring for a moment the fact that AWS doesn’t offer such a feature, what if a multimetric trigger is used but the future metrics levels’ trigger scale up again, this is another loop. So, a forecast metric calculation is also required. There’s also the need to decide which is the least utilized instance to scale, then making this a graceful removal draining client connections before termination, together with a cleanup of 0% utilized resources.https://github.com/omerxx/ecscale

ECSCALE answers all of the above. It is ready to run on AWS Lambda, which usually means this is a completely free resource, providing a virtual butler that cleans up the mess of unused instances.

The tool is ready to be deployed as a server-less function running on AWS Lambda. As such, it only requires ~3000 Lambda seconds every month. Since every AWS account is entitled to 1,000,000 free seconds every month, unless you’re running other server-less applications on Lambda, the resource is in fact completely free of charge. The scaling process is running every 60 minutes (configurable, one hour is default as EC2 instances are already paid for a whole hour even if terminated before), iterating over an account's ECS resources and cleaning them up.

Edit: AWS have recently announced a new EC2 billing model which charges and instance by the second. This means that when an instance is terminated, it's billed for the part of the hour it was running and no a whole hour. Therefore, It is recommended the scaling tool is run more frequently; since API calls are made, and the system should be given time to make unrelated changes such as deployments, scaling services and others, 1 second or even 1 minute are not ideal. After running some tests it seems that the sweet-spot is around the 20 minutes trigger.
You should, however, test this in your own cluster and make sure current procedures are not affected by the new frequent changes.

Best of luck!

How does it work?
  • The service iterates over existing ECS clusters.
  • Checking a cluster’s ability to scale-in based on predicted memory reservation capacity.
  • Look for empty hosts running no tasks that can be scaled.
  • Look for the least utilized scale candidate.
  • Move the candidate to drain state; draining service connections gracefully.
  • terminate a draining host that has no running tasks and decrease the desired number of instances.
Getting it started
  1. Create a Lambda function with a Maximum 5 seconds runtime using the provided role policy.
  2. Copy ecscale.py code to Lambda.
  3. ‍Create a trigger for running every hour

All set, your resources are automatically taken cared of.

How to scale-in ECS hosts
Omer Hamerman
Senior Software Operations Architect
Omer is an experienced software operations engineer and an open source contributor. He is always willing to go the extra mile to help our clients improve their software delivery. He is known for getting the job done very quickly and is clear-cut and very sharp, delivering almost any job on the spot. When he’s not helping our clients achieve scalable and resilient infrastructure, you’ll find him rock climbing and bouldering. He is passionate about beautiful code, cybersecurity and doing things right the first time. He is a keen writer of blog posts and a speaker at meetups.