Your container is mid-message and autoscaling just killed it

A worker pulls a message off the queue and starts a job that will take minutes. Utilisation across the service drops, because most workers are idle. The autoscaler does exactly what you configured it to do and terminates a task.

It picks the one doing the work.

The queue will redeliver, so nothing is lost. But you have paid for everything that task did before it died, you will pay for it again, and the user is waiting twice. Amazon ECS documents a mechanism for this, and it names your workload specifically.

For the engineering lead running queue consumers on ECS. 8 minute read.

AWS names this exact case in its first example

The feature is task scale-in protection, and its purpose is stated plainly:

"You can use Amazon ECS task scale-in protection to protect your tasks from being terminated by scale-in events from either service auto scaling or deployments."

Note "or deployments" — that half matters later.

The documentation then gives three examples of applications that need it, and the first one is a queue consumer running long jobs at low utilisation:

"You have a queue-processing asynchronous application such as a video transcoding job where some tasks need to run for hours even when cumulative service utilization is low."

That is the shape of every agent pipeline that reads from a queue and calls a model. Long units of work, low average utilisation, and a scaling signal that has no idea which tasks are busy.

The problem is not that autoscaling is wrong. It is that service utilisation is the wrong signal for "is this task safe to kill", and nothing else in the system knows the difference.

How a task protects itself

There are two mechanisms, and AWS recommends one of them for this case.

The container agent endpoint lets a task protect itself from the inside:

"When a container starts processing work, for example by consuming an SQS message, you can set the ProtectionEnabled attribute through the task scale-in protection endpoint path $ECS_AGENT_URI/task-protection/v1/state from within the container."

And AWS is explicit about when to use it:

"We recommend using the Amazon ECS container agent endpoint for tasks that can self-determine the need to be protected. Use this approach for queue-based or job-processing workloads."

The alternative is the API — UpdateTaskProtection and GetTaskProtection — which suits an external controller that tracks task state. AWS's example there is a game server with active sessions.

The distinction is who knows. A queue consumer knows the instant it takes a message and the instant it finishes, so it should protect itself. Something tracking sessions from outside is better placed to use the API. You can combine them, and AWS says so — set protection from inside the container, clear it from an external controller.

The pattern that follows is simple and belongs in your worker base class rather than in each handler: protect on receive, clear on completion, in a finally so a failure path cannot leave a task protected forever.

The numbers

All from the same page, and they are AWS's rather than ours.

Configurable via — ExpiresInMinutes
Bounds — minimum 1 minute, maximum 2880 minutes (48 hours)
Minimum agent version — 1.65.0 or later
IAM on the task role — ecs:GetTaskProtection, ecs:UpdateTaskProtection
Where it works — Tasks deployed from a service only

That last row is a real constraint. Standalone tasks cannot use this, so a design that runs RunTask per job rather than a long-lived service consumer does not have this option available.

And AWS states the cost of over-protecting directly:

"Determine how long a task would need to complete its requisite work and set the expiresInMinutes property accordingly. If you set the protection expiration longer than necessary, then you will incur costs and face delays in the deployment of new tasks."

Two hours is a generous default for a job measured in minutes. Taking it as-is means a crashed worker that never cleared its flag stays protected far longer than the work would have taken.

The trap: it blocks your deployments

Here is the part that turns a reliability feature into an incident, and AWS documents it in full.

Protection defends against deployments, not just autoscaling — that was the "or deployments" in the opening quotation. Which is correct behaviour and is exactly what you want mid-job. It also means:

"If the service uses a rolling update, new tasks will be created but tasks running older version will not be terminated until protectionEnabled is cleared or expires."

For blue/green the same applies — the old deployment lingers, traffic shifts to the new tasks, and the blue tasks only go when protection clears.

And CloudFormation has a hard limit that interacts badly:

"If you use CloudFormation, the update-stack has a 3 hour timeout. Therefore, if you set your task protection for longer than 3 hours, then your CloudFormation deployment may result in failure and rollback."

Read that against the maximum protection period of 48 hours. The feature permits a protection window sixteen times longer than the deployment tool will tolerate, and nothing warns you at configuration time.

There is also a failure you will hit while operating normally. If more tasks are protected than the service's desired count, protection updates start failing:

"While trying to update the protection status of a task, if you receive a DEPLOYMENT_BLOCKED error message, it means the service has more protected tasks than the desired count of tasks for the service."

AWS's remedies are to wait for expiry, to clear protection on tasks that can be stopped, or to raise the desired count. All three are things you do during an incident, not things your worker handles.

So the honest summary: this feature trades deployment agility for work integrity. That is usually the right trade for a pipeline where a lost message costs real money — but it is a trade, and the team that discovers it during a release did not know they had made it.

Where it breaks

Expiry is your only backstop, so it has to be set deliberately. A worker that dies without clearing protection leaves the task protected until the timer runs out. Set that timer to slightly more than your longest realistic job — not to the default, and not to the maximum.

The deployment interaction gets worse as jobs get longer. A pipeline with minute-scale jobs barely notices. One with hour-scale jobs has a deployment window that is effectively gated on the queue being quiet, which is the opposite of what a busy service does.

And it does not protect against everything. A spot interruption, a host failure or an out-of-memory kill will still take the task. Protection is a request to the scheduler, not a guarantee to the process — which is why it complements queue redelivery and two-tier retry rather than replacing them. Chapter 3 covers what happens after the task dies anyway.

One boundary worth naming. Our published piece on mixing FIFO and standard queues in an agent pipeline covers the queue topology beneath all of this — stage-by-stage queue types, message group IDs and per-queue dead-letter settings — and states its own scope: it does not cover ECS, task protection or container orchestration. This article is the layer directly above it, and the two are meant to be read together.

What we would build, and the check for Monday

How we build it. Put protect-and-clear in the worker base class, never in an individual handler, and clear in a finally. Set ExpiresInMinutes from your measured longest job with a margin, and export both the count of protected tasks and the age of the oldest protection as metrics. A protected task older than your longest job is a leak, and it will surface as a blocked deployment weeks later rather than as an error now.

Status, stated plainly. This is a system we built. It runs, it is demonstrable on a call, and it has zero delivered engagements. We build these on request, against your systems. No number here describes anyone's business but our own — the figures above are Amazon's, quoted from their documentation.

The check worth running, and it needs nothing from us. Take your longest-running queue consumer and ask what happens if the autoscaler picks it right now. If the answer is "the message is redelivered and we do the work twice", that is your scale-in policy quietly billing you for duplicate work — and the fix is a documented AWS feature you are probably already entitled to use.

About Cognilium Cognilium is an AI engineering company — agent systems, retrieval, knowledge graphs, voice AI and the production plumbing that makes them survive contact with real workloads. https://cognilium.ai · https://www.linkedin.com/company/37180269/

Want to know whether your queue workers are being killed mid-job? Book a 15-minute call — we will walk the scaling policy with you, on your configuration if you bring it. No deck.

Sources

Protect your Amazon ECS tasks from being terminated by scale-in events

Sources

docs.aws.amazon.com — task scale in protection

Your container is mid-message and autoscaling just killed it