What Happens When Your ECS Deployment Goes Wrong — And How to Make It Fix Itself

You push a new container image to ECS. Your app crashes on startup. ECS doesn't know your deploy broke — so it keeps trying. And trying. And trying. Tasks spin up, fail, and restart in a loop while your production service sits in a half-deployed state.

Without intervention, this can run for hours.

The ECS deployment circuit breaker is the feature that stops this. When it detects that new tasks keep failing, it halts the deployment and automatically rolls back to the version that was working before you pushed.

Here's how it works, how to turn it on, and the two things about it that catch engineers off guard.

Why Deployments Can Go Wrong Silently on ECS

When you update an ECS service — pushing a new task definition, for example — ECS starts a rolling deployment. It tries to bring up new tasks alongside the old ones.

The problem: ECS's default behavior is optimistic. It keeps trying to launch new tasks until it hits a configurable maximum deployment time (which defaults to hours). If your new container crashes immediately on startup, ECS will faithfully keep trying to restart it, over and over, while your service degrades.

You won't necessarily get an alert. You'll just notice, an hour later, that your service is acting strangely — because it's running a mix of old and new tasks, or because it's been in a failed deployment state the whole time.

The circuit breaker changes this. It counts failures and, when a threshold is hit, declares the deployment failed and rolls back.

How the Circuit Breaker Actually Works

ECS monitors two things during a deployment:

1. Whether new tasks reach the RUNNING state
2. Whether running tasks pass their health checks

If tasks keep failing to reach RUNNING, or keep failing health checks, ECS increments a failure counter. When that counter hits the threshold, the circuit breaker triggers.

The threshold is: ceil(0.5 × desired task count), with a minimum of 3 and a maximum of 200.

So if you're running 4 tasks, ECS will allow up to 3 failures before triggering. Running 10 tasks? Up to 5 failures. Running 100 tasks? Up to 50.

When the circuit breaker triggers with rollback enabled, ECS finds the most recent deployment that completed successfully and reverts to it. Your service goes back to running the last known-good version.

Common Causes of a Circuit Breaker Trigger

If you see a deployment roll back, it's usually one of these:

Container crashes immediately on startup. Your app throws an unhandled exception or exits with a non-zero code. The task goes from PROVISIONING → STOPPED without ever reaching RUNNING.

Health check failures. The container starts, but the load balancer health check keeps failing — because the app isn't listening on the right port, or it's taking too long to initialize, or it's returning 500s during startup.

Can't pull the image. The ECR image tag doesn't exist, or the ECS task execution role doesn't have permission to pull from ECR. Tasks fail before the container even runs.

Out of memory. Your new version uses more memory than the task definition allows. Containers start and immediately OOM-kill.

Can't reach a dependency. Your app tries to connect to a database or external service on startup, fails, and crashes. The circuit breaker sees the crash; it doesn't know the root cause.

Enabling the Circuit Breaker: Three Ways

In the AWS Console

Open your ECS service → Edit → Deployment configuration → enable "Use deployment circuit breaker" and "Automatic rollback."

With the AWS CLI

aws ecs update-service \
  --cluster your-cluster \
  --service your-service \
  --deployment-configuration '{
    "deploymentCircuitBreaker": {
      "enable": true,
      "rollback": true
    },
    "minimumHealthyPercent": 100,
    "maximumPercent": 200
  }'

In Terraform

resource "aws_ecs_service" "api" {
  name          = "api-service"
  cluster       = aws_ecs_cluster.main.id
  desired_count = 3
  launch_type   = "FARGATE"

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200
}

That's it. Enable both enable and rollback. If you only enable enable without rollback, the deployment will halt on failure but won't revert — your service stays in a degraded state until you manually intervene.

Checking if a Rollback Happened

aws ecs describe-services \
  --cluster your-cluster \
  --services your-service \
  --query 'services[0].deployments[*].{id:id,status:status,rolloutState:rolloutState,rolloutReason:rolloutStateReason,running:runningCount,failed:failedTasks}'

Look for rolloutState: "FAILED" and a rolloutStateReason that describes why. A new deployment entry with IN_PROGRESS or COMPLETED beside it means the rollback is underway or has completed.

Gotcha #1: The Circuit Breaker Won't Catch Everything

This is the one that surprises most engineers: the circuit breaker only catches startup failures and health check failures during deployment.

If your new container starts successfully, passes its health checks, and then starts behaving badly — returning errors, degrading slowly, consuming too much memory — the circuit breaker won't roll it back. As far as ECS is concerned, the deployment completed successfully.

For this scenario, you need a different mechanism: ECS deployment alarms, which watch CloudWatch metrics during the deployment window and can trigger a rollback if error rates or latency cross a threshold.

The circuit breaker is for "the new version immediately broken." Deployment alarms are for "the new version is slowly broken."

Gotcha #2: Rollback Requires a Working Previous Deployment

The circuit breaker rolls back to the most recent deployment that reached COMPLETED status. If your last completed deployment was also broken — it started and passed health checks, but was failing at the application level — the rollback won't help. You'll roll back to a broken state.

This is another reason to have application-level health checks, not just infrastructure-level ones. Your ECS health check should probe an endpoint that reflects whether the app is actually ready to serve traffic, not just whether a port is open.

Getting Alerts When a Rollback Happens

The circuit breaker rolls back silently unless you set up alerts. ECS emits EventBridge events for deployment state changes.

# Create an SNS topic for deployment events
aws sns create-topic --name ecs-deployment-alerts

# Subscribe your email
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:YOUR_ACCOUNT:ecs-deployment-alerts \
  --protocol email \
  --notification-endpoint your@email.com

Then create an EventBridge rule to catch circuit breaker events:

{
  "source": ["aws.ecs"],
  "detail-type": ["ECS Deployment State Change"],
  "detail": {
    "eventName": ["SERVICE_DEPLOYMENT_FAILED"]
  }
}

Route that rule to your SNS topic. Now when a deployment rolls back, you get an email with the deployment ID and reason.

Health Check Grace Period: The Other Config You Need

One deployment failure mode that trips up engineers: your app takes 30 seconds to start, but the load balancer health check starts probing immediately. The health check fails repeatedly during startup, the circuit breaker counts those as failures, and the deployment rolls back before your app even finished initializing.

Fix: set healthCheckGracePeriodSeconds on your ECS service. This tells ECS to ignore health check failures for X seconds after a task starts.

aws ecs update-service \
  --cluster your-cluster \
  --service your-service \
  --health-check-grace-period-seconds 60

Set this to slightly more than your actual startup time. If your app takes 20 seconds to be ready, set 30-45 seconds. If you set it too high, slow rollbacks will take longer to detect as failed — so tune it based on your actual startup behavior.

How NoahOps Handles This

NoahOps enables the deployment circuit breaker and auto-rollback on every service by default. You don't have to configure it.

When you deploy via NoahOps — whether through the dashboard, a GitHub Actions push, or Noah AI — the circuit breaker is already on. If a deployment fails, it rolls back automatically and you get a Slack notification with the failure reason.

Health check grace periods are set based on your service's historical startup time. NoahOps tracks how long your containers typically take to become healthy and configures the grace period accordingly.

You also get a deployment history view in the dashboard that shows rollout state, failure counts, and rollback events without having to query the AWS CLI.

---

FAQ

What's the difference between the circuit breaker and a blue/green deployment? The circuit breaker is for rolling deployments — ECS's default. It detects failures during the rollout and rolls back. Blue/green (via CodeDeploy) keeps the old version fully up while the new one spins up in a separate target group, then switches traffic over once the new version is healthy. Blue/green is more complex to set up but gives you a faster rollback and zero traffic impact during the switch.

Can I set a custom failure threshold? No. AWS calculates the threshold from your desired task count. You can't override it. For a 4-task service, that's 3 failures minimum. This is one of the circuit breaker's limitations.

Will the circuit breaker trigger if only one of my tasks fails? Only if the failure count reaches the threshold (minimum 3). For a small service with 4 desired tasks, you'd need 3 task failures before rollback triggers. A single task failure that then recovers won't trigger a rollback.

What if I deploy manually and the rollback takes too long? You can manually force a rollback by updating the service back to the previous task definition ARN. NoahOps lets you do this with one click from the deployment history view.

---

Request a free demo at noahops.com — see how NoahOps deploys your services into your own AWS account with circuit breakers, alerts, and rollback built in.