ECS Health Checks: Why Your Container Keeps Restarting (And How to Fix It)

ECS health checks are the reason your container loops at 2am. You push a deployment, it looks fine, then 10 minutes later the task is cycling — starting, failing, being replaced, starting again. The logs say health check failed but don't tell you which one, why, or what to do about it.

This guide explains how ECS health checks actually work, why they interact in ways that aren't obvious, and what to configure so your service is self-healing rather than self-destructing.

ECS Has Three Different Health Checks (They're Not the Same Thing)

This is the thing that trips people up. ECS doesn't have one health check — it has three, and they operate independently:

Container health check: runs a command inside your container at regular intervals. Defined in your task definition. ECS decides "is this container alive?" based on the exit code.

ALB health check: runs from your load balancer to your container's port over HTTP. Defined on your target group. The ALB decides "is this target ready to receive traffic?" based on the HTTP response.

ECS service health evaluation: the ECS service itself watches both of the above and decides whether to replace a task. If either check fails consistently, ECS kills the task and starts a new one.

You can have just the ALB check, just the container check, or both. Most production setups should have both. And they need to be configured to work together — because when they're misaligned, you get the restart loop.

Why the Restart Loop Happens

The most common cause: your app takes 45 seconds to start, but ECS starts checking health after 15 seconds.

Here's the sequence that kills your deployment:

1. ECS starts your new task
2. The container boots, starts loading your app
3. ECS starts running health checks immediately (default: no grace period)
4. Your app isn't ready yet — health check fails
5. After 3 failures (default), ECS marks the container unhealthy
6. ECS kills the task and starts a new one
7. Repeat forever

You're not broken. You just haven't told ECS to wait for your app to actually start before it starts judging it.

The fix is the startPeriod on the container health check and healthCheckGracePeriodSeconds on the ECS service. These are different settings that serve similar purposes at different layers, and you need to understand both.

Container Health Check: startPeriod Is the One That Matters Most

The container health check is defined in your task definition. Here's a minimal working example:

"healthCheck": {
  "command": ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"],
  "interval": 30,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 60
}

What each setting does:

• interval: how often the check runs (seconds). Default 30.
• timeout: how long to wait for the command to respond before calling it a failure. Default 5 seconds.
• retries: how many consecutive failures before the container is marked UNHEALTHY. Default 3.
• startPeriod: grace period after the container starts, during which failures don't count toward the retry limit.

startPeriod is the one you need to set. If your app takes 30 seconds to boot, set startPeriod to at least 60. If it takes 2 minutes (think Java apps, apps with large in-memory caches), set it to 300.

During the start period, the health check still runs — it just doesn't count failures. The moment the app passes a health check during the start period, the grace ends and the check starts counting normally.

ALB Health Check: Separate Config, Same Goal

Your load balancer runs its own health checks, completely independently. These are configured on the target group, not in ECS.

The ALB sends HTTP requests to your container's port at regular intervals. If it gets back a 200–299 response, the target is healthy and gets traffic. If it doesn't, the target is unhealthy and gets pulled from rotation.

You configure ALB health checks in the target group settings:

• Health check path: the endpoint the ALB hits (e.g., /health)
• Interval: default 30 seconds
• Healthy threshold: consecutive successes needed to mark a target healthy. Default 5 — meaning it takes 150 seconds at default interval before a new task starts getting traffic.
• Unhealthy threshold: consecutive failures to mark it unhealthy. Default 2.

For faster deployments, set healthy threshold to 2 and interval to 10. That means a new task starts getting traffic after 20 seconds of passing health checks instead of 150. It's more health check traffic to your app, but for a startup engineering team that ships frequently, the faster deployment feedback is worth it.

Service-Level Grace Period: Your Safety Net

The ECS service has its own grace period setting: healthCheckGracePeriodSeconds. This tells ECS to ignore ALB health check failures for that many seconds after a task starts.

This is your safety net when the container check and ALB check are both pointing at a slow-starting app. Set it to at least 2x your typical startup time.

aws ecs update-service \
  --cluster my-cluster \
  --service my-api \
  --health-check-grace-period-seconds 120

Note: this only affects how ECS interprets ALB check failures during startup. It doesn't affect the container health check startPeriod. You need both.

Write a Health Check Endpoint That Actually Tests Readiness

A health check endpoint that returns 200 OK immediately without checking anything is worse than no health check — it gives ECS a false signal that your app is ready when it isn't.

A useful health endpoint checks the things your app actually needs to work:

// Node.js example
app.get('/health', async (req, res) => {
  try {
    // Check your database connection
    await db.query('SELECT 1');
    
    // Check Redis if you use it
    await redis.ping();
    
    res.json({ status: 'ok', timestamp: new Date().toISOString() });
  } catch (err) {
    res.status(503).json({ status: 'error', message: err.message });
  }
});

One important caveat: if your health check depends on an external service (like your database), and that external service has a brief outage, all your containers might fail their health check simultaneously. ECS would then try to replace all of them at once — making a minor database blip into a full service outage.

The solution is two separate endpoints:

• /healthz — liveness check. Just verifies the process is alive and the web server is responding. Use this for the container health check.
• /ready — readiness check. Verifies the app can serve traffic (database connected, cache warm, etc.). Use this for the ALB health check.

ECS uses liveness to decide whether to kill and replace the container. The ALB uses readiness to decide whether to route traffic. A database blip fails readiness (ALB pulls the container from rotation) but not liveness (ECS doesn't kill and restart the container). That's the right behavior.

The Four Problems You'll Encounter

1. Container loops on startup

Symptoms: task starts, runs for 60–90 seconds, dies, repeat. CloudWatch logs show your app starting but getting killed before it completes initialization.

Fix: increase startPeriod on the container health check. If you don't have a container health check defined, add one. The default behavior with no startPeriod is to start counting failures from second zero.

2. New task takes forever to receive traffic

Symptoms: deployment completes (old tasks drained, new tasks running) but the service still shows unhealthy for several minutes.

Fix: reduce ALB healthy threshold from 5 to 2. With a 30-second interval and 5-threshold, it takes 150 seconds. With 2-threshold and 10-second interval, it takes 20 seconds.

3. Health check passes locally but fails on ECS

The most common causes:

• Your app is listening on 127.0.0.1 instead of 0.0.0.0. The ALB can't reach it. Fix: bind to 0.0.0.0.
• Security group doesn't allow the ALB to reach the container port. Fix: add an inbound rule on the container security group allowing traffic from the ALB security group.
• The health check path is wrong (case-sensitive). /health and /Health are different paths.
• Your ALB expects a 200 but your app returns 204. Fix: change the ALB matcher to accept 200-299.

4. All containers unhealthy at the same time

Symptoms: a brief external dependency issue causes all tasks to fail health checks simultaneously, cascading into a full replacement cycle.

Fix: separate liveness from readiness as described above. Use liveness for the container check, readiness for the ALB check. An external blip fails readiness (traffic stops), but the containers stay alive (no replacement cycle).

The NoahOps Shortcut

If you're using NoahOps, health check configuration is built into the deployment workflow. You set your health check path, startup time, and check behavior when you configure your service — no task definition JSON editing, no CLI commands.

NoahOps also automatically configures the ALB health check thresholds for fast deployment feedback, and sets the service-level grace period based on your specified startup time.

For teams who've spent time debugging restart loops and unhealthy target groups, this is the part that removes the most midnight incidents.

Request a free demo at noahops.com to see how deployments are configured end-to-end.

---

FAQ

What's the difference between container health check and ALB health check? Container health check runs a command inside the container and reports to ECS. ALB health check runs an HTTP request from the load balancer and controls traffic routing. You need both configured correctly for zero-downtime deployments.

How do I know which health check is causing my task to fail? Check the task's health status in ECS console — it shows containerHealthStatus and the ALB target health separately. The AWS CLI command aws elbv2 describe-target-health --target-group-arn <arn> shows ALB-level health and the specific reason for failure.

Should I use CMD or CMD-SHELL for my health check command? Use CMD-SHELL if your check uses shell features like || or pipes. Use CMD if you have a dedicated health check binary. For the common curl -f http://localhost:PORT/health || exit 1 pattern, use CMD-SHELL.

My app doesn't have curl installed in the container. What do I do? Add wget as an alternative (wget -q -O- http://localhost:PORT/health || exit 1), or write a simple health check binary in your app's language. For Node.js, you can use node -e "require('http').get('http://localhost:PORT/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))".

How long should I set the startPeriod? Set it to at least 2x your app's p95 startup time in production. If your app usually starts in 20 seconds but occasionally takes 45 due to a slow dependency, use 90 seconds. The cost of a too-long start period is slightly delayed failure detection on genuinely broken containers; the cost of a too-short period is restart loops on healthy containers.