ECS Logs and CloudWatch: Why Your Container Logs Are Disappearing (And What to Actually Monitor)

Your ECS service is running. Something's wrong — requests are failing, or a task keeps restarting. You go to check logs. They're not there. Or they're there but from three restarts ago. Or there are logs but you can't find the right one.

This is the most common debugging wall startup engineers hit on ECS. Not a hard infrastructure problem — a logging configuration problem that nobody explained in plain English.

This guide covers how ECS logging actually works, why logs disappear, what to monitor in CloudWatch, and how to set up alerts that actually matter. No unnecessary CLI ceremony.

How ECS Logging Works (The 30-Second Version)

Your container writes to stdout and stderr. That's it — you don't configure logging inside your app. You write normally (console.log, print, logger.info — whatever your app uses), and ECS captures it.

What happens to those bytes depends on your log driver configuration in your task definition.

The default and simplest option is awslogs — the AWS CloudWatch Logs driver. It streams everything your container writes to stdout/stderr directly into a CloudWatch log group.

If you haven't configured a log driver, your logs go nowhere. This is why they're disappearing.

Setting Up the awslogs Driver

In your task definition, under the container definition, add this:

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/your-service-name",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "ecs"
  }
}

That's the entire configuration. Three fields.

awslogs-group — the CloudWatch log group where your logs land. Create it manually first, or add "awslogs-create-group": "true" to the options and ECS will create it.
awslogs-region — must match the region your cluster is in.
awslogs-stream-prefix — a prefix for each log stream. Streams are named {prefix}/{container-name}/{task-id}. Using ecs is fine.

Your task execution role needs permission to write to CloudWatch Logs. Add this policy to the execution role:

{
  "Effect": "Allow",
  "Action": [
    "logs:CreateLogStream",
    "logs:PutLogEvents"
  ],
  "Resource": "arn:aws:logs:us-east-1:YOUR_ACCOUNT_ID:log-group:/ecs/your-service-name:*"
}

After deploying a task with this config, you'll find logs in CloudWatch → Log groups → /ecs/your-service-name.

The Log Stream Structure: Why You Can't Find the Right Log

Each task gets its own log stream, named like:

ecs/api-container/a3f2b4c1d5e6f7a8b9c0d1e2f3a4b5c6

That last part is the task ID. When a task restarts (crashes, new deployment, scaling event), it gets a new task ID — and a new log stream.

This means:

If your task crashed and restarted, the logs from before the crash are in the old stream
New streams appear at the top of the list by default
If you're looking at a specific task's logs, you need its task ID

How to find the right stream quickly:

In the AWS console: CloudWatch → Log groups → your group → Filter log streams by the task ID. Get the task ID from ECS → Clusters → your cluster → Tasks → the specific task.

Or using the CLI:

# List recent log streams
aws logs describe-log-streams \
  --log-group-name /ecs/your-service-name \
  --order-by LastEventTime \
  --descending \
  --max-items 5

This gives you the 5 most recently active streams — which is usually where your recent logs are.

Searching Logs: CloudWatch Insights

The CloudWatch console log viewer is fine for tailing live logs. For anything more complex — finding errors across multiple task restarts, correlating logs across time — use CloudWatch Logs Insights.

Go to CloudWatch → Logs Insights, select your log group, and run queries:

# Find all ERROR lines in the last hour
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

# Find specific request IDs
fields @timestamp, @message
| filter @message like /req-abc123/
| sort @timestamp asc

# Count errors by minute to spot a spike
filter @message like /ERROR/
| stats count() as errorCount by bin(1m)
| sort @timestamp asc

Insights queries are fast even on large log groups. This is the right tool when you're investigating an incident, not the log stream viewer.

Log Retention: Why Your Logs Disappear After a Few Months

By default, CloudWatch log groups have no retention policy — logs are stored indefinitely. This sounds fine until you get a bill for log storage.

Set a retention policy on each log group. 30 days is reasonable for most startup use cases:

aws logs put-retention-policy \
  --log-group-name /ecs/your-service-name \
  --retention-in-days 30

Or in the console: CloudWatch → Log groups → click the log group → Actions → Edit retention setting.

If you need logs longer for compliance or debugging history, consider shipping logs to S3 at lower cost using a CloudWatch subscription filter or a log forwarder.

What to Monitor: The Metrics That Actually Matter

ECS publishes two service-level metrics to CloudWatch automatically:

CPUUtilization — percentage of CPU your service is using vs. what it's reserved
MemoryUtilization — percentage of memory your service is using vs. what it's reserved

You find these in CloudWatch → Metrics → AWS/ECS namespace.

These two metrics tell you most of what you need to know:

High CPU (>80% average): Your service is under load or needs more CPU reserved. If this spikes during deploys, you might have a memory-intensive build step running in your container.

High memory (>85% maximum): This is the more dangerous one. Memory OOM kills are silent — your container just stops with exit code 137. The metric to watch is the maximum, not the average. An average of 60% with a maximum of 95% means you're one traffic spike away from an OOM restart loop.

The metric you're probably missing: task count. ECS doesn't publish this to the standard AWS/ECS namespace — you need Container Insights enabled to get RunningTaskCount and DesiredTaskCount. This is the most important signal during a deployment or an outage: if running tasks < desired tasks, something is preventing your service from being healthy.

Enabling Container Insights (Worth the Cost)

Container Insights costs extra (~$0.50/month per resource, roughly $5-15/month for a small setup), but it gives you:

Per-task CPU and memory metrics (not just service-level aggregates)
RunningTaskCount and DesiredTaskCount
Network I/O
Storage I/O

Enable it on your cluster:

aws ecs update-cluster-settings \
  --cluster your-cluster-name \
  --settings name=containerInsights,value=enabled

Once enabled, these metrics appear in the ECS/ContainerInsights namespace in CloudWatch.

The Three Alarms Every ECS Service Needs

Alarm 1: High Memory

aws cloudwatch put-metric-alarm \
  --alarm-name "ecs-your-service-high-memory" \
  --namespace "AWS/ECS" \
  --metric-name MemoryUtilization \
  --dimensions Name=ClusterName,Value=your-cluster Name=ServiceName,Value=your-service \
  --statistic Maximum \
  --period 300 \
  --threshold 85 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --treat-missing-data breaching \
  --alarm-actions "arn:aws:sns:us-east-1:ACCOUNT:your-alert-topic"

Use Maximum, not Average. Memory peaks matter more than averages.

Alarm 2: Task Count Below Desired (requires Container Insights)

aws cloudwatch put-metric-alarm \
  --alarm-name "ecs-your-service-task-deficit" \
  --namespace "ECS/ContainerInsights" \
  --metric-name RunningTaskCount \
  --dimensions Name=ClusterName,Value=your-cluster Name=ServiceName,Value=your-service \
  --statistic Minimum \
  --period 60 \
  --threshold 1 \
  --comparison-operator LessThanThreshold \
  --evaluation-periods 5 \
  --alarm-actions "arn:aws:sns:us-east-1:ACCOUNT:your-alert-topic"

This fires if your running task count drops to zero for 5 minutes. It's your "service is down" alarm.

Alarm 3: High CPU (for scaling trigger, not paging)

Set this higher — 80% average over 10 minutes — and wire it to autoscaling, not your pager. CPU spikes are normal; sustained high CPU that doesn't scale down is the problem.

The Log-to-Alarm Workflow During an Incident

When something breaks, here's the sequence:

Alarm fires → tells you which service and which metric
ECS console → Tasks → find the task that's failing (stopped tasks are visible for a few minutes after stopping)
Task details → Logs → direct link to the CloudWatch log stream for that specific task
CloudWatch Insights → query across the log group if you need to correlate across multiple task restarts

The task's log link in the ECS console is the fastest path during an incident. It takes you directly to the right stream without having to search.

How NoahOps Handles This for You

Setting up log drivers, retention policies, Container Insights, and CloudWatch alarms across multiple services is repetitive and easy to get wrong. NoahOps provisions all of this automatically when you deploy a service:

awslogs driver configured on every container
Log groups created with a 30-day retention policy
Container Insights enabled by default
Memory and task count alarms created per service, wired to Slack notifications

You get proper observability on day one — no manual CloudWatch configuration needed.

Request a free demo at noahops.com to see a full service deployment with logging and monitoring already configured.

FAQ

My container is running but I see no logs in CloudWatch. What's wrong?

Three likely causes: (1) your task definition doesn't have the awslogs log driver configured, (2) your task execution role doesn't have CloudWatch Logs write permissions, or (3) the log group doesn't exist and you haven't set awslogs-create-group: true. Check all three in that order.

How do I see logs from a crashed container?

In the ECS console, go to your cluster → Tasks → Stopped. Tasks stay visible for a few minutes after they stop. Click the stopped task → select the container → Logs. If the task stopped more than an hour ago, find the log stream in CloudWatch directly by filtering for the task ID.

Should I use awslogs or FireLens?

For most startups, awslogs is the right choice — it's simple, reliable, and costs nothing beyond CloudWatch storage. FireLens makes sense if you're routing logs to multiple destinations (Datadog, Splunk, an S3 archive) or doing log enrichment at collection time. If you don't know yet whether you need FireLens, you don't need FireLens.

Why does my alarm not fire even when CPU is at 100%?

Check the statistic — if you set Average and only one out of five tasks is maxing CPU, the average might stay below your threshold. For services with multiple tasks, use Maximum for CPU alarms or alarm on a per-task basis using Container Insights metrics.

How much does CloudWatch logging cost?

Roughly $0.50 per GB ingested and $0.03 per GB stored per month. For a typical startup service logging a few hundred MB per day, expect $5–20/month. Setting a 30-day retention policy keeps storage costs flat. The Container Insights add-on is roughly $0.50/month per resource monitored.

ECS Logs and CloudWatch: Why Your Container Logs Are Disappearing (And What to Actually Monitor)

ECS Logs and CloudWatch: Why Your Container Logs Are Disappearing (And What to Actually Monitor)

How ECS Logging Works (The 30-Second Version)

Setting Up the awslogs Driver

The Log Stream Structure: Why You Can't Find the Right Log

Searching Logs: CloudWatch Insights

Log Retention: Why Your Logs Disappear After a Few Months

What to Monitor: The Metrics That Actually Matter

Enabling Container Insights (Worth the Cost)

The Three Alarms Every ECS Service Needs

Alarm 1: High Memory

Alarm 2: Task Count Below Desired (requires Container Insights)

Alarm 3: High CPU (for scaling trigger, not paging)

The Log-to-Alarm Workflow During an Incident

How NoahOps Handles This for You

FAQ

My container is running but I see no logs in CloudWatch. What's wrong?

How do I see logs from a crashed container?

Should I use awslogs or FireLens?

Why does my alarm not fire even when CPU is at 100%?

How much does CloudWatch logging cost?

Ready to ship to AWS?