Docker Logs and Debugging

Docker Logs and Debugging

Debugging Approach

When something goes wrong with a container, follow this order:

1. docker ps -a          <- Is the container running or stopped?
2. docker logs <name>    <- What did the container output?
3. docker inspect <name> <- What is the container's full config and state?
4. docker exec -it bash  <- Get inside and investigate
5. docker stats          <- Is it running out of memory or CPU?
6. docker events         <- What events happened?

docker logs - View Container Output

The most important debugging command.

# View all logs
docker logs my-app

# Follow logs in real time
docker logs -f my-app
docker logs --follow my-app

# Show last 50 lines
docker logs --tail 50 my-app
docker logs -n 100 my-app

# Show logs with timestamps
docker logs -t my-app
docker logs --timestamps my-app

# Show logs since a time
docker logs --since 1h my-app        # Last hour
docker logs --since 30m my-app       # Last 30 minutes
docker logs --since "2024-01-01T00:00:00" my-app

# Show logs until a specific time
docker logs --until "2024-01-01T12:00:00" my-app

# Combine - last 100 lines with timestamps, following
docker logs -f -t --tail 100 my-app

# View logs of a stopped/crashed container
docker logs my-crashed-app

docker inspect - Deep Dive Into Container State

Returns full JSON info about a container.

# Full inspect
docker inspect my-app

# Get exit code (useful for debugging crashes)
docker inspect -f '{{.State.ExitCode}}' my-app

# Get error message
docker inspect -f '{{.State.Error}}' my-app

# Get status
docker inspect -f '{{.State.Status}}' my-app
# Output: running, exited, paused, restarting, dead

# Check if container is running
docker inspect -f '{{.State.Running}}' my-app

# Get the container's IP address
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' my-app

# Get all environment variables
docker inspect -f '{{json .Config.Env}}' my-app

# Get mounted volumes
docker inspect -f '{{json .Mounts}}' my-app

# Get port bindings
docker inspect -f '{{json .NetworkSettings.Ports}}' my-app

# Get start time and finish time
docker inspect -f '{{.State.StartedAt}}' my-app
docker inspect -f '{{.State.FinishedAt}}' my-app

# Get restart count
docker inspect -f '{{.RestartCount}}' my-app

# Get the image used
docker inspect -f '{{.Config.Image}}' my-app

# Get labels
docker inspect -f '{{json .Config.Labels}}' my-app

docker exec - Debug Inside a Running Container

Opens a shell or runs a command inside a container.

# Open bash shell
docker exec -it my-app bash

# Open sh (for containers without bash, like Alpine)
docker exec -it my-app sh

# Run as root (for debugging permission issues)
docker exec -u root -it my-app bash

# Check environment variables
docker exec my-app printenv

# Check a specific env var
docker exec my-app printenv DB_HOST

# Check running processes
docker exec my-app ps aux

# Check open ports
docker exec my-app netstat -tlnp
docker exec my-app ss -tlnp

# Check disk space inside container
docker exec my-app df -h

# Check a config file
docker exec my-app cat /etc/nginx/nginx.conf

# Test network connectivity
docker exec my-app ping google.com
docker exec my-app curl http://db:3306

# Check installed packages (Ubuntu/Debian)
docker exec my-app dpkg -l

# Check a log file inside container
docker exec my-app tail -f /var/log/app.log

docker stats - Monitor Resource Usage

# Live stats for all containers
docker stats

# One-time snapshot
docker stats --no-stream

# Stats for a specific container
docker stats my-app

# Custom format
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

If a container is using 100% CPU or hitting its memory limit, it may be crashing or behaving unexpectedly.

docker events - Real-Time Docker Events

Useful for debugging unexpected container restarts or crashes.

# Stream all events
docker events

# Filter by container
docker events --filter "container=my-app"

# Filter by event type
docker events --filter "event=die"       # Container died
docker events --filter "event=oom"       # Out of memory kill
docker events --filter "event=restart"   # Container restarted
docker events --filter "event=health_status"  # Health check result

# Show past events (last 30 minutes)
docker events --since 30m

# JSON output for scripting
docker events --format '{{json .}}'

docker top - Show Container Processes

# Show running processes in a container
docker top my-app

# With ps options
docker top my-app aux
docker top my-app -eo pid,ppid,cmd,%mem,%cpu

docker diff - Show Filesystem Changes

Shows what files were added (A), changed (C), or deleted (D) since the container started.

docker diff my-app

Output:

A /app/logs/app.log      <- Added file
C /etc/nginx/nginx.conf  <- Changed file
D /tmp/old-file.txt      <- Deleted file

Debugging Common Scenarios

Scenario 1 - Container Keeps Restarting

# See how many times it has restarted
docker ps
# Shows "Restarting (1) N seconds ago"

# Check restart count
docker inspect -f '{{.RestartCount}}' my-app

# Check the exit code
docker inspect -f '{{.State.ExitCode}}' my-app
# 0 = clean exit, non-zero = error

# Check the error
docker inspect -f '{{.State.Error}}' my-app

# Check logs before it crashed
docker logs --tail 50 my-app

Scenario 2 - Container Exits Immediately

# Run the container interactively to see what's happening
docker run -it --entrypoint /bin/sh my-app

# Or override the command
docker run -it my-app /bin/bash

# Check the logs
docker logs my-app

Scenario 3 - Container Can't Connect to Database

# Get inside the app container
docker exec -it my-app bash

# Test connectivity to the database by name
ping db
curl http://db:3306

# Check if DNS is working
nslookup db

# Check environment variables
printenv DB_HOST
printenv DB_PORT

Scenario 4 - Out of Memory (OOM Kill)

# Check for OOM kill event
docker events --filter "event=oom"

# Check exit code (137 = OOM kill)
docker inspect -f '{{.State.ExitCode}}' my-app

# Check memory usage
docker stats --no-stream my-app

# Increase memory limit
docker update --memory 1g my-app

Scenario 5 - Port Already in Use

# Check what's using the port on the host
netstat -tlnp | grep 8080

# Check Docker port mappings
docker port my-app

# Run on a different host port
docker run -d -p 9090:80 my-app

Logging Drivers

Docker supports multiple logging drivers that control where container logs are stored.

# Check current logging driver
docker info --format "{{.LoggingDriver}}"

# Run a container with a specific logging driver
docker run -d \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  my-app

# Available logging drivers
# json-file    <- Default, stored as JSON on disk
# local        <- Optimized local storage
# syslog       <- Send to system syslog
# journald     <- Send to systemd journal
# gelf         <- Graylog Extended Log Format
# fluentd      <- Send to Fluentd
# awslogs      <- Send to AWS CloudWatch
# splunk       <- Send to Splunk
# none         <- Discard all logs

# Run with no logging
docker run -d --log-driver none my-app

Debugging Quick Reference