Out of Memory Errors

Out of Memory Errors

When a Linux server runs out of physical memory and swap, the kernel OOM (Out of Memory) killer activates and kills the process it considers most expendable — typically the largest non-critical process. In production, this usually kills your database or web server. The OOM killer is the kernel's emergency brake, not a normal operating mode. If your server regularly triggers the OOM killer, you have a memory management problem that needs fixing, not just adding more RAM.

How the OOM killer works

OOM kill sequence:

  Physical RAM: 8GB (used: 7.9GB)
  Swap: 2GB (used: 1.9GB)
  New allocation request: 500MB

  1. Kernel: no free memory + no free swap
  2. OOM killer activated
  3. Scores each process (score = memory used * runtime factors)
  4. Kills process with highest OOM score
  5. Usually kills the largest non-critical process

  Common victims: MySQL (large buffer pool), Java heap,
                  Node.js with memory leak, PHP-FPM workers

Identifying OOM kills

# Check if OOM killer has been active:
sudo dmesg | grep -i "oom\|killed process" | tail -20

dmesg OOM kill output

[Mon Jun 09 14:30:05] [1234567.890123] Out of memory: Kill process 5678 (mysqld) score 892 or sacrifice child
[Mon Jun 09 14:30:05] [1234567.891234] Killed process 5678 (mysqld) total-vm:8345678kB, anon-rss:7234567kB
# MySQL was killed! Score 892 made it the highest-priority target

# Also check syslog and journal:
sudo journalctl -k | grep -i "oom\|killed" | tail -20
sudo grep -i "oom\|killed" /var/log/syslog | tail -20

# Check current memory pressure:
free -h

free -h output showing memory pressure

               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       7.6Gi       256Mi       456Mi       512Mi       234Mi
Swap:          2.0Gi       1.9Gi       156Mi
# available: 234Mi — nearly all memory consumed, swap nearly full
# Risk of OOM kill is very high

Finding memory-hungry processes

# Sort processes by memory usage:
ps aux --sort=-%mem | head -15

# Check memory over time (vmstat at 5-second intervals):
vmstat -s    # Memory statistics snapshot
vmstat 5 12  # 12 samples at 5-second intervals (look for si/so columns — swap in/out)

# smem shows proportional set size (true memory per process):
sudo apt install -y smem
sudo smem -r -s pss | head -15    # Sort by PSS (proportional share)

OOM prevention

# Protect critical processes from OOM kill (lower score = less likely to be killed):
# Set oom_score_adj = -1000 to completely exclude from OOM killing:
echo -1000 > /proc/$(pgrep -f mysqld)/oom_score_adj    # Immediate

# Persist across restarts (in systemd service):
sudo nano /etc/systemd/system/mysql.service.d/override.conf

/etc/systemd/system/mysql.service.d/override.conf

[Service]
OOMScoreAdjust=-900    # -1000 = never kill (use with caution)

# Check OOM score for a process:
cat /proc/$(pgrep mysqld)/oom_score

# Add swap if server has none (immediate relief, not a fix):
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Set swappiness lower for servers (default 60 is for desktops):
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.d/99-memory.conf
sudo sysctl -p /etc/sysctl.d/99-memory.conf

Conclusion

The oom_score_adj = -1000 for MySQL prevents the OOM killer from killing your database, but it means some other process will be killed instead when memory is exhausted. The real fix is finding why memory is being exhausted: check for memory leaks (processes growing memory without releasing it over time), oversized application memory settings (MySQL innodb_buffer_pool_size set too large), or PHP processes not being recycled (pm.max_requests = 0 means workers never restart and can leak memory indefinitely).