Memory Monitoring

Memory Monitoring

Linux memory management confuses many people because what looks like "used memory" in free is not what you think. Linux aggressively uses free RAM for disk cache (buff/cache), which improves performance but makes the system appear to have very little free memory even when applications have plenty of room. The number that matters for performance is "available" memory, not "free" — available includes cache that can be immediately reclaimed.

Linux memory concepts

Linux memory layout:
  Total RAM: 8GB

  Used by processes:    3.2GB  (RSS of all running processes)
  Kernel buffers:       0.3GB  (kernel internal structures)
  Page cache:           4.1GB  (cached file data — can be reclaimed instantly)
  Free (truly empty):   0.4GB

  free output shows:
    total  used  free  shared  buff/cache  available
    8GB    3.5GB 0.4GB  0.1GB    4.4GB      4.5GB
                  ↑                          ↑
            truly empty         free + reclaimable cache
            (rarely matters)    (this is what matters)

free and vmstat

free -h       # Human-readable memory summary
free -h -s 3  # Update every 3 seconds

free -h output interpreted

              total    used    free  shared  buff/cache  available
Mem:          7.7Gi   3.2Gi   412Mi   128Mi      4.1Gi      4.2Gi
Swap:         2.0Gi   256Mi   1.7Gi

→ available = 4.2GB: plenty of memory for new processes
→ 256MB swap used: some swap activity, worth investigating if growing

# vmstat shows real-time memory statistics:
vmstat 1 5

vmstat output — memory and swap columns

procs   --------memory--------  --swap--
 r  b   swpd   free   buff   cache   si   so
 1  0  256000  412000  98000  4100000   0    0   ← si/so = 0: no swapping
 2  1  258000  398000  98000  4088000   0    4   ← so=4: minor swap out
 0  2  260000  392000  98000  4080000  12   28   ← si/so > 0: memory pressure

# si = swap in (reading from swap to RAM)
# so = swap out (writing from RAM to swap)
# Any nonzero si/so = system is under memory pressure

Swap usage and OOM

# Check if OOM killer has been triggered:
dmesg | grep -i "oom\|killed"
journalctl -k | grep -i "oom\|killed"

OOM killer event in dmesg

kernel: Out of memory: Killed process 1234 (java) total-vm:4GB, anon-rss:2.5GB
→ Java process using 2.5GB RSS was killed to free memory
→ Fix: add more RAM, reduce JVM heap size (-Xmx), or add more swap

# Check swap usage per process:
for pid in /proc/[0-9]*; do
    pn=$(cat $pid/comm 2>/dev/null)
    sw=$(awk '/VmSwap/{print $2}' $pid/status 2>/dev/null)
    [[ $sw -gt 1024 ]] && echo "$sw KB $pn"
done | sort -rn | head -10

Finding memory hogs

# Sort processes by RSS (resident set size = actual RAM used):
ps aux --sort=-%mem | head -15

# smem provides more accurate per-process memory (requires install):
sudo apt install -y smem
smem -t -k    # -t = totals, -k = KB unit

# Check memory per process more accurately (PSS = proportional share of shared libs):
smem -s pss | head -15

smem output

User     Name         PID    RSS     PSS
root     java       12345  2.5GB   2.4GB
www-data nginx       3456   256MB   192MB
postgres postgres    5678   512MB   498MB

Conclusion

The key metric is "available" from free -h, not "free." When available drops below 10% of total RAM, the system will start swapping and performance degrades significantly. Monitor vmstat si/so (swap in/out) as a real-time indicator of memory pressure. OOM events in dmesg indicate the system has run out of memory and is killing processes — this needs immediate action (add RAM, reduce application memory usage, or tune application configuration).