Linux Performance Fundamentals

Linux Performance Fundamentals

Linux performance problems fall into four categories: CPU saturation (processes waiting for CPU time), memory pressure (processes using swap or being OOM-killed), I/O wait (processes blocked waiting for disk), or network bottleneck (bandwidth saturated or high latency). The first step in any performance investigation is identifying which resource is the bottleneck — without this, tuning efforts are guesswork. This article establishes the mental model and workflow before diving into specific tools.

Four pillars of Linux performance

Performance bottleneck identification:

CPU: high load average, high %usr or %sys in top, low %idle
  → Process is compute-bound or kernel-heavy

Memory: swapping (si/so in vmstat), high kswapd, OOM events in dmesg
  → Add RAM or reduce process memory usage

I/O: high %iowait in top/sar, high await in iostat, D-state processes
  → Storage is the bottleneck (slow disk, RAID degraded, NFS stall)

Network: interface at 100% utilization, high retransmits in ss/netstat
  → Bandwidth saturated, or packet loss causing retransmissions

Understanding load average

uptime

uptime output

14:30:01 up 7 days, 3:12, 2 users, load average: 1.25, 0.94, 0.82
                                                    ↑     ↑     ↑
                                                  1min  5min  15min

Number of CPU cores matters:
  Load 1.0 on a 1-core server = 100% utilized (one task always waiting)
  Load 1.0 on a 4-core server = 25% utilized (plenty of headroom)
  Load 4.0 on a 4-core server = 100% — processes are queuing for CPU
  Load 8.0 on a 4-core server = overloaded — investigate immediately

# Number of CPU cores:
nproc
grep -c processor /proc/cpuinfo

# Rule of thumb: load average/nproc > 0.8 = investigate, > 1.0 = actively degraded

Investigation workflow

# 60-second performance investigation (run these first, in order):
uptime               # Load average and how long the system has been up
dmesg | tail -10     # Recent kernel errors (OOM kills, I/O errors, hardware)
vmstat 1 5           # CPU, memory, swap, I/O every second for 5 seconds
mpstat -P ALL 1 3    # Per-CPU utilization
pidstat 1 3          # Per-process CPU usage
iostat -xz 1 3       # Disk I/O utilization
free -h              # Memory usage
sar -n DEV 1 3       # Network interface utilization

Key tools overview

sudo apt install -y sysstat htop

Conclusion

Start every performance investigation by identifying the bottleneck resource (CPU, memory, I/O, network) before looking at specific processes. Load average alone does not tell you which resource is under pressure — combine it with vmstat (shows CPU steal, swap activity) and iostat (shows disk wait) to identify the cause. The 60-second investigation checklist above gives you a complete picture of system health in one minute.