CPU Monitoring

CPU Monitoring

CPU monitoring tells you what the processor is actually doing. The most important distinction is between user CPU (application code running) and system CPU (kernel code running on behalf of the application). High system CPU often indicates too many system calls, context switches, or interrupt handling — problems that are not visible from application-level metrics alone. Another critical metric is steal time (st in top), which appears on virtual machines when the hypervisor is stealing CPU time from your VM.

CPU states explained

Using top and htop

top    # Press 1 to show per-CPU stats, P to sort by CPU, M to sort by memory

top header lines with CPU breakdown

top - 14:30:01 up 7 days,  load average: 2.15, 1.94, 1.82
Tasks: 234 total,   3 running, 231 sleeping
%Cpu0:  45.3 us,  12.1 sy,  0.0 ni, 38.2 id,  4.2 wa,  0.0 st
%Cpu1:  67.8 us,   8.4 sy,  0.0 ni, 21.4 id,  2.3 wa,  0.2 st
          ↑           ↑                   ↑          ↑          ↑
       user CPU    system CPU          idle        iowait     steal

%wa = 4.2% → some I/O wait, not critical
%st = 0.2% → minimal steal, hypervisor not a problem here

# htop — better interactive view:
# F6: sort by any column
# F5: tree view (show parent-child process relationships)
# Space: tag a process for batch operations
htop

Per-core analysis with mpstat

# mpstat shows per-CPU utilization:
mpstat -P ALL 1 3    # All CPUs, 1-second interval, 3 samples

mpstat -P ALL output

CPU  %usr  %sys  %iowait  %idle
all  34.5   8.2     2.1   55.2
  0  78.3  10.1     0.5   11.1   ← CPU 0 is hot (single-threaded bottleneck?)
  1  23.4   7.8     2.9   65.9
  2  18.2   6.5     3.8   71.5
  3  18.1   8.5     1.4   72.0

# One hot CPU while others are idle = single-threaded workload bottleneck
# Find what's on the hot CPU:
ps -eo pid,psr,pcpu,comm | awk '$3 > 10 {print}' | sort -k3 -rn

CPU profiling with perf

# perf identifies which functions are consuming CPU:
sudo apt install -y linux-tools-generic linux-tools-$(uname -r)

# Record 30 seconds of CPU activity for process PID 1234:
sudo perf record -p 1234 -g sleep 30

# Show top functions by CPU time:
sudo perf report --stdio | head -30

perf report output

Overhead  Command  Symbol
  34.12%   myapp   [.] process_request
  18.52%   myapp   [.] json_parse
   9.21%   myapp   [k] __memcpy
   5.43%  [kernel]  [k] page_fault
→ process_request and json_parse are the hot spots — optimize there

Conclusion

For day-to-day CPU monitoring: watch load average relative to CPU count, check iowait (disk bottleneck) and steal (hypervisor stealing) in top. When investigating a CPU-bound process, use mpstat -P ALL to find if one core is saturated (single-threaded bottleneck) or if all cores are busy (genuinely CPU-bound, needs more hardware or code optimization). Use perf to identify which specific functions are consuming CPU before optimizing.