Disk Performance Monitoring

Disk Performance Monitoring

Disk I/O is one of the most common performance bottlenecks on Ubuntu servers running databases, log aggregation, or container workloads. Before you can tune anything, you need to be able to measure and understand disk performance metrics. Knowing the difference between latency, throughput, and I/O wait — and which tools measure each — is the foundation of storage performance work.

Key disk metrics explained

Monitoring with iostat

# Install sysstat (includes iostat)
sudo apt install -y sysstat

# Basic disk stats (one snapshot)
iostat -d

# Extended stats with human-readable output, update every 2 seconds
iostat -xhd 2

iostat -xhd output

Device      r/s   w/s  rMB/s  wMB/s  rrqm/s  wrqm/s  r_await  w_await  aqu-sz  %util
sda        2.50  45.5    0.1    2.3     0.0     8.2      0.5ms  125.5ms    5.7    98%
sdb        5.00   1.0    2.0    0.1     0.0     0.0      0.8ms    1.2ms    0.0    12%

← sda is saturated: 98% util, 125ms write latency, large queue (aqu-sz=5.7)

# Monitor continuously (3-second intervals)
iostat -xhd 3

# Focus on specific device
iostat -xhd 3 sdb

# Log to file for analysis
iostat -xhd 3 > /tmp/iostat-$(date +%Y%m%d-%H%M).log &

Monitoring with iotop

# Install iotop
sudo apt install -y iotop

# Show processes with active disk I/O
sudo iotop -o    # -o = only show processes doing I/O

iotop -o output

Total DISK READ:    0.00 B/s | Total DISK WRITE:   42.45 M/s
Actual DISK READ:   0.00 B/s | Actual DISK WRITE:  42.45 M/s
  TID  PRIO  USER   DISK READ  DISK WRITE  SWAPIN  IO>   COMMAND
23456  be/4  mysql    0.00 B/s  38.45 M/s   0.00% 95.02% mysqld
45678  be/4  postgres 0.00 B/s   4.00 M/s   0.00%  4.98% postgres

# Non-interactive mode (for scripts/logging)
sudo iotop -obn 3    # Batch mode, 3 iterations

# Sort by read or write
sudo iotop -o -a    # Accumulated I/O since start

Disk benchmarking

# Quick read speed test (uses OS cache — not a raw disk test)
dd if=/dev/sdb of=/dev/null bs=1M count=1024 status=progress

# Write test (bypasses cache with oflag=direct)
dd if=/dev/zero of=/tmp/disk-test.bin bs=1M count=1024 oflag=direct status=progress
rm /tmp/disk-test.bin

# Accurate benchmark with fio (industry standard)
sudo apt install -y fio

# Sequential read test (simulates large file reads)
sudo fio --name=seqread --filename=/dev/sdb --bs=1M --iodepth=1     --size=4G --rw=read --ioengine=libaio --direct=1 --runtime=30

# Random read IOPS test (simulates database random access)
sudo fio --name=randread --filename=/dev/sdb --bs=4k --iodepth=32     --size=4G --rw=randread --ioengine=libaio --direct=1 --runtime=30

fio result (NVMe SSD example)

  read: IOPS=245k, BW=956MiB/s (1003MB/s)(28.7GiB/30004msec)
  lat (usec): min=113, max=8765, avg=130.5, stdev=89.3

Identifying I/O bottlenecks

# Step 1: Check I/O wait in top (wa column)
top    # Look at %wa in CPU stats line
# wa > 10% = processes are waiting on I/O

# Step 2: Identify saturated devices with iostat
iostat -xhd 2    # Look for %util > 80%, high await

# Step 3: Find which process is causing the I/O
sudo iotop -o    # Find the top I/O consumer

# Step 4: Find which files the process is writing
sudo lsof -p PID | grep REG    # Regular files (not sockets/pipes)

# Step 5: Check if it's a scheduling issue
cat /sys/block/sda/queue/scheduler    # Current I/O scheduler
# Options: none (NVMe), mq-deadline, kyber, bfq
# For databases: echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

Conclusion

Use iostat -xhd 2 for real-time disk performance metrics — focus on %util (saturation) and w_await (write latency). Use iotop -o to identify which process is causing high I/O. Use fio for accurate disk benchmarking when you need to establish baseline performance or verify that new storage meets specifications. A disk at >80% utilization with latency exceeding 10ms for SSDs is a bottleneck that needs attention — either the workload needs optimization or additional storage is needed.