strace Command in Linux: Trace System Calls

Quick take: strace command runs a command and prints every system call it makes. strace -p PID attaches to a running process. Use -e trace=open,read,write to filter by call type, and -o file.txt to save output.

What Is strace in Linux?

strace is a diagnostic tool that intercepts system calls — the requests a process makes to the Linux kernel. Every file open, network connection, memory allocation, and signal delivery goes through a system call, and strace records them all with arguments and return values.

When a program fails silently, refuses to start, or behaves unexpectedly, strace often reveals the root cause in seconds: a missing config file, a permission denied on a socket, or a library that cannot be found at the expected path.

Install strace

# Ubuntu / Debian
sudo apt install strace

# RHEL / CentOS / Rocky
sudo dnf install strace

# Verify
strace -V

Syntax

# Trace a new command
strace command [args]

# Attach to a running process
strace -p PID

# Trace with output to file
strace -o trace.log command

Common Options

Practical Examples

# Trace all system calls made by ls
strace ls /etc

# Attach to a running nginx worker
sudo strace -p $(pgrep -n nginx)

# Show only file open calls with full path strings
strace -e trace=openat -s 256 myapp

# Count calls and show summary
strace -c wget https://example.com

# Trace a command and save to file
strace -o /tmp/app_trace.txt ./myapp

# Follow child processes (e.g. for shell scripts)
strace -f bash install.sh

# Show network-related calls only
strace -e trace=network curl http://example.com

# Show time spent in each call
strace -T ls /var/log

Reading strace Output

Each line shows: syscall(arguments) = return_value. A return of -1 with an error code means failure.

# Successful file open
openat(AT_FDCWD, "/etc/nginx/nginx.conf", O_RDONLY) = 3

# Permission denied
openat(AT_FDCWD, "/etc/shadow", O_RDONLY) = -1 EACCES (Permission denied)

# File not found
openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)

# Successful network connection
connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = 0

The error codes are the most valuable part: EACCES = permission denied, ENOENT = file not found, ECONNREFUSED = connection refused, ETIMEDOUT = connection timed out.

Real Debugging Scenarios

App fails to start with no error message:

strace -e trace=openat -s 256 ./myapp 2>&1 | grep "ENOENT\|EACCES"

This filters to only failed file opens, immediately showing missing config files or libraries.

Find what config file a program actually reads:

strace -e trace=openat -s 256 nginx -t 2>&1 | grep "= [0-9]"

Only successful opens (positive return values) — shows exactly which files were read.

Debug a service that hangs:

sudo strace -p $(pgrep myservice) -e trace=network,file

Shows whether the process is waiting on a network call, a file lock, or a missing resource.

Performance Impact of strace

strace has significant overhead because it stops and restarts each process for every system call via ptrace attach. The typical slowdown is 10–50x. Guidelines for safe use:

• Do not attach strace to a production process handling live traffic — use -c (count/summary mode) which has lower overhead for brief diagnostics
• For persistent production monitoring with minimal overhead, use eBPF-based tools like bpftrace or BCC (1–5% overhead)
• For diagnosing a startup failure — process not yet serving traffic — strace is safe and the right tool

# Low-overhead summary mode (-c) — safer for production, shows statistics not individual calls
sudo strace -c -p $(pgrep nginx | head -1)
# Press Ctrl+C after a few seconds to print the summary table

strace vs ltrace — System Calls vs Library Calls

strace intercepts Linux kernel system calls (the boundary between userspace and kernel). ltrace intercepts library function calls (glibc, libssl, libpq, etc.) in userspace. They reveal different layers of application behavior:

# Install ltrace
sudo apt install ltrace

# Trace shared library calls — malloc, fopen, SSL_connect, etc.
ltrace ./myapp 2>&1 | head -30

# Filter: SSL/TLS calls only
ltrace -e "SSL_*" ./myapp

Use strace when you suspect a kernel-level issue (file permissions, network connections, signal handling). Use ltrace when the issue is in application logic or library behavior (memory allocation failures, crypto errors).

Practical Debugging Workflows

Find why a service fails to start — check what config files it cannot open:

sudo strace -f -e trace=openat -s 256 systemctl start myservice 2>&1 |   grep "ENOENT\|EACCES"

Identify what port a process is binding to:

sudo strace -e trace=bind -p $(pgrep myapp) 2>&1 | grep bind

Measure time spent in each system call to find bottlenecks:

strace -c myapp
# After exit: sorted table shows which calls consumed the most time
# High time in read/write = I/O bound; high time in futex = lock contention

Common strace Errors and Fixes

Reference: strace man page — man7.org. Tested on Ubuntu 22.04 LTS with strace 5.16.

Final Thoughts

strace is not for everyday use — it imposes significant overhead and produces voluminous output. But when a program fails silently or behaves mysteriously, it is the most definitive debugging tool available at the OS level. Learn to filter its output with -e trace= and grep, and you will resolve in minutes issues that would otherwise take hours.