Understanding Linux Logs

Understanding Linux Logs

Logs are the primary source of truth for understanding what happened on a Linux system. They record authentication events, service starts and stops, application errors, kernel messages, and security events. On Ubuntu, logs come from two systems: the traditional text-based log files in /var/log/ maintained by rsyslog, and the binary journal maintained by systemd-journald. Understanding both systems and knowing which log to check for which type of event makes troubleshooting dramatically faster.

Log sources on Ubuntu

Log sources on Ubuntu:

  Kernel messages
      ↓
  systemd-journald ────────────────────────→ /var/log/journal/ (binary)
      ↓ (optionally forwarded)                queryable with journalctl
  rsyslog (text log daemon)
      ↓
  /var/log/ (text files)
  ├── syslog         — general system messages
  ├── auth.log       — authentication events
  ├── kern.log       — kernel messages
  ├── dpkg.log       — package install/remove history
  └── app-specific:
      ├── nginx/     — web server logs
      ├── mysql/     — database logs
      └── ...        — other application logs

Important log files

Reading log entries

# Traditional syslog format:
# TIMESTAMP  HOSTNAME  PROCESS[PID]: MESSAGE
tail -20 /var/log/auth.log

/var/log/auth.log entries explained

Jun  9 14:30:05 server sshd[1234]: Accepted publickey for irfan from 192.168.1.5 port 54321 ssh2
#     ↑              ↑      ↑       ↑                       ↑                ↑
# timestamp    hostname  process  event                   user           source IP

Jun  9 14:31:20 server sshd[5678]: Failed password for invalid user admin from 1.2.3.4 port 44444 ssh2
# → Brute force attempt — username "admin" doesn't exist

# View logs with more context
less /var/log/syslog    # Scrollable view
zcat /var/log/syslog.1.gz | less    # Previous day's rotated log

Searching logs effectively

# grep: search for patterns in log files
grep "Failed password" /var/log/auth.log | tail -20
grep "error" /var/log/nginx/error.log | grep "$(date +%Y/%m/%d)"

# Combine with awk to extract fields:
# Find top attacking IPs from auth.log
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -10

# Real-time monitoring of multiple logs
sudo tail -f /var/log/auth.log /var/log/syslog

# Use journalctl for systemd-managed service logs (more reliable than log files)
sudo journalctl -u nginx -f            # Follow nginx logs
sudo journalctl --since "1 hour ago" -p err  # Errors from last hour
sudo journalctl -b -p warning          # All warnings from current boot

Log security and integrity

# Log files should not be writable by service accounts
# Check permissions:
ls -la /var/log/auth.log /var/log/syslog

# If an attacker has root, they can delete logs — consider remote log shipping
# Configure rsyslog to forward to a remote syslog server:
sudo nano /etc/rsyslog.conf
# Add: *.* @syslog-server:514    (UDP) or @@syslog-server:514 (TCP)
sudo systemctl restart rsyslog

# Log entries should be timestamped with timezone info
# Check rsyslog timestamp format in /etc/rsyslog.conf:
# $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat   # has timezone

Conclusion

Know which log file to check for each type of event: /var/log/auth.log for authentication events, journalctl -u servicename for service logs, dmesg or /var/log/kern.log for hardware/kernel issues. Use grep for searching text logs and journalctl with --since and -p (priority) flags for journal queries. Ship logs to a remote server so an attacker cannot cover their tracks by deleting local logs — this is essential for any server where security is important.