Journald Logging

Journald Logging

Journald is systemd's logging daemon. It collects log output from every service that has StandardOutput=journal (the default for systemd services), kernel messages, boot messages, and anything written to syslog. Unlike traditional syslog files, journal entries are structured binary data with metadata like unit name, PID, and priority — making filtering precise and fast.

What is journald?

Log sources → journald → binary journal files → journalctl

Sources:
  - Service stdout/stderr (if StandardOutput=journal)
  - Kernel ring buffer (dmesg)
  - Syslog socket (/dev/log)
  - systemd itself (service start/stop events)
  - Audit daemon (auth events)

Storage:
  /run/log/journal/    — volatile (lost on reboot, always available)
  /var/log/journal/    — persistent (survives reboot, must be enabled)

Basic journalctl usage

# Show all logs (newest last, uses pager)
journalctl

# Show newest logs first
journalctl -r

# Follow logs in real time (like tail -f)
journalctl -f

# Show kernel messages (like dmesg)
journalctl -k

# Show logs from current boot only
journalctl -b

# Show logs from previous boot
journalctl -b -1

Filtering logs

# Filter by service unit (most useful daily command)
journalctl -u nginx.service
journalctl -u nginx.service -f    # Follow nginx logs
journalctl -u nginx.service -n 50 --no-pager  # Last 50 lines

# Filter by time range
journalctl --since "2026-06-09 14:00:00"
journalctl --since "2026-06-09 14:00:00" --until "2026-06-09 15:00:00"
journalctl --since "1 hour ago"
journalctl --since today

# Filter by priority level (0=emerg to 7=debug)
journalctl -p err           # Only errors (priority 3 and above)
journalctl -p warning       # Warnings and above
journalctl -p debug         # All messages

# Filter by user
journalctl _UID=1000        # Logs from UID 1000 processes

# Output formats
journalctl -u nginx -o json-pretty  | head -30    # JSON format
journalctl -u nginx -o cat          # No metadata, just message

Typical journalctl -u nginx output

Jun 09 14:30:05 server systemd[1]: Starting A high performance web server...
Jun 09 14:30:05 server nginx[1234]: nginx: the configuration file /etc/nginx/nginx.conf test is successful
Jun 09 14:30:05 server systemd[1]: Started A high performance web server.
Jun 09 15:12:03 server nginx[1234]: 2026/06/09 15:12:03 [error] 1235#1235: *1 connect() failed (111: Connection refused)

Persistent journal storage

# By default on Ubuntu, journal storage is volatile (/run/log/journal)
# Logs are lost on reboot
# Check current storage location:
journalctl --disk-usage

# Enable persistent storage
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

# Verify persistence is active
journalctl --disk-usage
# Now shows /var/log/journal usage

Managing journal size

# Configure journal size limits in /etc/systemd/journald.conf
sudo nano /etc/systemd/journald.conf

Key journald.conf settings

[Journal]
Storage=persistent          # auto, persistent, volatile, or none
Compress=yes                # Compress older entries
SystemMaxUse=2G             # Max total size on disk
SystemKeepFree=500M         # Always keep this much disk free
MaxRetentionSec=1month      # Delete entries older than 1 month
MaxFileSec=1week            # Rotate journal files weekly

# Apply configuration
sudo systemctl restart systemd-journald

# Manually vacuum old entries (without config change)
sudo journalctl --vacuum-size=500M      # Keep newest entries up to 500M
sudo journalctl --vacuum-time=30d       # Delete entries older than 30 days

# Check current disk usage
journalctl --disk-usage

Conclusion

The most useful daily journalctl commands: journalctl -u servicename -f to follow a service's logs in real time, journalctl -u servicename -n 100 --no-pager to review recent logs, and journalctl -p err -b to see all errors from the current boot. Enable persistent storage with mkdir /var/log/journal if you need logs to survive reboots. Set SystemMaxUse=2G in journald.conf to prevent the journal from filling your disk — on busy servers, the default uncapped behavior will eventually cause problems.