Centralized Logging

Centralized Logging

When you manage more than 2-3 servers, reading logs by SSHing into each server individually becomes unmanageable. If a server is compromised, its local logs can be deleted or modified. Centralized logging solves both problems: all servers ship their logs to a central system, giving you a single place to search across all servers and tamper-evident audit trails. Two approaches are common: syslog forwarding (simple, built-in) and log shippers like Filebeat (richer, works with Elasticsearch).

Why centralize logs?

Without centralized logging:            With centralized logging:
  server1: ssh in, grep auth.log          Central log server / SIEM
  server2: ssh in, grep auth.log          ↑           ↑           ↑
  server3: ssh in, grep auth.log        server1     server2     server3
  → 30 servers = 30 SSH sessions          Logs forwarded immediately after write
  → Compromised server logs deleted       Attackers cannot delete already-shipped logs

Forwarding logs with rsyslog

# On each client server, configure rsyslog to forward to a central server:
sudo nano /etc/rsyslog.d/99-central.conf

/etc/rsyslog.d/99-central.conf (client)

# Forward all messages to central log server via TCP (reliable delivery)
*.* @@192.168.1.200:514

# Forward only critical messages to central server:
# *.crit @@192.168.1.200:514

# Queue messages locally if central server is unreachable:
$ActionQueueType LinkedList
$ActionQueueFileName fwdRule1
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on

sudo systemctl restart rsyslog

# Test: send a test message and verify it arrives at the central server
logger -t testapp "Test message from $(hostname)"

Setting up a central log server

# On the central log collection server:

# Enable rsyslog to listen on TCP/UDP port 514
sudo nano /etc/rsyslog.conf
# Uncomment these lines:
# module(load="imudp")
# input(type="imudp" port="514")
# module(load="imtcp")
# input(type="imtcp" port="514")

# Route incoming messages to per-host log files:
sudo nano /etc/rsyslog.d/10-remote.conf

/etc/rsyslog.d/10-remote.conf (server)

# Store logs from remote hosts in their own directory
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
if $fromhost-ip != '127.0.0.1' then ?RemoteLogs
& stop    ← stop processing remote logs after writing (don't also write to local syslog)

sudo mkdir -p /var/log/remote
sudo chown syslog:adm /var/log/remote
sudo chmod 755 /var/log/remote
sudo systemctl restart rsyslog

# Open firewall for syslog:
sudo ufw allow from 192.168.1.0/24 to any port 514 proto tcp
sudo ufw allow from 192.168.1.0/24 to any port 514 proto udp

Structured log shipping with Filebeat

# Filebeat (from Elastic) ships log files to Elasticsearch/Logstash
# Useful when you want structured querying and dashboards

# Install Filebeat:
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo add-apt-repository "deb https://artifacts.elastic.co/packages/8.x/apt stable main"
sudo apt install -y filebeat

# Configure /etc/filebeat/filebeat.yml:
# filebeat.inputs:
#   - type: log
#     enabled: true
#     paths:
#       - /var/log/nginx/access.log
#       - /var/log/auth.log
#
# output.logstash:
#   hosts: ["logstash-server:5044"]

sudo systemctl enable --now filebeat

Conclusion

For a simple multi-server setup (under 20 servers), rsyslog forwarding to a central server is sufficient and requires no additional software beyond what's already installed. For larger environments or when you need searchable dashboards, use Filebeat + Elasticsearch. The key practice: enable reliable forwarding with disk-buffered queues so logs are not lost if the central server is briefly unreachable. Once centralized logging is in place, auth events across your entire fleet are searchable from a single location.