Server Recovery After Reboot

Server Recovery After Reboot

A server that works perfectly while running can fail in unexpected ways after a reboot. Services that were started manually (not enabled via systemd) do not restart, NFS mounts and network filesystems may not reconnect, application startup order dependencies may cause services to start before their dependencies are ready, and kernel updates can change driver behavior. A systematic post-reboot verification process catches these issues quickly.

Why reboots break things

Common post-reboot failures:

  Service not enabled (systemctl enable was never run):
    → Was started manually, died on reboot, stays dead

  Service starts too fast (dependency not ready):
    → MySQL starts before the RAID array is fully initialized
    → Application starts before MySQL is ready to accept connections

  Kernel module change (after kernel upgrade):
    → Custom NIC driver no longer loaded
    → CPU performance governor reset to default

  NFS/CIFS mount fails:
    → Network not ready when /etc/fstab mounts run
    → Remote server certificate expired

  Hardcoded IP addresses:
    → IP changed after reboot (DHCP lease expired)
    → Application config points to old IP

Services not starting after reboot

# Check which services are failing after reboot:
sudo systemctl --failed

systemctl --failed output

  UNIT           LOAD   ACTIVE SUB    DESCRIPTION
* myapp.service  loaded failed failed My Application
* nfs-mount.service loaded failed failed NFS Mount

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state
SUB    = The low-level unit activation state

2 loaded units listed.

# Investigate each failed service:
sudo journalctl -u myapp.service -b    # Logs since last boot (-b flag)
sudo journalctl -u nfs-mount.service -b

# Check if service is enabled (will auto-start on boot):
systemctl is-enabled myapp.service
# enabled = will start on boot
# disabled = will NOT start on boot

# Enable a service:
sudo systemctl enable --now myapp.service

# Fix startup ordering (service depends on MySQL being ready):
sudo nano /etc/systemd/system/myapp.service

/etc/systemd/system/myapp.service — with proper ordering

[Unit]
Description=My Application
After=network.target mysql.service    # Start after network AND MySQL
Requires=mysql.service                # Fail if MySQL not running

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5    # Wait 5 seconds before restart attempt

[Install]
WantedBy=multi-user.target

Filesystem and mount issues

# Check if all expected filesystems are mounted:
df -h    # Shows mounted filesystems
mount | grep -v "tmpfs\|proc\|sys\|devpts"    # Non-virtual mounts
cat /etc/fstab    # Expected mounts

# If an NFS mount failed silently, the application may be writing to local disk:
ls -la /mnt/nfs-share    # If empty or missing → not mounted

# Mount all fstab entries that are not yet mounted:
sudo mount -a

# Check fstab entry for common mistakes:
# Missing _netdev option for network mounts (prevents boot hang if network slow):
# //server/share  /mnt  cifs  credentials=/etc/cifs.cred,_netdev  0  0

# Check if a disk changed device name after reboot (sda vs sdb):
sudo blkid    # Show devices by UUID
sudo lsblk    # Show block devices with their mount points
# Use UUID in /etc/fstab instead of /dev/sda1 to avoid ordering changes

Post-reboot verification checklist

# Run this verification sequence after every reboot:

# 1. Check for failed services:
sudo systemctl --failed

# 2. Verify critical services are running:
for svc in nginx mysql php8.1-fpm redis; do
    status=$(systemctl is-active $svc 2>/dev/null)
    echo "$svc: $status"
done

# 3. Check disk mounts:
df -h | grep -v tmpfs

# 4. Check logs for errors during boot:
sudo journalctl -b -p err    # Errors from current boot only

# 5. Verify network is correct:
ip addr show
ip route show
ping -c 3 8.8.8.8

# 6. Test application endpoints:
curl -s http://localhost/health | head -5
curl -s https://example.com/ | head -5

Conclusion

After any planned maintenance reboot, run through the verification checklist before declaring the server healthy. The journalctl -b -p err command is particularly valuable — it shows only error-level messages from the current boot, filtering out the noise of normal startup messages. Use After= and Requires= in systemd unit files to express service dependencies explicitly so systemd starts services in the correct order automatically.