SSH Access Lost

SSH Access Lost

Losing SSH access to a server is one of the most stressful incidents in system administration, particularly on cloud servers where physical console access is not immediately available. This almost always happens after making changes to SSH configuration, firewall rules, or network settings. Recovery options depend on what access you still have: cloud console, hypervisor console, or physical access. The best approach is prevention: always test SSH in a second session before closing the session used to make changes.

Preventing lockouts before they happen

Safe SSH configuration change procedure:

  1. Open TWO SSH sessions to the server
  2. In session 1: make the sshd_config change
  3. In session 1: run  sudo sshd -t  (validate config)
  4. In session 1: run  sudo systemctl reload sshd
  5. In session 2: try to SSH again in a NEW window
  6. If session 2 works: you are safe, close session 1
  7. If session 2 fails: session 1 is still open to fix the problem!

  Never close your only session while sshd is reloading.

Diagnosing SSH failure

# From a working machine, determine what is failing:

# Step 1: is the server reachable at all?
ping SERVER_IP

# Step 2: is port 22 open?
nc -zv SERVER_IP 22 -w 5

nc output — what each result means

# Port 22 open:
Connection to 203.0.113.5 22 port [tcp/ssh] succeeded!
# → SSH daemon is running but you cannot authenticate
# → Check: key file, username, key permissions

# Port 22 refused (Connection refused):
nc: connectx to 203.0.113.5 port 22 (tcp) failed: Connection refused
# → SSH daemon not running, or firewall blocking

# No response (timeout):
# → Firewall blocking, server unreachable, or wrong IP

# Verbose SSH connection (run from client to see exactly where it fails):
ssh -vvv user@SERVER_IP 2>&1 | head -50

# Common failure messages:
# "Permission denied (publickey)" → Wrong key or username
# "Connection refused" → sshd not running or firewall blocking
# "No route to host" → Network issue, server down
# "Connection timed out" → Firewall silently dropping packets

Console-based recovery

# Cloud providers have an emergency console (no SSH needed):
# AWS EC2: EC2 Instance Connect, Systems Manager Session Manager
# GCP: Cloud Console → Compute Engine → Serial Console
# Azure: Azure Portal → VM → Boot Diagnostics → Serial Console
# DigitalOcean: Dashboard → Droplet → Console (in browser)

# Once on console — fix common SSH issues:

# 1. sshd not running:
sudo systemctl start sshd
sudo systemctl status sshd    # Check for config errors

# 2. Firewall blocking SSH (UFW):
sudo ufw status
sudo ufw allow 22/tcp
sudo ufw reload

# 3. Wrong sshd_config (syntax error caused reload failure):
sudo sshd -t    # Test config
# If errors shown: fix /etc/ssh/sshd_config
sudo systemctl restart sshd

Emergency access methods

# Fix locked-out user (wrong authorized_keys):
# On console as root:
cat /home/username/.ssh/authorized_keys    # Check current keys
# Paste correct public key:
echo "ssh-ed25519 AAAA...your-pub-key" >> /home/username/.ssh/authorized_keys
chmod 600 /home/username/.ssh/authorized_keys
chown username:username /home/username/.ssh/authorized_keys

# Fix: PasswordAuthentication accidentally set to no (before key was added):
sudo nano /etc/ssh/sshd_config
# Change: PasswordAuthentication yes  (temporary — add key then set back to no)
sudo systemctl reload sshd

# Add firewall exception for your specific IP:
sudo ufw allow from YOUR_IP_HERE to any port 22
sudo ufw reload

💡 TIP: Before any change to SSH config, firewall rules, or network settings, schedule a task or cron job to revert the change in 5 minutes: echo "sudo ufw delete allow 22/tcp" | at now + 5 minutes. If your change locks you out, the scheduled task reverts it automatically, restoring access.

Conclusion

Cloud console access (AWS EC2 Instance Connect, GCP Serial Console, DigitalOcean in-browser console) is the primary recovery path for cloud servers when SSH fails — bookmark the console URL for every server you manage. On bare metal, IPMI/iDRAC/iLO remote console serves the same purpose. The most important prevention habit: always run sudo sshd -t to validate the sshd_config syntax before reloading, and always test in a second terminal before closing the session used to make changes.