How to Set Up SSH Key Authentication on Linux

Quick take: SSH key authentication is the single most effective change you can make to secure a Linux server. It takes ten minutes to set up and eliminates the entire class of password brute-force attacks permanently.

Introduction

The moment a Linux server gets a public IP, it is under attack. On a fresh VPS I deployed for a client last year, running grep "Failed password" /var/log/auth.log | wc -l returned over 4,000 attempts within 48 hours — from automated bots scanning every IP on the internet around the clock. A strong password helps, but it is still a credential that can be phished, leaked in a data breach, or captured on a compromised network.

SSH key authentication eliminates this entire category of attack. Instead of something you know, you use something you physically possess — a cryptographic key pair stored on your machine. The math behind it makes brute-forcing computationally impossible. Combined with disabling password login entirely, it removes the most targeted entry point on any internet-facing Linux server.

How SSH Key Authentication Works

SSH keys come in pairs: a private key and a public key. The private key stays on your local machine and never leaves it. The public key goes on the server inside a file called authorized_keys. These two keys are mathematically linked — what one encrypts, only the other can decrypt.

When you connect, the server issues a random challenge encrypted with your public key. Your SSH client uses your private key to solve it and sends back proof. The server verifies the proof without ever seeing the private key or transmitting a password. An attacker watching the network captures nothing useful.

Ed25519 is the recommended algorithm for new key pairs — modern, compact, and faster than RSA. RSA-4096 remains the choice when connecting to legacy systems that do not support Ed25519.

Step 1: Generate Your Key Pair

Run this on your local machine — your laptop or workstation, not the server:

ssh-keygen -t ed25519 -C "your-email@example.com"

The command asks three things: where to save the key (accept the default ~/.ssh/id_ed25519 or give it a name), a passphrase to encrypt the key file on disk, and confirmation of the passphrase. Always use a passphrase on keys stored on laptops or shared machines.

This creates two files:

~/.ssh/id_ed25519        # private key — never share or upload this
~/.ssh/id_ed25519.pub    # public key — this goes on the server

Verify permissions — SSH refuses keys with loose permissions:

ls -la ~/.ssh/id_ed25519
# Expected: -rw------- 1 youruser youruser ...

# Fix if wrong:
chmod 600 ~/.ssh/id_ed25519
chmod 700 ~/.ssh/

For legacy systems that require RSA, use: ssh-keygen -t rsa -b 4096 -C "your-email@example.com". Never use RSA below 2048 bits — 1024-bit RSA is considered broken.

Step 2: Copy the Public Key to the Server

The easiest method when you can still log in with a password:

ssh-copy-id -i ~/.ssh/id_ed25519.pub username@server-ip

It asks for your server password one final time, then appends the public key to ~/.ssh/authorized_keys with the correct permissions automatically.

When ssh-copy-id is unavailable, do it manually. On your local machine, print the public key:

cat ~/.ssh/id_ed25519.pub
# Output: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... you@example.com

Log into the server with a password and run:

mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "ssh-ed25519 AAAA...your-full-key-here..." >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

The permissions on these files are not optional. SSH silently refuses to use an authorized_keys file that is world-readable, or any key if the ~/.ssh directory is world-writable.

Step 3: Configure the SSH Daemon

Once key login is confirmed to work, disable password authentication to lock down the server:

sudo nano /etc/ssh/sshd_config

Set these values — uncomment any lines that start with #:

PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PermitRootLogin no
LoginGraceTime 30
MaxAuthTries 3

Save and reload the daemon. Use reload, not restart — reload keeps your existing session alive:

sudo systemctl reload sshd

Critical: always keep a second terminal window open when changing SSH config. Test the new connection from a fresh terminal before closing your working session. A misconfigured sshd_config with no working session to fix it is how engineers get locked out of production servers at 2am.

Step 4: Test and Verify

From a new terminal on your local machine:

ssh username@server-ip

It should connect without asking for a server password (it may ask for your key passphrase, which is a local operation). Verify which auth method was used:

ssh -v username@server-ip 2>&1 | grep -E "Authentication succeeded|Authentications that"

You should see debug1: Authentication succeeded (publickey). If password still appears in the methods list, password auth is still enabled — recheck /etc/ssh/sshd_config and confirm you reloaded the daemon.

Managing Multiple Servers with SSH Config

Once you manage more than two or three servers, typing full connection strings becomes error-prone. The file ~/.ssh/config lets you define aliases for each server:

nano ~/.ssh/config

Host prod-web
    HostName 203.0.113.10
    User ubuntu
    IdentityFile ~/.ssh/id_ed25519
    Port 22

Host staging
    HostName staging.example.com
    User deploy
    IdentityFile ~/.ssh/deploy_key
    Port 2299

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    AddKeysToAgent yes

chmod 600 ~/.ssh/config

Now connecting to production is simply ssh prod-web. The config also works with scp, rsync, Git over SSH, and any tool that uses SSH under the hood. The global Host * block sends keepalive packets every 60 seconds, preventing idle disconnections during long operations.

Other useful per-host options:

ForwardAgent yes       # Forward local SSH agent — lets you git pull on server using local key
ConnectTimeout 10      # Fail fast if server is unreachable
IdentitiesOnly yes     # Only try the specified key, ignore all agent keys

Using SSH Agent for Passphrases

If your private key has a passphrase (it should), you would normally type it on every connection. SSH agent caches the decrypted key in memory so you only type the passphrase once per session:

# Start the agent (usually running automatically on Linux desktops)
eval $(ssh-agent -s)

# Add your key — prompts for passphrase once
ssh-add ~/.ssh/id_ed25519

# List cached keys
ssh-add -l

On macOS, add AddKeysToAgent yes and UseKeychain yes to your SSH config. The macOS Keychain then remembers the passphrase across reboots so you only type it once ever.

For automated scripts and CI/CD pipelines where no human is present to type a passphrase, generate a dedicated key without a passphrase and store it in a secrets manager. Name it clearly — ~/.ssh/deploy_key — so it is never confused with a personal key.

Security Best Practices

Switching to key auth is the essential first step. These practices complete the picture on any production server I manage:

• Change the default SSH port from 22. Moving SSH to a high-numbered port (e.g., 10022 or 59200) eliminates almost all automated scan noise. It will not stop a targeted attacker, but it removes the constant background hammering from opportunistic bots.
• Generate separate keys for different purposes. One key for personal servers, one for client A, one for client B, one for GitHub. If one is compromised, revoke only that key without touching anything else.
• Rotate keys at least annually. Or immediately when someone with key access leaves your organisation. Key rotation across a fleet of servers can be scripted in minutes with a simple bash loop or Ansible.
• Audit authorized_keys files regularly. Know exactly which keys are on which servers. A departed employee's key left in place is an open door.
• Back up private keys securely. Store an encrypted copy in a reputable password manager. Losing your only private key when the server accepts no passwords is a very bad situation.
• Run Fail2Ban alongside key auth. Key auth prevents password attacks from succeeding, but Fail2Ban stops repeated connection attempts from filling your logs and wasting server resources.

Troubleshooting Common Issues

Permission denied (publickey): Almost always a file permission issue on the server. Check:

ls -la ~/.ssh/              # must be drwx------ (700)
ls -la ~/.ssh/authorized_keys  # must be -rw------- (600)

# The home directory itself must not be world-writable:
ls -ld ~

Too many authentication failures: Your SSH agent has too many keys loaded and the server disconnects after hitting MaxAuthTries. Force the right key:

ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes username@server-ip

Host key changed warning: The server's fingerprint differs from what was previously recorded. Investigate before proceeding — it could mean the server was rebuilt, or it could indicate something more serious. If you know the server was reinstalled:

ssh-keygen -R server-ip-or-hostname
# Then reconnect and accept the new fingerprint

Verbose output for diagnosis: Add -vvv to any SSH command to see exactly what it tries and why it fails:

ssh -vvv username@server-ip 2>&1 | head -60

Final Thoughts

SSH key authentication is one of those changes that takes ten minutes to set up and pays dividends indefinitely. Once it is in place with password login disabled, the entire brute-force attack category simply disappears from your threat surface. Pair it with Fail2Ban, a non-default SSH port, and a properly configured firewall, and your server's attack surface becomes very small.

The steps in this guide — generate, copy, configure, test — follow the same pattern whether you are setting up one server or a hundred. Build the habit of testing before closing sessions, always using passphrases on laptop keys, and keeping an audit of which keys are authorised on which servers. These habits are what separate a well-managed infrastructure from a liability.