SSH Keys Demystified

Posted bymarwalk

Using SSH (Secure Shell) with crypto keys can be a pull your hair out exercise in frustration—not only is the discussion on the Internet sometimes more snooty than helpful, some of the advice is outright wrong and in following it you can disable your system. The points below should get you to workable SSH connections on your Linux machines (and, with some tweaks for different utilities and system personality, your Mac and Windows machines too).

I use SSH routinely between my Linux computers, manually and in bash scripts. This also includes using the SFTP subsystem (SSH File Transfer Protocol). I avoid using passwords by configuring the SSH server services on each machine to use SSH keys. These key pairs are generated with the command ssh-keygen, which I won’t elaborate on here as the documentation on that is fairly straightforward—it’s after key generation when the trouble (err, fun) starts. As SSH is part of the basic Linux build, it’s terminal commands are the same on mostly all Linux distributions (Ubuntu, Fedora, etc.).

If at all possible, start with the server and client machines physically next to you (before deploying the server to your operational site after set up confirmation). This allows for local intervention if necessary in case a remote connection, such as via password, breaks after configuration changes. If this is not part of your available configuration, such as the server being on a remote hosted platform, there should be a help desk to contact for assistance—otherwise try to have arrangements with someone on site at the remote server who could carry out local tasks there if needed.

Here are the things that can be stumbling blocks in setting up a system to use SSH keys:

  • File permissions (and ownership) on the various key directories and files
  • Placing the keys into the proper directories, on both the server and client side
  • “Installing” your public key(s) in the user .ssh/authorized_keys file on the remote (server) machine—this requires that you have a corresponding user account on the server to match the user account on the client you are logging into the server from
  • Properly editing your server side sshd_config file (must be done as root)

Let’s start with permissions, as those are immediate show stoppers at several levels. Keep these permissions in mind, check and double check them. Here’s what you should have:

  • Directory on the server:1
    • the /etc/ssh directory permissions should be the normal 755, thus the ls -l listing of it should display as drwxr-xr-x
    • owner is root
  • Directories on the clients:
    • for all user /home/<user>/.ssh and /root/.ssh directories, the permissions must be 700, thus all of their ls -l listings should display as drwx------ (anything looser will break SSH, and properly so)
    • owner is the <user>, which is root in the case of the /root “home” directory (and different from the /home/<user> directory paths)
  • On all key files (usually with “key” in their file names) in both the user and root .ssh directories, the following permissions are required:
    • Private keys:
      • the permissions must be 600, thus the ls -l listing should display as -rw------- (anything looser will break SSH, and properly so)
      • owner is the <user>, which is root in the case of the /root “home” directory (and different from the /home/<user> directory paths)
    • Public keys (usually ending in .pub unless you deliberately rename them):
      • the permissions must be 644, thus the ls -l listing should display as -rw-r--r-- (anything looser will break SSH, and properly so)
      • owner is the <user>, which is root in the case of the /root “home” directory (and different from the /home/<user> directory paths)

Now that you have all of your directories and key files assigned the proper permissions and ownership, let’s get them into their correct directories.

  • Copy or move all keys (private and public) to your /home/<user>/.ssh directory, and /root/.ssh directory if applicable (or confirm they’re there already, which they should be if you ran ssh-keygen correctly as the intended user)
  • Copy (not move) all public keys to the /etc/ssh directory on the server (this may not be necessary if you have the AuthorizedKeysFile configuration set properly in the server sshd_config file, which we’ll work with below, but it won’t hurt anything as they’re public after all)

You now need to get your public keys into each <user>/.ssh/authorized_keys file—on both the client and the server. A handy way to do this is copy them all as appended additions to authorized_keys with this command from within the applicable <user>/.ssh directory: cat *key*.pub >> ./authorized_keys (be sure to use the double >>, unless you intentionally want to blow away any previous keys already in the file and in such case a single > will do that quite thoroughly).

If you don’t have direct access to the server, you can do this remotely with the ssh-copy-id command, by using a login that already works for you on the server. All this command does is copy your public keys into your /home/<user>/.ssh/authorized_keys file on the remote server (similar to the cat *key*.pub command described above). Here is an example of the syntax: ssh-copy-id /home/<user>/.ssh/id_ed25519_key <server address>. Note that it’s the private key that is given as an argument; the ssh-copy-id command only copies public keys for which there is a corresponding private key in the .ssh directory (there’s no point in sending public keys that are unrelated to your private keys, or that the remote server already has in its authorized_keys file).2

Now let’s get to the server configuration, and we’ll do that by editing the /etc/ssh/sshd_config file. (The .conf files are not for the server, so be sure to find the file that spells out “config” and starts with “sshd” (with a “d” for daemon) and has an underscore “_” in it.) Naturally you’ll want to set the Port parameter if you’re using a non-standard port other than 22, which may be advisable because hacking—both ends will need to use the same port (you may set multiple ports for the server to listen on) for a successful connection. Here are some other lines of note in sshd_config:

  • PubkeyAuthentication yes
  • AuthorizedKeysFile %h/.ssh/authorized_keys # checks the user’s authorized_keys file in the relevant .ssh directory
  • PasswordAuthentication yes # you can set this to no after confirming the keys are working
  • ChallengeResponseAuthentication no
  • PermitEmptyPasswords no
  • GSSAPIAuthentication no
  • UsePAM yes # you want the PAM system protecting from auth attacks, and SSH on some systems require it; best to use PAM for all those reasons
  • X11Forwarding yes # if you ever want to login with a GUI, this will allow it
  • Subsystem sftp internal-sftp # later versions of sshd have sftp (SSH File Transfer Protocol) as part of the main binaries, with there no longer being a need to call an outside module—use it if your system has it (if not you may need to use Subsystem sftp /usr/libexec/openssh/sftp-server)
  • (There are multiple other settings you can configure—some informative discussions are available via web search if your system needs those or different configurations, but for a basic SSH system using keys, this should get you started)

Now to restart the sshd service to load your new configs—as root enter systemctl restart sshd. Then check that it loaded and is operating correctly with systemctl status sshd.

You’ve done everything right: permissions, keys, sshd_config. If it won’t connect, a likely suspect is a firewall setting somewhere in the network path. Check the following to start with as root:

  • firewall-cmd --list-ports # if your configured SSH port is not listed, enter:
    • firewall-cmd --add-port=<port-number>/<port-type> # e.g. firewall-cmd --permanent --add-port <port-number>/tcp
    • firewall-cmd --reload # to make your new changes active
  • If you have SELinux active and in Enforcing mode on you system, enter as root:
    • semanage port -a -t ssh_port_t -p tcp <port-number>
    • double check with: semanage port -l | grep ssh to confirm the port is opened by SELinux
  • Check your router (at both ends) for:
    • firewall settings
    • port forwarding

After doing all that, you can reboot the system to force all the new configurations to reload.

You should have been connected with SSH using keys several steps ago. If you still have no connection, you have enough information from what you did above to answer most any reasonable question from someone trying to help diagnose your system.

This should provide you long term trouble free use of your SSH keys. Enjoy!

—

End notes:

1: Naturally you’ll need root access to the server for all server admin tasks, either via direct login or via sudo -i from a user prompt. If it’s a hosted server, you’ll likely not have admin privileges on it, but the platform admins may have already made provision for installing your public keys on your hosted server; consult their knowledge base or help desk for their particular procedures. If you installed the system on the server computer yourself, and you made your user login an admin account, you can check in the /etc/group (singular) file—grep your username and see if your user account is listed as a member of the wheel group (which can do most root level tasks); the line in the file will look something like this: wheel:x:10:<username> (if you can do all of that you’re already there anyway). Return to text

2: Some systems make an authorized_keys directory, and this has caused problems when the SSH ecosystem is looking for basically a text file with crypto text in it. To avoid this file name collision, I’ve simply renamed the directory to something like authorized_keys_dir, and the created the authorized_keys file that SSH really needs. Return to text