Start using SSH certificates
Its late and copy pasting notes ive written down on my search for changing hosts without re-using ssh_host_keys and avoiding the MITM/DNS SPOOF warning for 'potential clients'.
Its poorly formatted as they were notes in a .sh file but its better to keep track here. Will format in the coming days.
We followed our steps to test and indeed hit the ## If we do get a MITM prompt, \_(-_-)_/ step.
Our journey brought us to the (hopeful) solution of SSH certificates, as seen in the except below taken from the link.
Look into SSH certificates
https://smallstep.com/blog/use-ssh-certificates/
Little excerpt
This makes it operationally challenging to reuse host names. If prod01.example.com has a hardware failure, and it’s replaced with a new host using the same name, host key verification failures will ensue. This usually results in a bunch engineers contacting secops to tell them they’re being hacked.
Ignoring host key verification failures has the exact same attack surface area as not knowing the key at all. Curiously, OpenSSH chooses to soft-fail with an easily bypassed prompt when the key isn’t known (TOFU), but hard-fails with a much scarier and harder to bypass error when there’s a mismatch.
In any case, certificates fix all of this since a current name-to-public-key binding is communicated when a connection is established. Changing the host’s public key is fine, as long as the host also gets a new certificate. You can safely reuse host names and even run multiple hosts with the same name. You’ll never see a host key verification failure again. Beyond name reuse, we’ll soon see that eliminating host key verification failures is one of the many ways certificate authentication facilitates good security hygiene.
General links SSH cert links
https://unix.stackexchange.com/a/183947
https://goteleport.com/blog/how-to-ssh-properly/
Rotating ssh host key (in attempt to support certs and host keys)
https://lwn.net/Articles/637156/
https://unix.stackexchange.com/questions/334597/how-to-roll-over-ssh-host-keys?rq=1
Generate new ssh_host keys
#ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
#ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
#ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa -q
#ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa -b 521
# or
# ssh-keygen -Ahttps://unix.stackexchange.com/questions/334597/how-to-roll-over-ssh-host-keys?rq=1
The Host Key rotation is supported since OpenSSH 6.8 (both client and server adds support in this version).
So the process should work like this:
Generate and add new keys with the option HostKey newkey (after the existing ones) to the /etc/ssh/sshd_config
Restart sshd
The clients have to set up UpdateHostKeys yes in their configuration (either globally, or per-host)
The connecting clients will pick up all the new keys
After some time (months?) you can remove the old keys from the sshd_config and restart sshd
The clients (that connected during the transition period) will already have the new keys (the old will not be removed,
which is the only problem here) and they will not show the MitM attack warning.The new enough-clients will be able to pick up the new keys. This feature is not enabled by default, probably because it is quite new and soon showed some security consideration. But these days, it should be fine to use it.
SSH TODO: Want to switch away from backing up and importing ssh keys and use fresh keys for every system migration, at the very least to practice rotating keys in case of a system/backup compromise. This ensures the ssh keys are only ever on the server and minimizes attack surface (access to where the ssh keys are backed up to creates an attack vector)
The goal is in the short term after every system migration, require every ssh connection to the server or for git clone be prompted with accepting the systems new fingerprint and NOT be prompted with the "WARNING POSSIBLE MITM" prompt. The prompt was the original reason we started backing up the keys but there has to be a better method in the case of standard system migrations or keys DO actually get compromised, caught, and rotated before a user reconnects to the service. After a system migration and change of keys a user should absolutely NOT get a huge "WARNING".
Steps to test: Wipe local .ssh/known_hosts Backup gitlab instance Import gitlab to test environment/domain Connect/Git clone accepting the default prompt Destroy and re-import to test env/domain to new machine/ip without importing old keys Attempt to connect/git clone hoping for a generic prompt and not a MITM prompt
According to "https://pikedom.com/migrate-gitlab-instance-to-new-host/" we should only be given that prompt if we use the SAME IP, say a floating/elastic IP. In which case backing up keys seems to the only viable solution.
If we do get a MITM prompt, _(--)/
Also, if we simply change the keys on the remote server I assume that falls under the same case as rebuilding the machine under the same IP and would prompt a MITM since the keys for that IP have now changed. Suppose this means the ONLY way to rotate ssh keys would be a system migration and solidifies the reason to NOT backup keys and use new keys for each migration (at least until we're satisfied with it working)