Add Vault
Better secret/credential management with Vault
Once we start using Vault, we'll be implementing SSH Certificates. So atm, this blocks #19.
Using SSH Certificates with GitLab.
Vault also does authorization with OIDC (at least in some capacity as far what I glossed over)
General spitballing idea im writing down.
The OpenSSH sshd_config file specifies in the HostKey section (maybe debian only) the following:
It is possible to have multiple host key files. It is also possible to specify public host key files instead. In this case operations on the private key will be delegated to an ssh-agent(1).
Would it be a good idea to have the private SSH keys in vault, and load them into the ssh-agent? This requires the server to provide some type of authentication to Vault that it can retrieve those keys, but we can rotate that type of authentication.
Would having the private sshkeys as a file on the server or being in the agent be safer. Can the contents of the ssh-agent be just as easily read as the ssh_host_key file?
This may be a viable option (obviously more research) but according to this answer, it's not possible to read private keys from the ssh-agent without a memory dump.
One would argue that its more common to have a private key accidentally stored and leaked vs someone doing a memory dump? If you can do a memory dump, you can access the key anyway. Doing a memory dump to retrieve it becomes an actively sought out action by a currently active malicious user where the former can be accidental and be done at any time. In either case, one is a file that is passed around, the other absolutely required to have root access to the current system.
Hmm, after more research and thought, I don't think I've ever seen any demonstration about adding an ssh servers ssh_host_keys to an ssh-agent. It appears the ssh-agent adds usability and sacrificing a bit of security as the agent holds onto it unencrypted (say a passphrase was used). Because it would require work and seems to be just as easy to retrieve given a few whitehat tools, its probably not worth investigating and to rather implement better methods of diligently securing and rotating host_keys.
Also the memory dump issue was noted as one of the flaws of Vault as well to get the encryption key, however Vault indicates the decryption key is never stored and separated into shards that can also be individually encrypted with users GPG keys.
Eventually we'll be implementing SSH Certificates with in Vault in one capacity or another, namely SSH certificate/signing. But for the immediate future I believe we'll encrypt our keys and back them up before transfer.
Whether we set this or setup Ansible first will determine how we initially setup Vault.