UPDATE - It happened AGAIN!!!
Ok, so this just happened AGAIN! MAN is this frustrating!!! But this time I dug a little deeper and found that for some reason, my private keys were unloaded.
Specifically, when I call this...
ssh-add -l -E md5
I get this...
The agent has no identities.
However, if I then run this...
ssh-add /Users/[username]/.ssh/[private key]
Everything works again! SourceTree connects just as it's supposed to.
The question is why do I have to keep running the 'ssh-add' command?! Why does it keep forgetting my keys?!
As mentioned elsewhere, not sure if this makes a difference, but I'm running a MacBook Pro with High Sierra, although this happens on Sierra too.
Original Post:
This one has me both stumped, and annoyed as heck!! SourceTree (or ssh or something!) keeps forgetting/not applying/ignoring my SSH keys every day! I don't know why.
Note: Updated to use BitBucket's info instead of GitHub.
Here's the relevant portion of my current config file
# --- Sourcetree Generated ---
Host MarqueIV-Bitbucket
HostName bitbucket.org
User MarqueIV
PreferredAuthentications publickey
IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
UseKeychain yes
AddKeysToAgent yes
# ----------------------------
Here's a 'ls' of my ~/.ssh folder (truncated)
-rw-r--r--@ 1 MarqueIV staff 421 Dec 14 11:25 config
-rw-r--r--@ 1 MarqueIV staff 1808 Dec 9 14:20 known_hosts
-rw------- 1 MarqueIV staff 3243 Dec 6 23:33 MarqueIV-Bitbucket
-rw-r--r-- 1 MarqueIV staff 781 Dec 6 23:33 MarqueIV-Bitbucket.pub
Here's my known_hosts file (keys redacted)
bitbucket.org,104.192.143.3 ssh-rsa [redacted]
bitbucket.com,104.192.143.9 ssh-rsa [redacted]
104.192.143.2 ssh-rsa [redacted]
Note: Not sure if this matters, but you can see lines 1 and 2 seem to be duplicates.
And here's the output of ssh -Tv git@bitbucket.org
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/MarqueIV/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to bitbucket.org port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version conker_1.0.315-a08d059 app-153
debug1: no match: conker_1.0.315-a08d059 app-153
debug1: Authenticating to bitbucket.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /Users/MarqueIV/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:h+6zCXg32Uw4fYxSUMwYst3zee8RFb9Z47H1QUTz58E /Users/MarqueIV/.ssh/MarqueIV-GitHub
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/MarqueIV/.ssh/id_rsa
debug1: Trying private key: /Users/MarqueIV/.ssh/id_dsa
debug1: Trying private key: /Users/MarqueIV/.ssh/id_ecdsa
debug1: Trying private key: /Users/MarqueIV/.ssh/id_ed25519
debug1: No more authentication methods to try.
git@bitbucket.org: Permission denied (publickey).
See how it doesn't appear to be applying the key defined in config and known_hosts? Seems like that would be a problem, no?
Note: I'm using macOS Sierra, and I have updated my path to include /usr/bin before /usr/local/bin as outlined here. If I don't do that, I get an error saying ssh doesn't recognize UseKeychain yes in the config.
As a result, almost daily here's the routine I have to go through. I'll use GitHub as my example.
I open SourceTree and try to pull the latest from GitHub. It fails with a 'git@github.com: Permission denied (publickey).' message.
I remove my GitHub account from SourceTree.
I delete both the public and private keys for GitHub from the .ssh folder on my machine.
I go to GitHub and delete my old public key from my account.
Back in SourceTree, I log into GitHub again using my username and password.
Once logged in, using SourceTree, I generate a new SSH key-pair for GitHub.
I copy my public key to the SSH area in my GitHub account. (Sometimes I notice it adds it for me, but I like to be safe and double-check.)
Now I can push and pull again just fine.
I go home for the day and log on at home. It fails again. Repeat all of the steps above.
How do I get SourceTree/ssh/whatever to remember my da*n keys so I don't have to keep doing this every time I change locations?! What step am I missing???
So can anyone offer suggestions on how to make my SSH keys 'stick'?
Ok, I think I have all the parts figured out.
To help people get what they're after, here's the solution right up front:
- Make sure the keys you want to auto-load are configured in your
config file and have the UseKeychain and AddKeysToAgent set
- Make sure to connect to those config-defined hosts from terminal!!
- Create a LaunchAgent to run
ssh-add -A to automatically reload your Keychain-stored keys
Ok now that you know what to do, here's the 'why'.
The Meat
As explained in my question, lately, whenever I rebooted, I (incorrectly) thought the system was losing my private keys. It wasn't losing them, it was just ignoring them. This was because of a bunch of things that all came together in a perfect storm of confusion for someone like me who never uses the terminal for GIT.
- In the latest versions of macOS, Apple changed how it's implemented SSH so that It better matches the implementation of OpenSSH
- As a result of #1,
ssh-add -K [privateKey] no longer stores the keys in the keychain (it essentially ignores the -K.) While they do get added to ssh for that session--and thus your connections will work again--as soon as you reboot, they will no longer work. (This is what's been driving me mad!)
- Even for keys that are in the Keychain, Apple no longer loads them automatically meaning you manually have to call
ssh-add -A from the terminal to reload them every time you reboot.
- However, as stated above,
ssh-add -K [privateKey] no longer adds the keys to keychain, so ssh-add -A is pointless anyway for keys added that way. (They can be added to Keychain another way. More on that in a minute.)
Because of the above, any keys manually added with the -K option prior to upgrading your OS will still be in your Keychain. However, keys added after Apple's change are not.
That said, Apple does still have the ability to store keys in the keychain, but not from ssh-add anymore. It now only works for hosts defined in your config file.
This is now the only way to get your keys in your Keychain.
Again, here's my config:
Host MarqueIV-Bitbucket
HostName bitbucket.org
User git <-- Make sure this is 'git', not what SourceTree puts here
PreferredAuthentications publickey
IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
UseKeychain yes <-- Note here
AddKeysToAgent yes <-- ...and here
But wait! If you look in my config file, it does have those values set! So why didn't it work?
Two things.
- I don't use Terminal, ever. I use SourceTree which doesn't use the host entry in that file
- Apple technically only adds (and stores) the key on demand when that host is accessed, not when the file is (re)loaded meaning unless you explicitly access that host, nothing happens.
In my case, adding the keys via SourceTree would add them for that initial session, but as soon as I rebooted, the keys would again not be loaded and thus all connections would fail. ssh-add -A wouldn't fix it either because again, they weren't in the keychain, meaning I was back to manually adding each one on the command line with ssh-add [privateKey]. What a pain!!
Then it occurred to me... if that setting is in the config file, and that entry can be used from the command line, then shouldn't I be able to directly connect to that host, thus adding the keys to my keychain? Let's find out! I typed this...
ssh -T MarqueIV-BitBucket
And sure enough, not only was the key added to ssh, but it was also again added to my Keychain! I confirmed this by checking Keychain Access directly and it was there.
To further test, I ran this...
ssh-add -D
which deleted all my keys. Sure enough, my SourceTree connections all failed again.
Then I ran this...
ssh-add -A
and the keychain-stored keys magically came back and connections started working again! WOOT!!
Ok, almost there, but not quite! What about reboots? Again, Apple no longer automatically loads keys from Keychain. Sure, it's just a quick jaunt now to terminal to type ssh-add -A, but again, I shouldn't have to do that!
Enter LaunchAgents!
LaunchAgents and LaunchDaemons are beyond the discussion of this post, but in short, they allow you to execute something on reboot, on a schedule, when changes happen to the system, etc.
In my case, I wanted something that would run when I logged onto my mac, so a LaunchAgent was the best choice.
Here's my plist defining how to execute ssh-add -A every time I logged into my account (even if I never touched Terminal):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>ssh-add-a</string>
<key>ProgramArguments</key>
<array>
<string>ssh-add</string>
<string>-A</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
Since I only want this for my particular user, I stored it here:
~/Library/LaunchAgents
Note: Make sure to change the permissions to allow it to be executed, or it won't start!
Sure enough, on reboot, all my keys came back and were active! Connections all worked, children played, grown men cried, and it was a good day in the Code-dom!
So to recap:
- Apple changed how their SSH worked
- Keys were no longer added to Keychain from the command line
- Apple also no longer auto-loaded keys that were stored in the keychain
- Using terminal to connect to config-defined hosts fixed #2
- Using a LaunchAgent fixed #3
Hope this helps! Now time to go get some Icy-Hot for my sore shoulder that I've been patting myself on so hard for figuring this all out! ;)
That ~/.ssh/config excerpt is only applicable for the host MarqueIV-Bitbucket. If your SSH remotes are listed as MarqueIV-Bitbucket:owner/repo then SSH and SourceTree should respect that config; you can confirm this with ssh -Tv MarqueIV-Bitbucket and by updating one or more of the remotes to the MarqueIV-Bitbucket:owner/repo.git format.
First, install the latest Git for Windows release (the 2.15.1.2 one, by simply uncompressing the archive PortableGit-2.15.1.2-64-bit.7z.exe anywhere you want, and adding it to your PATH)
Second, make sure your SourceTree is using the System Git
Third, test in command-line if your ssh key is recognized:
ssh -T git@github.com
Hi username! You've successfully authenticated,
but GitHub does not provide shell access.
Finally, make sure that SourceTree / Tools / Option uses as SSH client the OpenSSH one (not putty)
Then SourceTree should have nop problem reusing your ssh key.
From your logs, the ~/.ssh/config generate is wrong: it mentions as User your username.
Whenever you establish an SSH connection to github.com/bitbucket.org, it is never as "you". It is always as git.
Host MarqueIV-Bitbucket
HostName bitbucket.org
User MarqueIV
PreferredAuthentications publickey
IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
UseKeychain yes
AddKeysToAgent yes
Test it with ssh -Tv MarqueIV-Bitbucket
