GitLab SSH keys stopped working

Question

OK, I've stumbled on this topic many times, but this is the first time that none of the regular solutions worked.

I have a CentOS 6.4 server running GitLab. It's been working great with 20+ users and 60+ projects, but about 5 hours ago, my main staging server was unable to connect to the GitLab machine for the first time using key authentication and it prompted for password. I regenerated the RSA key and added it to my deploy keys, but that failed as well.

Next, I tried to create a new user on the staging server, create a key for it, and add it to GitLab but it still fails.

Permissions:

drwxr-x---  22 root root 4.0K Oct 28 13:20 root

Inside root:

drwx------   2 root root     4096 Oct 28 11:49 .ssh

Inside .ssh:

-rw-------  1 root root  227 Oct 28 11:48 authorized_keys
-rw-------  1 root root 1675 Oct 28 13:09 id_rsa
-rw-------  1 root root  398 Oct 28 13:09 id_rsa.pub
-rw-r--r--  1 root root  413 Oct 28 11:49 known_hosts

When I try to connect to the git machine:

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to git.mygitlab.com [212.29.122.24] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'git.mygitlab.com' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-    mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-    mic,password
debug1: Trying private key: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password
[e-mail]'s password:

When I add SSH keys through the web interface, they aren't added to .ssh/authorized_keys.

I don't really know what to try next :(

Answer 1

I just ran into this problem when I switched the GitLab server from HTTP to HTTPS. Everything looked fine on the web server - logins, etc. were all working normally but git@gitlab SSH connections were failing.

After looking at #2 in https://stackoverflow.com/a/19637026/2162639 (above) I discovered that I needed to modify the gitlab_url setting in /home/git/gitlab-shell/config.yaml to use https://gitlab.server.fqdn instead of http://gitlab.server.fqdn. I changed that setting, restarted the gitlab service and everything was working normally.

Answer 2

Had to delete all previous keys for the host. Problem is gitlab takes any older keys and if the match does not exist, it fails there. Your working key may be listed later in order and never picked.

Answer 3

If keys that you add to GitLab aren't making it into .ssh/authorized_keys:

1. Make sure that sidekiq is running. Keys are added to gitlab-shell in a Sidekiq worker, so if Sidekiq is down or backlogged, they won't make it in. You can check this in the output of ps -fu git and by checking the "background jobs" tab on the admin page.
2. Ensure that GitLab can execute gitlab-shell properly. The Sidekiq worker adds keys by exec'ing a gitlab-shell process. In particular, this won't work if the ssh_user setting is incorrect in gitlab.yml, or if gitlab-shell is installed to a location other than ~/gitlab-shell for that user.
3. Verify that the server's /home partition isn't full. If the disk that the authorized_keys file is stored on fills up, key appends with fail! This one has gotten me a few times. Use df -h /home to see if you still have room.

Check your logs for error messages from gitlab-shell: depending on the problem, error messages could appear in unicorn's or sidekiq's logs.

Answer 4

Well, now i am under 5.1 i did it step by step 4.1 > 4.2 4.2 > 4.3 and finally everything is up and running.

Just for 4.1 users to know - > one of the developers added a bad key including the $#root... and this is what broke the sync.

Thanks for you help