GitLab cloning via SSH always prompts for password

Question

I've already followed the guide in setting up ssh key pairs on my own local machine (where i plan to clone my source from GitLab). However, every time i executing git clone [e-mail] i am always prompted to input my password.

I expect to input nothing and automatically clone my source. Since i set the ssh-keygen command to have NO passphrase.

I have read in the GitLab guide some workarounds and debugging methods. Nothing works so far. I would like to ask any of you guys to take a look at it. See sample -v logs on ssh -Tv [e-mail]

[root@localhost ~]# ssh -Tvvv [e-mail]
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "mygitlab-ce.locadomain.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to mygitlab-ce.locadomain.com [192.168.0.135] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to mygitlab-ce.locadomain.com:22 as 'git'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from mygitlab-ce.locadomain.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: [e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: [e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail],[e-mail],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail],aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail],aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[e-mail],zlib
debug2: compression stoc: none,[e-mail],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail],aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail],aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[e-mail]
debug2: compression stoc: none,[e-mail]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zCtfepU7IAyTrf2GzZjN/MXrHjSC1mFmVhBQqmgQ8B8
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from mygitlab-ce.locadomain.com
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 192.168.0.135
debug1: Host 'mygitlab-ce.locadomain.com' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /root/.ssh/id_rsa (0x55f8a93bc520), agent
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[e-mail]'s password:

I don't understand this part

debug1: Next authentication method: publickey

debug1: Offering RSA public key: /root/.ssh/id_rsa

debug1: Authentications that can> continue: publickey,gssapi-keyex,gssapi-with-mic,password

[root@localhost ~]# ls -alh ~/.ssh/
total 16K
drwx------.  2 root root   57 Oct  2 04:26 .
dr-xr-x---. 23 root root 4.0K Oct  2 04:26 ..
-rw-------.  1 root root 3.4K Oct  2 04:17 id_rsa
-rw-r--r--.  1 root root  750 Oct  2 04:17 id_rsa.pub

Why can't it detect my private key?.

Answer 1

As the server is running CentOS with SELinux, policys might have to be adjusted.

First test if SELinux might be the issue by running the following on the server

sudo setenforce 0

If you are able to use public/private key pairs with setenforce 0 re-enable SELinux with

sudo setenforce 1

Solution if SELinux stops cloning by using public/private key pairs

Provided that /var/opt/gitlab/ is your gitlab installation git user's home folder run the following

sudo semanage fcontext -a -t  ssh_home_t /data/gitlab/.ssh/
sudo semanage fcontext -a -t  ssh_home_t /data/gitlab/.ssh/authorized_keys
sudo restorecon -Rv /data/gitlab/

The file /var/opt/gitlab/gitlab-shell/config.yml will also be access and thus needs to have its context edited

sudo semanage fcontext -a -t ssh_home_t /data/gitlab/gitlab-shell/config.yml
sudo restorecon -Rv /data/gitlab/gitlab-shell/

as well as /data/gitlab/gitlab-rails/etc/gitlab_shell_secret

sudo semanage fcontext -a -t ssh_home_t /data/gitlab/gitlab-rails/etc/gitlab_shell_secret
sudo restorecon -Rv /data/gitlab/gitlab-rails

Confirm that the context changes have been applied by running

sudo semanage fcontext --list

You should see

/data/gitlab/.ssh/authorized_keys                  all files          system_u:object_r:ssh_home_t:s0
/data/gitlab/.ssh/                                 all files          system_u:object_r:ssh_home_t:s0
/data/gitlab/gitlab-shell/config.yml               all files          system_u:object_r:ssh_home_t:s0
/data/gitlab/gitlab-rails/etc/gitlab_shell_secret  all files          system_u:object_r:ssh_home_t:s0

If that is the case try to clone from your repository. For details refer to lemoncodes support thread on gitlab.com

Answer 2

After some digging and some inputs from @Mr.Christer. i have fixed the issue. The fix is somewhat lengthy as there was a issue ticket issue on GitLab itself. So i might as well link the info to the GitLab issue since there was several comments in there that is helpful in fixing the problem.

Cloning via SSH always prompts password when SSH Key passphrase is set to None

Basically, what im trying to say is that do not execute semanage fcontext on a symlink but do it on the actual physical location.

-------------------- OLD -----------------------

after a collab with @Mr.Christer, we've found out that SELinux was blocking ssh public-key authentication. When executing setenforce 0 on GitLab server, this will allow git cloning over ssh successful.

I'am still searching for guides as to how to configure GitLab with SELinux. But just to give you answer, configure SELinux for GitLab and cloning via ssh will work.