id_rsa.pub file SSH Error: invalid format

Question

My co worker and I have been trying to get my macbook to ssh into our AWS production server with no luck, this did work several months ago, but when we went to do it yesterday - we kept getting the following error message.

(I have replaced my username and hostname with a placeholder on all below sections)

$ ssh titan-data
Load key "/Users/aronlilland/.ssh/id_rsa.pub": invalid format
user@my_ip_address: Permission denied (publickey).

Relavant config file data

<!-- config file -->
Host titan-data
    HostName my_ip_address
    User user
    IdentityFile ~/.ssh/id_rsa.pub
    AddKeysToAgent yes

I have gone through every step imaginable to get it working again, these are the following steps I have done in order:

Step 1) ensured that the .pub file was added on the server

this step was repeated several times, I regenerated my .pub file 4 times through out this process of elimination. The error seems to be something local though, not something from the server, someone will need to correct me if I am wrong.

Step 2) checked that the following folders and files have the correct permissions

drwx------    6 aronlilland  staff   192B Jan 18 09:41 .ssh/

drwx------    6 aronlilland  staff   192B Jan 18 09:41 ./
drwxr-xr-x+ 104 aronlilland  staff   3.3K Jan 16 16:09 ../
-rw-r--r--    1 aronlilland  staff   114B Jan 18 09:37 config
-rw-------    1 aronlilland  staff   1.7K Jan 18 09:41 id_rsa
-r--------@   1 aronlilland  staff   414B Jan 18 09:41 id_rsa.pub
-rw-r--r--    1 aronlilland  staff   4.6K Jan 18 09:48 known_hosts

this did not have any effect on fixing the error message.

Step 3) deleted my id_rsa file and id_rsa.pub file and regenerate a new one, having my coworker add the new .pub file to the server

1. first way i generated my keys ssh-keygen -t rsa -b 4096 -C "[e-mail]" as suggested in the github documentation for generating a new SSH key
   • added the new pub file to the server through my co worker, did not work. still getting the same error.
2. Second way I generated my .pub file after deleting the old one, was to run ssh-keygen -t rsa as suggested in the joyent manually generating your ssh key on OSX documentation
   • still got the same error

Step 4) Upgraded ssh with homebrew

this was done following the documentation listed here, by Tim Hilliard

Step 5) deleted the ip address from my "known hosts" file (reaching for straws)

did not do anything, just had to confirm the IP address again.

Edit 1)

One thing I did do recently, and I remember seeing error messages during the installation, was I installed another version of python the other night, so there are two versions of python installed on my computer. Im not sure if this would cause any issues with ssh.

$ python -V
Python 2.7.10
$ python3 -V
Python 3.6.4
$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. 
Thanks!

Warning: Python is installed at /Library/Frameworks/Python.framework

Homebrew only supports building against the System-provided Python or a
brewed Python. In particular, Pythons installed to /Library can interfere
with other software installs.

Edit 2)

added -v to ssh command, suggested by Alfabravo in comments

aronlilland:.ssh$ ssh titan-data -v
OpenSSH_7.6p1, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /Users/aronlilland/.ssh/config
debug1: /Users/aronlilland/.ssh/config line 1: Applying options for titan-data
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to my_ip_address [my_ip_address] port 22.
debug1: Connection established.
debug1: identity file /Users/aronlilland/.ssh/id_rsa.pub type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aronlilland/.ssh/id_rsa.pub-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to my_ip_address:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [e-mail]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:2l8QDc7x/fyzshKaU32jj9MeFtMAccu25MZBFNQuStQ
debug1: Host 'my_ip_address' is known and matches the ECDSA host key.
debug1: Found key in /Users/aronlilland/.ssh/known_hosts:19
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:MuOjm+6gAzaxGr/n/If4LVjk/0H5/VT6zvib9/9C7c8 /Users/aronlilland/.ssh/id_rsa.pub
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
Load key "/Users/aronlilland/.ssh/id_rsa.pub": invalid format
debug1: No more authentication methods to try.
user@my_ip_address: Permission denied (publickey).
aronlilland:.ssh$ 

Edit 3)

Checked to see if the .pub file is a valid format

aronlilland:.ssh$ ssh-keygen -l -f id_rsa.pub
2048 SHA256:MuOjm+6gAzaxGr/n/If4LVjk/0H5/VT6zvib9/9C7c8 [e-mail] (RSA)
aronlilland:.ssh$ touch foo.txt
aronlilland:.ssh$ ls
config       foo.txt      id_rsa       id_rsa.pub   known_hosts
aronlilland:.ssh$ ssh-keygen -l -f foo.txt
foo.txt is not a public key file.

Edit 4)

Confirming that my id_rsa key and id_rsa.pub validate (I chopped off the end of my ssh-rsa key, there's no reason to include it on the web)

aronlilland:.ssh$ ssh-keygen -y -f id_rsa
Enter passphrase: 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQA .......

Confirming permissions that updating file permissions did not change access (I tried 0444, 444, 0644, 644 just to be sure)

aronlilland:.ssh$ chmod 0444 id_rsa.pub
aronlilland:.ssh$ ssh titan-data
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0444 for '/Users/aronlilland/.ssh/id_rsa.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/Users/aronlilland/.ssh/id_rsa.pub": bad permissions
user@my_ip_address: Permission denied (publickey).

and the other

aronlilland:.ssh$ chmod 0644 id_rsa.pub
aronlilland:.ssh$ ssh titan-data
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/Users/aronlilland/.ssh/id_rsa.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/Users/aronlilland/.ssh/id_rsa.pub": bad permissions
user@my_ip_address: Permission denied (publickey).

setting it back to chmod 400 id_rsa.pub

Edit 5)

running -vv

aronlilland:.ssh$ chmod 400 id_rsa.pub
aronlilland:.ssh$ ssh titan-data -vv
OpenSSH_7.6p1, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /Users/aronlilland/.ssh/config
debug1: /Users/aronlilland/.ssh/config line 1: Applying options for titan-data
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug2: resolving "my_ip_address" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to my_ip_address [my_ip_address] port 22.
debug1: Connection established.
debug1: identity file /Users/aronlilland/.ssh/id_rsa.pub type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aronlilland/.ssh/id_rsa.pub-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to my_ip_address:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[e-mail],zlib
debug2: compression stoc: none,[e-mail],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[e-mail]
debug2: compression stoc: none,[e-mail]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [e-mail]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:2l8QDc7x/fyzshKaU32jj9MeFtMAccu25MZBFNQuStQ
debug1: Host 'my_ip_address' is known and matches the ECDSA host key.
debug1: Found key in /Users/aronlilland/.ssh/known_hosts:19
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /Users/aronlilland/.ssh/id_rsa.pub (0x7fe9ff41c510), explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:MuOjm+6gAzaxGr/n/If4LVjk/0H5/VT6zvib9/9C7c8 /Users/aronlilland/.ssh/id_rsa.pub
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:MuOjm+6gAzaxGr/n/If4LVjk/0H5/VT6zvib9/9C7c8
Load key "/Users/aronlilland/.ssh/id_rsa.pub": invalid format
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
user@my_ip_address: Permission denied (publickey).

Answer 1

In my case, I was attempting to use the -i flag to point to the default key: ssh -i ~/.ssh/id_rsa.pub user@0.0.0.0

Removing the -i key, and simply calling ssh user@0.0.0.0 resolved the error

Answer 2

The IdentityFile configuration parameter should be pointed at the private key which the SSH client uses to prove its identity to the remote server. (The remote server, then, should have the contents of id_rsa.pub installed in its authorized_keys file, or an equivalent location).

You should be putting the path to id_rsa, not id_rsa.pub, as an argument to IdentityFile in your ~/.ssh/config.

Answer 3

If anyone, getting this error in Jenkins pipeline while using ssh-agent plugin; please try to put a new line after end of the key.

Source: https://wiki.jenkins.io/display/JENKINS/SSH+Agent+Plugin

Answer 4

your error is in having used the -f option with the public key name... you have to generate a new key, as you have sent (and compromised) your private key, by storing it in -f .ssh/id_rsa.pub

The private key is stored by default in .ssh/id_rsa and the public key is stored in .ssh/id_rsa.pub. By specifying .ssh/id_rsa.pub as the place of your private key, probably you'll have the public in .ssh/id_rsa.pub.pub (CHECK THIS) but anyway, you have sent your private key over the network, so it is compromised and you should have better generating a new key pair.

Next time, you can trust the default settings proposed for the key filenames or use the proper (without the .pub extension) for the key file.