AWS - Locked out of EC2 after rsync'ing /home/ubuntu directory

TL;DR
I rsynced a few directories from /home/ubuntu (set to 777) of one remote server to my new AWS EC2 instance. I am now locked out of sshing into it by a Permission denied (publickey) error.

I am in the process of migrating my production environment from SoftLayer to AWS.

I had to rsync several directories to EC2 (EBS) and in the process, I did transfer a few directories from the old /home/ubuntu/ to my current EC2 instance's /home/ubuntu/.

My rsync command (on destination) looked like this.

ubuntu@[aws.remote.ec2]:~$ sudo rsync --include 'dir1' --include '*.sh' --include '.py' --include 'api_logs' --include 'database_backups' --exclude '*' -avz -e "ssh -p $portNumber" ubuntu@[softlayer.remote]:/home/ubuntu/ /home/ubuntu/ 

Files were successfully transferred. When I tried to ssh into my EC2 the next time I got a Permission denied (publickey) with the following log with the ssh -v option: (I have masked private info like IP with {} in the log below)

OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016 debug1: Reading configuration data /home/{localuser}/.ssh/config debug1: /home/{localuser}/.ssh/config line 1: Applying options for aws-fr- ec2 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to {aws.ec2.ip} [{aws.ec2.ip}] port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file /home/{localuser}/Documents/AWS-Files/EC2-FR.pem type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/{localuser}/Documents/AWS-Files/EC2-FR.pem-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000 debug1: Authenticating to {aws.ec2.ip}:22 as 'ubuntu' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: MAC: <implicit> compression: none debug1: kex: client->server cipher: MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:g3nWVGmjJYVrNrwsDJMhzbLSw0FzBOLoUx80seD9qIs debug1: Host '{aws.ec2.ip}' is known and matches the ECDSA host key. debug1: Found key in /home/{localhost}/.ssh/known_hosts:11 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/{localhost}/Documents/AWS-Files/EC2- FR.pem debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. Permission denied (publickey). 

I did stumble upon this question but without help. I also found this thread on the AWS Forum.

I did follow the steps listed:

Posted by: mary@AWS:
Could you please verify the permissions on the /home/ubuntu/.ssh directory and files contained in it on this instance?

To verify the permissions, you can stop the instance and detach the root volume (make note of the device that it's attached to). Then attach the volume to another instance on an available device. Create a mount point, such as /fixroot, if needed and mount the device to this mount point. Once mounted, cd to /fixroot/home/ec2-user and check the directory and file permissions. The .ssh directory should allow rwx for the user (owner) and the files should be readable only by the user.

Another thing to check while you are there is that the known_hosts file doesn't have duplicate entries for the client you are attempting to connect from.

Once you have done this, you can unmount the volume and detach it from the instance. Then attach it back to the original instance to the device that you noted in the first step and start the instance.

As well as

Posted by: yromanenko:
turns out it was the relaxed permissions on the home/ubuntu folder rather than ssh. I was able to fix it by detaching the root volume and fixing permissions. The following video was very helpful in guiding me through the steps:

I spawned a new t2.micro instance and followed Mary's steps to confirm the permissions and yromanenko's resolution to set 755 on the /home/ubuntu directory.

I reattached the problematic EBS device back to the first EC2 as /dev/sda1 and tried only to fail with the same Permission denied (publickey) error!!

Consequently, now I am getting the same error on the secondary t2.micro instance. :(

Any help would be appreciated!

1 Answer

I had this exact issue. I was using an Amazon EC2 instance so I could spin up another and identify the differences. The broken machine had /home/user permissions 777, the not broken machine had 700. For some inexplicable reason, rsyncing to /home/user changes the permissions of /home/user to 777. Apparently SSH requires /home/user to be 700 for security reasons. This also explains why its turtles all the way down, rsync again and it's broken again.

If you are fortunate enough to have access to the directory in some other way,

chmod 700 /home/user 

fixes it.

In the future, rsync to a subdirectory of /home/user.