I'm migrating files from one remote server to S3. There are about 10k files (all accessible via http URLs from the remote server). The total size is about 300GB (no individual file is more than 1GB). I'm trying to figure out the most efficient way to make this migration. So far I have a EC2 instance and I have the S3CMD installed; PHP-SDK, I have a text file with all the URL's as well. I'm able to move files from EC2 to S3 without any issue. but the problem is if I download everything in EC2 I run out of storage. Is there a way where I can download a file in EC2 (maybe look in the txt file) move it to S3 (using S3CMD) and then delete the file from EC2 before I go to the next file.
Ideally I would want to download everything straight to S3 from the remote location, but I don't think that is possible, unless someone here says it is.
Thanks in advance for the help.
I don't see what OS your current ec2 instance is running. But if it is linux you could use S3fs
https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
that will allow you to mount your bucket like a local drive/folder. Then you can simple move the files there. It will upload them to the bucket in the background. I would moving them in some kind of batches to make it easy to track. Moving them would remove them from your local file system after uploading. You could also just copy them to the bucket this way. When done you could do a simple comparison to make sure the same files exist in both folders and then you are done.
EDIT based on question asked in comment for clarity
On the remote machine, setup Fuse with your AWS credentials.
Mount your S3 bucket. It will look like a local folder structure in Ubuntu.
Lets say your current files are in
/var/myfiles/folder1 and /var/myfiles/folder2
mount your S3 bucket to /mybucket
mv /var/myfiles/folder1 /mybucket/folder1
Again, I would move them in batches and make sure the folders match up before continuing.
END EDIT
If you EC2 instance is windows then there are other ways to mount an S3 bucket as a local drive. Then the same process could take place.
