问题:

I want to run a multi-node hadoop cluster, with each node inside a docker container on a different host. This image - https://github.com/sequenceiq/hadoop-docker works well to start hadoop in a pseudo distributed mode, what is the easiest way to modify this to have each node in a different container on a separate ec2 host?

回答1:

I did this with two containers running master and slave nodes on two different ubuntu hosts. I did the networking between containers using weave. I have added the images of the containers on docker hub account div4. I installed hadoop in the same way, as its installed on different hosts. I have added the two images with coomands to run haddop on them here:

https://registry.hub.docker.com/u/div4/hadoop_master/ https://registry.hub.docker.com/u/div4/hadoop_slave/.

回答2:

The people from sequenceiq have created a new project called cloud-break that is designed to work with different cloud providers and create hadoop clusters on them easily. You just have to enter your credentials and then it works the same for all providers, as far as I can see.

So for ec2, this will now probably be the easiest solution(especially because of a nice GUI):

https://github.com/sequenceiq/cloudbreak-deployer