Elasticsearch in docker container cluster

Question

I want to run 2 instance of ElasticSeach on 2 differents hosts. I have build my own docker image based on Ubuntu 14.04 and the 1.3.2 version of elasticsearch. If I run 2 ES' container on 1 host each instance see the other one and can communicate, but when I run 2 instance of ES on 2 differents host it's didn't work. The 9300 port of the container is bind to the 9300 host's port.

My question is : it's possible to create an ES' cluster with my configuration ?

Best regards, Lucas Rival

Answer 1

Using docker-compose is much easier than running it manually in command line:

elasticsearch_master:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.node.master=true -Des.node.data=false"
    environment:
       - ES_HEAP_SIZE=512m
    ports:
      - "9200:9200"
      - "9300:9300"

elasticsearch1:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
    links:
      - elasticsearch_master
    volumes:
      - "/opt/elasticsearch/data"
    environment:
       - ES_HEAP_SIZE=512m
elasticsearch2:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
    links:
      - elasticsearch_master
    volumes:
      - "/opt/elasticsearch/data"
    environment:
       - ES_HEAP_SIZE=512m

Answer 2

I was able to get clustering working using unicast across two docker hosts. I just happen to be using the ehazlett/elasticsearch image, but I do not think this should matter all that much. The really important bit seems to be setting the network.publish_host setting to a public or routable IP its docker host.

Configuration

---

docker-host-01

eth0: 192.168.1.10
Docker version 1.4.1, build 5bc2ff8/1.4.1

docker-host-02

eth0: 192.168.1.20
Docker version 1.4.1, build 5bc2ff8/1.4.1

Building the Cluster

---

On Docker Host 01

docker run -d \
  -p 9200:9200 \
  -p 9300:9300 \
  ehazlett/elasticsearch \
  --cluster.name=unicast \
  --network.publish_host=192.168.1.10 \
  --discovery.zen.ping.multicast.enabled=false \
  --discovery.zen.ping.unicast.hosts=192.168.1.20 \
  --discovery.zen.ping.timeout=3s \
  --discovery.zen.minimum_master_nodes=1

On Docker Host 02

docker run -d \
  -p 9200:9200 \
  -p 9300:9300 \
  ehazlett/elasticsearch \
  --cluster.name=unicast \
  --network.publish_host=192.168.1.20 \
  --discovery.zen.ping.multicast.enabled=false \
  --discovery.zen.ping.unicast.hosts=192.168.1.10 \
  --discovery.zen.ping.timeout=3s \
  --discovery.zen.minimum_master_nodes=1

Answer 3

You should be able to communicate the two containers running in different hosts as far as the host machines are accessible between them in the ports needed. I think your problem is that you are trying to use ElasticSearch multicast discovery, but if then you need to expose also port 54328 of the containers. If it doesn't work you can also try to configure ElasticSearch using unicast, setting adequately the machines IP's in your elasticsearch.yml.