Kafka Brokers for Hyperledger Fabric could not est

Question

I need some assistance with configuring my kafka brokers docker containers for hyperledger fabric. My setup will be a cluster of 4 brokers and I am having an issue when I introduce the second broker. All the brokers are currently sitting on a single machine and I need them to advertise their external address so that my orderers on different machines can discover them.

My compose file with 1 Zookeeper and 1 Broker:

version: '2'

services:
  zookeeper0.hyperfabric.xyz:
    image: hyperledger/fabric-zookeeper
    restart: always
    container_name: zookeeper0.hyperfabric.xyz
    environment:
      - ZOO_SERVERS=server.1=zookeeper0.hyperfabric.xyz:2888:3888
      - ZOO_MY_ID=1
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888

  kafka0.hyperfabric.xyz:
    image: hyperledger/fabric-kafka
    restart: always
    container_name: kafka0.hyperfabric.xyz
    environment:
      - KAFKA_MESSAGE_MAX_BYTES=103809024
      - KAFKA_REPLICA_FETCH_MAX_BYTES=103809024
      - KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
      - KAFKA_MIN_INSYNC_REPLICAS=1
      - KAFKA_DEFAULT_REPLICATION_FACTOR=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper0.hyperfabric.xyz:2181
      - KAFKA_BROKER_ID=0
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka0.hyperfabric.xyz:9092
    ports:
      - 9092:9092
      - 9093:9093

Running this works fine.

When I introduce another broker using:

kafka1.hyperfabric.xyz:
  image: hyperledger/fabric-kafka
  restart: always
  container_name: kafka1.hyperfabric.xyz
  environment:
    - KAFKA_MESSAGE_MAX_BYTES=103809024
    - KAFKA_REPLICA_FETCH_MAX_BYTES=103809024
    - KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
    - KAFKA_MIN_INSYNC_REPLICAS=1
    - KAFKA_DEFAULT_REPLICATION_FACTOR=1
    - KAFKA_ZOOKEEPER_CONNECT=zookeeper0.hyperfabric.xyz:2181
    - KAFKA_BROKER_ID=1
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka1.hyperfabric.xyz:10092
  ports:
    - 10092:9092
    - 10093:9093

I get the followin infinitely looping error:

[2018-05-11 02:04:08,310] WARN [Controller id=0, targetBrokerId=1] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

The error is somewhat related to my KAFKA_ADVERTISED_LISTENERS (same thing happens with KAFKA_ADVERTISED_HOST_NAME and KAFKA_ADVERTISED_PORT) environmental variable. If I remove it, it works, but then my orderers will not receive the brokers address and I will lose communication between orderers and brokers.

How do I maintain the internal connection between my brokers and the external connection to my orderers?

Answer 1

I was able to get everything connected by changing my kafka1 container to look like this:

kafka1.hyperfabric.xyz:
  image: hyperledger/fabric-kafka
  restart: always
  container_name: kafka1.hyperfabric.xyz
  environment:
    - KAFKA_PORT=10092
    - KAFKA_MESSAGE_MAX_BYTES=103809024
    - KAFKA_REPLICA_FETCH_MAX_BYTES=103809024
    - KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
    - KAFKA_MIN_INSYNC_REPLICAS=1
    - KAFKA_DEFAULT_REPLICATION_FACTOR=1
    - KAFKA_ZOOKEEPER_CONNECT=zookeeper0.hyperfabric.xyz:2181
    - KAFKA_BROKER_ID=1
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka1.hyperfabric.xyz:10092
  ports:
    - 10092:10092
    - 10093:9093

The difference being - KAFKA_PORT=10092 and - 10092:10092

Answer 2

Within the container is where your listener variables are configured. 10092 is the port on your host, not the container.

Therefore, you want this

PLAINTEXT://kafka1.hyperfabric.xyz:9092

Along with

ports:
    - 10092:9092

Or use 10092 for both the listener and ports (plus KAFKA_ADVERTISED_PORT)

---

Alternatively, if you are on a unix host, you could do network_mode: host mode, lose the port forwarding, then you can use different ports for advertised listeners.