kafka broker not available at starting

Question

I set on a ubuntu node of a cluster a kafka 0.11.0.0 instance. Until some weeks ago everything worked fine, today I'm trying to starting it and I obtain this error after the boot:

[2017-09-11 16:21:13,894] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2017-09-11 16:21:18,998] WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2017-09-11 16:21:21,991] WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
... and so on...

My server.properties:

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true

############################# Socket Server Settings ##########################$

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://9092

# Hostname and port the broker will advertise to producers and consumers. If no$
# it uses the value for "listeners" if configured.  Otherwise, it will use the $
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://hidden_ip:55091

I edited advertised.listeners because there is a proxy to redirect requests to the broker. Anyway until some weeks ago everything worked fine...

My step to start kafka:

1- service zookeeper start 
2- ./kafka_2.11-0.11.0.0/bin/kafka-server-start.sh ~/kafka_2.11-0.11.0.0/config/server.properties

Any advises? Thank you

Answer 1

I was also facing the same problem on WINDOWS 10 and went through all the answers in this post. What caused this problem for me and how I solved is this, On my fresh windows machine, I did a jre (jre-8u221) installation and then followed the steps mentioned in Apache Kafka documentation to start zookeeper, kafka server, send messages through producer etc. But I was getting errors related to broker not available. I then downloaded jdk (jdk-8u221 from Oracle) and installed it. Set the JAVA_HOME and Path variables and restarted the machine. Everything after worked in one go.

Answer 2

I server.propertes:

listeners=PLAINTEXT://hidden_ip:9092

when I run:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic lt1 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic lt1
I get same error just like you get.and I try run :

bin/kafka-console-producer.sh --broker-list hidden_ip:9092 --topic lt1

it work,I get no error.
So i think you should check what's port kafka server used.This error when port is not esestablished or you connect wrong ip.
PS:I run this on same machine.

Answer 3

error message: "Connection to node -1 could not be established. Broker may not be available"

Solution:

I was getting this error because I was having a different version of kafka running on cluster vs local. Make sure the version is the same. I was specifying the version in pom.xml under properties tag

Answer 4

i configed the kafka server.properties's listener as listeners=PLAINTEXT://**10.127.96.151**:9092

but i request the consumer as ./kafka-console-consumer.sh --bootstrap-server **localhost**:9092 --topic topic1 --from-beginning , then it bursts the WARNs.

when i fix the consumer bootstrap-server as 10.127.96.151:9092, it runs all well with no more WARNs

Answer 5

I also had the same issue...

So I start by asking Zookeeper some data

echo dump | nc localhost 2181

I get back something like this

SessionTracker dump:
Session Sets (3):
0 expire at Sun Dec 03 18:15:48 GST 2017:
0 expire at Sun Dec 03 18:15:51 GST 2017:
1 expire at Sun Dec 03 18:15:54 GST 2017:
0x1601c9a25190000
ephemeral nodes dump:
Sessions with Ephemerals (1):
0x1601c9a25190000:
/controller
/brokers/ids/0

I read this to mean that I have 1 Broker available - and it is has an id of 0.

So, lets find about that broker.

bin/zookeeper-shell.sh localhost:2181 <<< "get /brokers/ids/0"

And I see

{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.220:9092"],"jmx_port":-1,"host":"192.168.1.220","timestamp":"1512308520781","port":9092,"version":4}
cZxid = 0x69
ctime = Sun Dec 03 17:42:00 GST 2017
mZxid = 0x69
mtime = Sun Dec 03 17:42:00 GST 2017
pZxid = 0x69
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1601c9a25190000
dataLength = 196
numChildren = 0

This rather startled me ... PLAINTEXT://192.168.1.220:9092, so when I changed my connect string to read

bin/kafka-console-producer.sh  --broker-list PLAINTEXT://192.168.1.220:9092  --topic test

It all now worked.

Good luck !!

Answer 6

We faced the same situation, when we started to search the logs for the actual cause of the error.

When we were starting the Kafka cluster, it was using by default

listeners=PLAINTEXT://:9092

to connect and was not able to find the host.

We changed the line to

listeners=PLAINTEXT://<our ip address> :9092,

then

listeners=PLAINTEXT://<our ip address> :9093, 

in our server-1.properties ( another broker id file ) and the same we replicated in other server.properties file and restarted our cluster.