Kafka Consumer Marking the coordinator 2147483647

2020-05-22 03:42发布

站内文章 / 前沿技术
86 0 0

问题:

I am using Kafka Server 0.9 with consumer kafka-client version 0.9 and kafka-producer 0.8.2.

Every thing is working great except i am getting lot of info that the coordinator is dead on the consumer

2016-02-25 19:30:45.046  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.048  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.049  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.050  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.051  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.052  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.053  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.054  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.055  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.056  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.057  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.058  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.059  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.060  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.061  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.063  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.064  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.065  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.066  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.067  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.069  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.070  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.071  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.073  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.074  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.076  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.077  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.078  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.080  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.081  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.082  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.084  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.085  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.087  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.088  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.090  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.091  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.093  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.

I also noticed that the producer is having disconnect connect every 10 minute as the below 

2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Fetching metadata from broker id:0,host:192.168.72.30,port:9092 with correlation id 41675 for 1 topic(s) Set(act)
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Connected to 192.168.72.30:9092 for producing
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Disconnecting from 192.168.72.30:9092
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Disconnecting from kafkauk.XXXXXXXXXX.co:9092
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Connected to kafkauk.XXXXXXXXXX.co:9092 for producing

this is my producer configuration

metadata.broker.list=192.168.72.30:9092
serializer.class=kafka.serializer.StringEncoder
request.required.acks=1
linger.ms=2000
batch.size=500

and consumer config

bootstrap.servers: kafkauk.xxxxxxxx.co:9092
group.id: cdrServer
client.id: cdrServer
enable.auto.commit: true
auto.commit.interval.ms: 1000
session.timeout.ms: 30000
key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.StringDeserializer

I could not figure out what does these mean and should i neglect them or i am missing something in the configuration

After i change kafka to debug level on the consumer i found the below 

2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.NetworkClient   : Node 2147483647 disconnected.
2016-03-13 18:21:55.586  INFO 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Issuing group metadata request to broker 0
2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.NetworkClient   : Sending metadata request ClientRequest(expectResponse=true
, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=183025,client_id=cdrServer}, body={topics=[act]}), isInitiatedByNetworkCli
ent, createdTimeMs=1457893315586, sendTimeMs=0) to node 0
2016-03-13 18:21:55.591 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.Metadata        : Updated cluster metadata version 296 to Cluster(nodes = [N
ode(0, kafkauk.xxxxxxxxx.co, 9092)], partitions = [Partition(topic = act, partition = 0, leader = 0, replicas = [0,], isr = [0,]])
2016-03-13 18:21:55.592 DEBUG 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Group metadata response ClientResponse(receivedTimeMs=1457
893315592, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFu
tureCompletionHandler@1e2de777, request=RequestSend(header={api_key=10,api_version=0,correlation_id=183024,client_id=cdrServer}, body={group_id=cdrServer}), c
reatedTimeMs=1457893315586, sendTimeMs=1457893315586), responseBody={error_code=0,coordinator={node_id=0,host=kafkauk.xxxxxxxx.co,port=9092}})

I am not sure it is a network problem because it happen every 9 minute exactly

Update

I found that is directly related to 

connections.max.idle.ms: 300000

What ever i put then i will get disconnected at this value

回答1:

In my case the message was in logs when I try to assign partitions manually. After I've read in api docs of the new consumer follow notice:

It is also possible for the consumer to manually assign specific partitions (similar to the older "simple" consumer) using assign(Collection). In this case, dynamic partition assignment and consumer group coordination will be disabled.

That is, if you have code like this:

    KafkaConsumer<String, String> consumer = new KafkaConsumer(props);
    consumer.assign( Arrays.asList(
            new TopicPartition("topic", 0),
            new TopicPartition("topic", 1)
    ));

then the message "Marking the coordinator 2147483647 dead" puts in our logs always.



回答2:

Marking the coordinator dead happens when there is a Network communication error between the Consumer Client and the Coordinator (Also this can happen when the Coordinator dies and the group needs to rebalance). There are a variety of situations (offset commit request, fetch offset, etc) that can cause this issue. I will suggest that you research what's causing this situations



回答3:

I have faced the same issue. Finally after follow Shannon recommendation about TRACING logs, I used:

logging.level.org.apache.kafka=TRACE

To find out that my client was trying to resolve Euler:9092 as coordinator... Local name!! So I commented out and changed listeners and advertised.listeners values in server.properties file. It is working now! :-)



回答4:

This is basically you are not able to reach to Kafka.
In my case I was running Kafka in vagrant box, and if I start VPN it refresh
vagrant ip hence it was not able to connect to it.
Possible Solution: In this case stop VPN and start your vagrant.



回答5:

This may also be related to a long garbage collection stop-the-world phase. In my case I encountered this message after > 10 sec GCs.



回答6:

I faced this issue today and solved it (temporarily, might I add). I've posted an answer here on how I did it.



标签： apache-kafka kafka-consumer-api
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~