Kafka Streams Topology stuck in num.stream.threads

2019-07-23 13:30发布

问题:

There is a topology:

kStreamBuilder.stream(kafkaProperties.getInboundTopicName(), consumed)
            .filterNot((k,v) -> Objects.isNull(v))
            .transform(() -> new CustomTransformer(...))
            .transform(() -> new AnotherTransformer(...))
            .to(kafkaProperties.getOutTopicName(), resultProduced);

with configured

num.stream.threads: 50

On startup application stuck with constantly logging messages(I'm not 100% sure it stuck but after 20 minutes there are no changes in the state and CPU, network usage is very high) :

State transition from RUNNING to PARTITIONS_REVOKED    
AbstractCoordinator          : [Consumer clientId=consumer_id-StreamThread-1-consumer, groupId=group_id] (Re-)joining group
AbstractCoordinator          : [Consumer clientId=consumer_id-StreamThread-2-consumer, groupId=group_id] (Re-)joining group
AbstractCoordinator          : [Consumer clientId=consumer_id-StreamThread-3-consumer, groupId=group_id] (Re-)joining group
AbstractCoordinator          : [Consumer clientId=consumer_id-StreamThread-4-consumer, groupId=group_id] (Re-)joining group
AbstractCoordinator          : [Consumer clientId=consumer_id-StreamThread-5-consumer, groupId=group_id] (Re-)joining group

etc.

Topic has 100 partitions.

What we noticed: every transformer uses it's own persistentStateStore. After replacing it to inMemoryStateStore there still were logs written above but after ~3 minutes topology started successfully.

Kafka streams version 2.1.0. Broker version 1.1.0