Why can't I increase session.timeout.ms?

Question

I want to increase session.timeout.ms to allow longer time for processing the messages received between poll() calls. However when I change session.timeout.ms to a higher value than 30000, it fails to create Consumer object and throws below error.

Could anyone tell why can't I increase session.timeout.ms value or if I am missing something?

0    [main] INFO  org.apache.kafka.clients.consumer.ConsumerConfig  - ConsumerConfig values: 

request.timeout.ms = 40000
check.crcs = true
retry.backoff.ms = 100
ssl.truststore.password = null
ssl.keymanager.algorithm = SunX509
receive.buffer.bytes = 262144
ssl.cipher.suites = null
ssl.key.password = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.provider = null
sasl.kerberos.service.name = null
session.timeout.ms = 40000
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [server-name:9092]
client.id = 
fetch.max.wait.ms = 500
fetch.min.bytes = 50000
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
sasl.kerberos.kinit.cmd = /usr/bin/kinit
auto.offset.reset = latest
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
ssl.endpoint.identification.algorithm = null
max.partition.fetch.bytes = 2097152
ssl.keystore.location = null
ssl.truststore.location = null
ssl.keystore.password = null
metrics.sample.window.ms = 30000
metadata.max.age.ms = 300000
security.protocol = PLAINTEXT
auto.commit.interval.ms = 5000
ssl.protocol = TLS
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
ssl.trustmanager.algorithm = PKIX
group.id = test7
enable.auto.commit = false
metric.reporters = []
ssl.truststore.type = JKS
send.buffer.bytes = 131072
reconnect.backoff.ms = 50
metrics.num.samples = 2
ssl.keystore.type = JKS
heartbeat.interval.ms = 3000

Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:624) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:518) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:500)

Answer 1

i am using spring-kafka

i had added the following config but the consumer still was not up:

buildProperties.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, env.getProperty("kafka.user-events-min-bytes"));
    buildProperties.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, env.getProperty("kafka.user-events-wait-time-ms") );
    buildProperties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, env.getProperty("kafka.user-events-wait-time-ms") );
    buildProperties.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, env.getProperty("kafka.user-events-request-timeout-ms"));
    buildProperties.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, env.getProperty("kafka.user-events-wait-time-ms"));

i figured out it was failing because poll.timeout was 1000; adding the following config helped:

factory.getContainerProperties().setPollTimeout(Integer.parseInt(env.getProperty("kafka.user-events-wait-time-ms")));

Answer 2

The range of consumer session timeout is controlled by broker group.max.session.timeout.ms(default 30s) and group.min.session.timeout.ms(default 6s).

You should increase group.max.session.timeout.ms first in broker side, otherwise you will get "The session timeout is not within an acceptable range.".

Answer 3

• These conditions needed to be keep in mind to change session.timeout.ms:
  1. group.max.session.timeout.ms in the server.properties > session.timeout.ms in the consumer.properties.
  2. group.min.session.timeout.ms in the server.properties < session.timeout.ms in the consumer.properties.
  3. request.timeout.ms > session.timeout.ms and fetch.max.wait.ms
  4. (session.timeout.ms)/3 > heartbeat.interval.ms
  5. session.timeout.ms > Worst case processing time of Consumer Records per consumer poll(ms).