Kafka new producer timeout

Question

I'm using the new kafka producer client and set the timeout.ms property to 50 ms.

Here is the complete configuration used in the producer:

props.put("acks", "1");
props.put("buffer.memory", "33554432");
props.put("retries", "1");
props.put("batch.size", "16384");
props.put("client.id", "foo");
props.put("linger.ms", "0");
props.put("timeout.ms", "50");

The request average response time in some moments of high load is 4 seconds, but I don't get any timeout error.

Does Someone know how this timeout is calculated, when it begins to be counted and when it finishes? Is there a way to configure a timeout to start from the moment the producer's send method is called?

Answer 1

The new timeout.ms property works with the ack configuration of the producer. For example consider the following situation

ack = all
timeout.ms = 3000

In this case ack = all means that the leader will not respond untill it receives acknowledgement for the full set of in-sync replicas (ISR) and the maximum wait time to get this acknowledgement will be 3000 ms. If it didn't receive the expected number of acknowledgements within the given time it will return an error.

Also note this property does not consider network latency.

From the doc page :

The configuration controls the maximum amount of time the server will wait for acknowledgments from followers to meet the acknowledgment requirements the producer has specified with the acks configuration. If the requested number of acknowledgments are not met when the timeout elapses an error will be returned. This timeout is measured on the server side and does not include the network latency of the request.

So in your case with ack=1 (I am not 100% sure about this and open to any correction if applicable) if the leader is not able to respond (after writing the record to its own log without awaiting full acknowledgement from all followers) within 50ms should throw an error.

Answer 2

In the new Kafka 2.0 Producer API, you can use one of the following properties : https://kafka.apache.org/documentation/#producerconfigs

See usage examples on https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

The timeout is now defined by the max.block.ms property.