Test Kafka Streams topology

Question

I'm searching a way to test a Kafka Streams application. So that I can define the input events and the test suite shows me the output.

Is this possible without a real Kafka setup?

Answer 1

1. As you are asking if it is possible to test Kafka Streams application without a real Kafka setup, you might try this Mocked Streams library in Scala. Mocked Streams 1.0 is a library for Scala >= 2.11.8 which allows you to unit-test processing topologies of Kafka Streams applications (since Apache Kafka >=0.10.1) without Zookeeper and Kafka Brokers. Reference: https://github.com/jpzk/mockedstreams

2. You can also use scalatest-embedded-kafka which is a library that provides an in-memory Kafka broker to run your ScalaTest specs against. It uses Kafka 0.10.1.1 and ZooKeeper 3.4.8.
   Reference: https://github.com/manub/scalatest-embedded-kafka#scalatest-embedded-kafka-streams

Good luck!

Answer 2

You should check Kafka Unit here.

Your test setup should look something like this:

KafkaUnit kafkaUnitServer = new KafkaUnit();
kafkaUnitServer.startup();
kafkaUnitServer.createTopic(testTopic);
KeyedMessage<String, String> keyedMessage = new KeyedMessage<>(testTopic, "key", "value");
kafkaUnitServer.sendMessages(keyedMessage);

And then to read your messages and assert that everything went ok you do something like this:

List<String> messages = kafkaUnitServer.readMessages(testTopic, 1);

This actually spins up an embedded kafka that helps you have everything you need contained in a test.

You could get a little bit fancier and setup your embedded kafka as a setup() method (or setupSpec() in Spock) and stop your embedded kafka in a teardown().

Answer 3

If you want to test a Kafka Stream Topology that uses Processor API, the code provided by Dmitry may not work properly. So after a few hours of researching in the Javadocs and official docs I came out with a working code in order to test a custom processor that you have implemented using JUnit.

public class TopologySpec {

private TopologyTestDriver testDriver;

@Before
public void setup() {
    // Processor API
    Topology topology = new Topology();
    topology.addSource("sourceProcessor", "input-topic");
    // In this case, 'EventProcessor' is a custom processor
    // that I implemented and I want to test
    topology.addProcessor("processor", EventProcessor::new, "sourceProcessor");
    topology.addSink("sinkProcessor", "output-topic", "processor");

    // Setup test driver
    Properties config = new Properties();
    config.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
    config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
    // EventProcessor is a <String,String> processor 
    // so we set those serders
    config.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
    config.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
    testDriver = new TopologyTestDriver(topology, config);
}

@After
public void tearDown() {
    testDriver.close(); // Close processors after finish the tests
}

@Test
public void firstTest() {
    // Simulate a producer that sends the message "value,val" without key
    ConsumerRecordFactory factory =
            new ConsumerRecordFactory(new StringSerializer(), new StringSerializer());

    testDriver.pipeInput(factory.create("input-topic", "value,val"));

    // Simulate a consumer that reads from the output topic 
    // where are supposed to be the messages after being processed
    // by your custom processor
    ProducerRecord<String, String> record1 =
            testDriver.readOutput("output-topic", new StringDeserializer(), new StringDeserializer());

    // Compare the output to ensure that your custom processor
    // is working properly. In this case, my processor consumes
    // the message, concatenates ":::processed" to it, and
    // push it to the output-topic
    OutputVerifier.compareValue(record1, "value,val:::processed");
}
}

Answer 4

you can use https://github.com/jpzk/mockedstreams see the example below...

import com.madewithtea.mockedstreams.MockedStreams

val input = Seq(("x", "v1"), ("y", "v2"))
val exp = Seq(("x", "V1"), ("y", "V2"))
val strings = Serdes.String()

MockedStreams()
  .topology { builder => builder.stream(...) [...] }
  .input("topic-in", strings, strings, input)
  .output("topic-out", strings, strings, exp.size) shouldEqual exp

hope this helps you...

Answer 5

Update Kafka 1.1.0 (released 23-Mar-2018):

KIP-247 added official test utils. Per the Upgrade Guide:

There is a new artifact kafka-streams-test-utils providing a TopologyTestDriver, ConsumerRecordFactory, and OutputVerifier class. You can include the new artifact as a regular dependency to your unit tests and use the test driver to test your business logic of your Kafka Streams application. For more details, see KIP-247.

From the documentation:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams-test-utils</artifactId>
    <version>1.1.0</version>
    <scope>test</scope>
</dependency>

The test driver simulates the library runtime that continuously fetches records from input topics and processes them by traversing the topology. You can use the test driver to verify that your specified processor topology computes the correct result with the manually piped in data records. The test driver captures the results records and allows to query its embedded state stores:

// Create your topology
Topology topology = new Topology();
Properties config = new Properties();
config.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");

// Run it on the test driver
TopologyTestDriver testDriver = new TopologyTestDriver(topology, config);

// Feed input data
ConsumerRecordFactory<String, Integer> factory = new ConsumerRecordFactory<>("input-topic", new StringSerializer(), new IntegerSerializer());
testDriver.pipe(factory.create("key", 42L));

// Verify output
ProducerRecord<String, Integer> outputRecord = testDriver.readOutput("output-topic", new StringDeserializer(), new LongDeserializer());

See the documentation for details.

---

ProcessorTopologyTestDriver is available as of 0.11.0.0. It is available in the kafka-streams test artifact (specified with <classifier>test</classifier> in Maven):

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>0.11.0.0</version>
    <classifier>test</classifier>
    <scope>test</scope>
</dependency>

You will also need to add the kafka-clients test artifact:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.11.0.0</version>
    <classifier>test</classifier>
    <scope>test</scope>
</dependency>

Then you can use the test driver. Per the Javadoc, first create a ProcessorTopologyTestDriver:

StringSerializer strSerializer = new StringSerializer();
StringDeserializer strDeserializer = new StringDeserializer();
Properties props = new Properties();
props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9091");
props.setProperty(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, CustomTimestampExtractor.class.getName());
props.setProperty(StreamsConfig.KEY_SERIALIZER_CLASS_CONFIG, strSerializer.getClass().getName());
props.setProperty(StreamsConfig.KEY_DESERIALIZER_CLASS_CONFIG, strDeserializer.getClass().getName());
props.setProperty(StreamsConfig.VALUE_SERIALIZER_CLASS_CONFIG, strSerializer.getClass().getName());
props.setProperty(StreamsConfig.VALUE_DESERIALIZER_CLASS_CONFIG, strDeserializer.getClass().getName());
StreamsConfig config = new StreamsConfig(props);
TopologyBuilder builder = ...
ProcessorTopologyTestDriver driver = new ProcessorTopologyTestDriver(config, builder);

You can feed input into the topology as though you had actually written to one of the input topics:

driver.process("input-topic", "key1", "value1", strSerializer, strSerializer);

And read output topics:

ProducerRecord<String, String> record1 = driver.readOutput("output-topic-1", strDeserializer, strDeserializer);
ProducerRecord<String, String> record2 = driver.readOutput("output-topic-1", strDeserializer, strDeserializer);
ProducerRecord<String, String> record3 = driver.readOutput("output-topic-2", strDeserializer, strDeserializer);

Then you can assert on these results.

Answer 6

You can just run a single Zookeeper and broker locally to test a Kafka Streams application.

Just follow those quick start guides:

• local ZK and broker setup: http://kafka.apache.org/quickstart
• http://docs.confluent.io/current/streams/quickstart.html

Also check out this Kafka Streams examples (with detailed walk through instructions in the JavaDocs):

• https://github.com/confluentinc/examples/tree/3.1.x/kafka-streams