我读卡夫卡一个JSON,FieldExtractionBolt读取JSON数据提取到元组值并将其传递给CassandraWriterBolt,这反过来写在卡珊德拉的记录写入所有元组值成单独的列。
卡夫卡JSON消息 -
{"pair":"GBPJPY","bid":134.4563,"ask":134.4354}
FieldExtractionBolt -
String message = tuple.getStringByField("message");
Map values = new Gson().fromJson(message, Map.class);
basicOutputCollector.emit(new Values(values.get("pair"), values.get("bid"), values.get("ask")));
CassandraWriterBolt -
return (CassandraWriterBolt) new CassandraWriterBolt(async(simpleQuery("INSERT INTO currency(pair, ask, bid) VALUES (?, ?, ?);").with(fields("pair", "ask", "bid")))
我试着写基于这里给出的答案测试- 如何通过编程方式将消息发送到端到端风暴拓扑的测试功能
在我的项目,我在Spring中定义的配置我所有的螺栓,嘴和溪流。 这使得读/写我的拓扑结构非常简单。 我通过获取螺栓,嘴和流咖啡豆的ApplicationContext建立拓扑。 在我的Spring配置,KafkaSpout和CassandraWriterBolt在下面“督促”配置文件定义,让他们只在督促下“测试”使用档案I定义KafkaSpout和CassandraWriterBolt存根。 对于KafkaSpout,我用FixedToupleSpout和CassandraWriterBolt我用TestWordCounter。
这是我的测试
@Test
public void testTopology(){
StormTopology topology = SpringBasedTopologyBuilder.getInstance().buildStormTopologyUsingApplicationContext(applicationContext);
TestJob COMPLETE_TOPOLOGY_TESTJOB = (cluster) -> {
MockedSources mocked = new MockedSources();
mocked.addMockData("kafkaSpout",
new Values("{\"pair\":\"GBPJPY\",\"bid\":134.4563,\"ask\":134.4354}"),
new Values("{\"pair\":\"GBPUSD\",\"bid\":1.4563,\"ask\":1.4354}"));
Config topoConf = new Config();
topoConf.setNumWorkers(2);
CompleteTopologyParam ctp = new CompleteTopologyParam();
ctp.setMockedSources(mocked);
ctp.setStormConf(topoConf);
Map<String, List<FixedTuple>> results = Testing.completeTopology(cluster, topology, ctp);
List<List<Object>> cassandraTuples = Testing.readTuples(results, "cassandraWriterBolt");
List<List<Object>> expectedCassandraTuples = Arrays.asList(Arrays.asList("GBPJPY", 1), Arrays.asList("GBPUSD", 1),
Arrays.asList("134.4563", 1), Arrays.asList("1.4563", 1), Arrays.asList("134.4354", 2));
assertTrue(expectedCassandraTuples + " expected, but found " + cassandraTuples,
Testing.multiseteq(expectedCassandraTuples, cassandraTuples));
MkClusterParam param = new MkClusterParam();
param.setSupervisors(4);
Testing.withSimulatedTimeLocalCluster(param, COMPLETE_TOPOLOGY_TESTJOB);
}
@Configuration
@Import(MainApplication.class)
public static class TestConfig
{
@Bean
public IRichSpout kafkaSpout(){
return new FixedTupleSpout(Arrays.asList(new FixedTuple(Arrays.asList("{\"pair\":\"GBPJPY\",\"bid\":134.4563,\"ask\":134.4354"))), new Fields(new String[]{"message"}));
}
@Bean
public IBasicBolt cassandraWriterBolt(){
return new TestWordCounter();
}
}
结果我得到的是不是我期待的。 我收到以下错误 -
java.lang.AssertionError: [[GBPJPY, 1], [GBPUSD, 1], [134.4563, 1], [1.4563, 1], [134.4354, 2]] expected, but found [[GBPJPY, 1], [GBPUSD, 1]]
看起来,TestWordCounter只是读第一个值作为一个元组(仅货币对以及跳过买入和卖出)。 似乎TestWordCounter这里不是正确的选择。 什么是正确的存根CassandraWriterBolt,这样我可以断言,这样会得到2只记录一个GBPJPY,另一个为英镑兑美元与他们的买入价和卖出价呢?