为什么一个水槽源需要认识到的消息的格式?(Why does a Flume source need

2019-10-18 19:23发布

据来自水槽文件在这里

水槽源消耗由如web服务器的外部源交付给它的事件。 外部源发送事件到水槽在由所述目标水槽源识别的格式。 例如,阿夫罗水槽源可以被用于接收来自阿夫罗客户端或其他水槽剂阿夫罗事件,其从阿夫罗宿发送事件的流动。

为什么一个水槽源需要认识或了解该消息的格式? 虽然所有它它是将所述消息转发到所述通道中的一个。

Answer 1:

由于我所学到的,水槽封装在由头部和有效载荷(该transfering数据)制作的事件数据包中的数据transfering。 从文档:

水槽事件被定义为具有字节有效载荷和一组可选的串属性的数据流的单元。

之前你的文档引用。

您所指定的格式是事件数据包的格式,而不是你的数据的格式。

让我们假设你有这样的代理:

plain_to_avro_translator.sources = plain-source avro-source
plain_to_avro_translator.sinks = avro-sink local-file-sink
plain_to_avro_translator.channels = mem-channel1 mem-channel2

plain_to_avro_translator.sources.plain-source.channels = mem-channel1
plain_to_avro_translator.sources.plain-source.type = exec
plain_to_avro_translator.sources.plain-source.restart = true
plain_to_avro_translator.sources.plain-source.restartThrottle = 40000
plain_to_avro_translator.sources.plain-source.command = cat /home/user/data.log

plain_to_avro_translator.sinks.avro-sink.channel = mem-channel1
plain_to_avro_translator.sinks.avro-sink.type = thrift
plain_to_avro_translator.sinks.avro-sink.hostname = 192.168.200.43
plain_to_avro_translator.sinks.avro-sink.port = 6000

plain_to_avro_translator.channels.mem-channel1.type = memory
plain_to_avro_translator.channels.mem-channel1.capacity = 100
plain_to_avro_translator.channels.mem-channel1.transactionCapacity = 100

plain_to_avro_translator.sources.avro-source.channels = mem-channel2
plain_to_avro_translator.sources.avro-source.type = thrift
plain_to_avro_translator.sources.avro-source.bind = 0.0.0.0
plain_to_avro_translator.sources.avro-source.port = 6000

plain_to_avro_translator.channels.mem-channel2.type = memory
plain_to_avro_translator.channels.mem-channel2.capacity = 100
plain_to_avro_translator.channels.mem-channel2.transactionCapacity = 100

plain_to_avro_translator.sinks.local-file-sink.channel = mem-channel2
plain_to_avro_translator.sinks.local-file-sink.type = file_roll
plain_to_avro_translator.sinks.local-file-sink.sink.directory = /home/user/flume_output

这将没有任何问题的工作,并且不依赖从data.log格式(你可以写任何你需要在任何格式)。 如果您尝试设置Avro的散热器类型Avro的,而不是节俭,你会从Avro的源得到的错误,因为它需要节俭格式的事件。

汇和源需要知道如何解析事件包。

希望我很好。 请人纠正我,如果我错了。



文章来源: Why does a Flume source need to recognize the format of the message?
标签: flume