我想了解如何尾部水槽-NG文件，这样我可以把数据导入HDFS。 在第一种情况下我已经安装一个简单的conf文件：
tail1.sources = source1
tail1.sinks = sink1
tail1.channels = channel1
tail1.sources.source1.type = exec
tail1.sources.source1.command = tail -F /var/log/apache2/access.log
tail1.sources.source1.channels = channel1
tail1.sinks.sink1.type = logger
tail1.channels.channel1.type = memory
tail1.channels.channel1.capacity = 1000
tail1.channels.channel1.transactionCapacity = 100
tail1.sources.source1.channels = channel1
tail1.sinks.sink1.channel = channel1
这是一个测试，我的期望是，我会在控制台上看到输出。 我用下面的命令运行以下命令：
flume-ng agent --conf-file tail1.conf -n tail1 -Dflume.root.logger=DEBUG,INFO,console
我得到以下的输出：
12/12/05 11时01分07秒INFO lifecycle.LifecycleSupervisor：启动周期监1 12/12/05 11时01分07秒INFO node.FlumeNode：水槽节点开始 - tail1 12/12/05 11时01分07秒INFO nodemanager.DefaultLogicalNodeManager：节点管理器启动12/12/05 11时01分07秒INFO lifecycle.LifecycleSupervisor：启动生命周期监事8 12/12/05 11时01分07秒INFO properties.PropertiesFileConfigurationProvider：配置提供商开始12/12/05 11 ：01：07 INFO properties.PropertiesFileConfigurationProvider：重新加载配置文件：tail1.conf 12/12/05 11时01分07秒INFO conf.FlumeConfiguration：处理：信宿1 12/12/05 11时01分07秒INFO conf.FlumeConfiguration：处理：信宿12/12/05 11时01分07秒INFO conf.FlumeConfiguration：新增汇：信宿代理：tail1 12/12/05 11时01分07秒INFO conf.FlumeConfiguration：后验证水槽配置包含配置代理： tail1] 12/12/05 11时01分07秒INFO properties.PropertiesFileConfigurationProvider：创建渠道12/12/05 11时01分08秒INFO instrumentatio n.MonitoredCounterGroup：类型Monitoried计数器组：CHANNEL，名称：通道1，注册成功。 12/12/05 11时01分08秒INFO properties.PropertiesFileConfigurationProvider：创建的通道通道1 12/12/05 11时01分08秒INFO sink.DefaultSinkFactory：信宿1，类型：记录器12/12/05 11水槽的创建实例： 01:08 INFO nodemanager.DefaultLogicalNodeManager：启动新的配置：{sourceRunners：{source1中= EventDrivenSourceRunner：{源：org.apache.flume.source.ExecSource@1839aa9}} sinkRunners：{信宿1 = SinkRunner：{政策：org.apache。 flume.sink.DefaultSinkProcessor@11f0c98 counterGroup：{名称：空计数器：{}}}}渠道：{channel1=org.apache.flume.channel.MemoryChannel@1740f55}} 12/12/05 11时01分08秒INFO节点管理器.DefaultLogicalNodeManager：起始通道通道1 12/12/05 11时01分08秒INFO instrumentation.MonitoredCounterGroup：组件类型：CHANNEL，名称：通道开始12/12/05 11时01分08秒INFO nodemanager.DefaultLogicalNodeManager：启动汇信宿12 / 12/05 11时01分08秒INFO nodemanager.DefaultLogicalNodeManager：起始源源1 12/12/05 11时01分08秒INFO source.ExecSource：执行 源开始命令：尾-F /var/log/apache2/access.log
但是没有进一步发生。
我还有一个会议，我有以下命令：
tail -F /var/log/apache2/access.log
我在哪里可以看到该文件被写入：
192.168.1.81 - - [05/Dec/2012:10:58:07 +0000] "GET / HTTP/1.1" 200 483 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
192.168.1.81 - - [05/Dec/2012:10:58:07 +0000] "GET /favicon.ico HTTP/1.1" 404 502 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
192.168.1.81 - - [05/Dec/2012:10:58:21 +0000] "GET / HTTP/1.1" 304 209 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
192.168.1.81 - - [05/Dec/2012:10:58:22 +0000] "GET /favicon.ico HTTP/1.1" 404 502 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
你能帮我吗？ 我彻底糊涂了。
你的配置文件看起来不错。 我用它在CDH4并担任你的预期，我所做的只是更改日志文件位置的尾巴。 我看到控制台上的输出。 在我的情况下新的日志数据被连续写入到我拖尾文件。 在您的数据的时间戳使其看起来像这可能不是你的情况为例。
下面是一个更完整的conf例如更符合什么，我认为你正在尝试做的。 它将尾部的文件，并写入新的HDFS文件中的每个10分钟10K记录。 在agent1.sources.source1.command更改为您tail命令，并更改根据您的HDFS配置的agent1.sinks.sink1.hdfs.path和agent1.sinks.sink1.hdfs.filePrefix。
# A single-node Flume configuration
# uses exec and tail and will write a file every 10K records or every 10 min
# Name the components on this agent
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure source1
agent1.sources.source1.type = exec
agent1.sources.source1.command = tail -f /home/cloudera/LogCreator/fortune_log.log
# Describe sink1
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = hdfs://localhost/flume/logtest/
agent1.sinks.sink1.hdfs.filePrefix = LogCreateTest
# Number of seconds to wait before rolling current file (0 = never roll based on time interval)
agent1.sinks.sink1.hdfs.rollInterval = 600
# File size to trigger roll, in bytes (0: never roll based on file size)
agent1.sinks.sink1.hdfs.rollSize = 0
#Number of events written to file before it rolled (0 = never roll based on number of events)
agent1.sinks.sink1.hdfs.rollCount = 10000
# number of events written to file before it flushed to HDFS
agent1.sinks.sink1.hdfs.batchSize = 10000
agent1.sinks.sink1.hdfs.txnEventMax = 40000
# -- Compression codec. one of following : gzip, bzip2, lzo, snappy
# hdfs.codeC = gzip
#format: currently SequenceFile, DataStream or CompressedStream
#(1)DataStream will not compress output file and please don't set codeC
#(2)CompressedStream requires set hdfs.codeC with an available codeC
agent1.sinks.sink1.hdfs.fileType = DataStream
agent1.sinks.sink1.hdfs.maxOpenFiles=50
# -- "Text" or "Writable"
#hdfs.writeFormat
agent1.sinks.sink1.hdfs.appendTimeout = 10000
agent1.sinks.sink1.hdfs.callTimeout = 10000
# Number of threads per HDFS sink for HDFS IO ops (open, write, etc.)
agent1.sinks.sink1.hdfs.threadsPoolSize=100
# Number of threads per HDFS sink for scheduling timed file rolling
agent1.sinks.sink1.hdfs.rollTimerPoolSize = 1
# hdfs.kerberosPrin--cipal Kerberos user principal for accessing secure HDFS
# hdfs.kerberosKey--tab Kerberos keytab for accessing secure HDFS
# hdfs.round false Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
# hdfs.roundValue1 Rounded down to the highest multiple of this (in the unit configured using
# hdfs.roundUnit), less than current time.
# hdfs.roundUnit second The unit of the round down value - second, minute or hour.
# serializer TEXT Other possible options include AVRO_EVENT or the fully-qualified class name of an implementation of the EventSerializer.Builder interface.
# serializer.*
# Use a channel which buffers events to a file
# -- The component type name, needs to be FILE.
agent1.channels.channel1.type = FILE
# checkpointDir ~/.flume/file-channel/checkpoint The directory where checkpoint file will be stored
# dataDirs ~/.flume/file-channel/data The directory where log files will be stored
# The maximum size of transaction supported by the channel
agent1.channels.channel1.transactionCapacity = 1000000
# Amount of time (in millis) between checkpoints
agent1.channels.channel1.checkpointInterval 30000
# Max size (in bytes) of a single log file
agent1.channels.channel1.maxFileSize = 2146435071
# Maximum capacity of the channel
agent1.channels.channel1.capacity 10000000
#keep-alive 3 Amount of time (in sec) to wait for a put operation
#write-timeout 3 Amount of time (in sec) to wait for a write operation
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
两个可能的原因：第一，一旦你发出的命令和源被启动后，接收端必须获得本身注册，并开始。 我不觉得这两条线在日志中，你表现出来了。 我希望你没有错过它。 通常它应该是这个样子：
apache@hadoop:/hadoop/projects/apache-flume-1.4.0-SNAPSHOT-bin$ bin/flume-ng agent -n agent1 -c /conf -f conf/agent1.conf
Info: Including Hadoop libraries found via (/hadoop/projects/hadoop-1.0.4/bin/hadoop) for HDFS access
Warning: $HADOOP_HOME is deprecated.
Warning: $HADOOP_HOME is deprecated.
Info: Excluding /hadoop/projects/hadoop-1.0.4/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /hadoop/projects/hadoop-1.0.4/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
+ exec /usr/lib/jvm/java-7-oracle/bin/java -Xmx20m -cp '/conf:/hadoop/projects/apache-flume-1.4.0-SNAPSHOT-bin/lib/*:/hadoop/projects/hadoop-1.0.4/libexec/../conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:/hadoop/projects/hadoop-1.0.4/libexec/..:/hadoop/projects/hadoop-1.0.4/libexec/../hadoop-core-1.0.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/asm-3.2.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/aspectjrt-1.6.5.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/aspectjtools-1.6.5.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-beanutils-1.7.0.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-beanutils-core-1.8.0.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-cli-1.2.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-codec-1.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-collections-3.2.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-configuration-1.6.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-daemon-1.0.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-digester-1.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-el-1.0.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-httpclient-3.0.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-io-2.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-lang-2.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-logging-1.1.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-logging-api-1.0.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-math-2.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/commons-net-1.4.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/core-3.1.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/guava-13.0.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/hsqldb-1.8.0.10.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jackson-core-asl-1.8.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jasper-compiler-5.5.12.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jasper-runtime-5.5.12.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jdeb-0.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jersey-core-1.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jersey-json-1.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jersey-server-1.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jets3t-0.6.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jetty-6.1.26.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jetty-util-6.1.26.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jsch-0.1.42.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/junit-4.5.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/kfs-0.2.2.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/log4j-1.2.15.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/mockito-all-1.8.5.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/oro-2.0.8.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/protobuf-java-2.3.0.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/servlet-api-2.5-20081211.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/xmlenc-0.52.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/zookeeper-3.4.3.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-2.1.jar:/hadoop/projects/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=:/hadoop/projects/hadoop-1.0.4/libexec/../lib/native/Linux-amd64-64 org.apache.flume.node.Application -n agent1 -f conf/agent1.conf
12/12/15 02:55:29 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
12/12/15 02:55:29 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:conf/agent1.conf
12/12/15 02:55:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/12/15 02:55:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/12/15 02:55:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/12/15 02:55:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/12/15 02:55:29 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: agent1
12/12/15 02:55:29 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
12/12/15 02:55:29 INFO node.AbstractConfigurationProvider: Creating channels
12/12/15 02:55:29 INFO channel.DefaultChannelFactory: Creating instance of channel MemoryChannel-2 type memory
12/12/15 02:55:29 INFO node.AbstractConfigurationProvider: Created channel MemoryChannel-2
12/12/15 02:55:29 INFO source.DefaultSourceFactory: Creating instance of source tail, type exec
12/12/15 02:55:29 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs
12/12/15 02:55:30 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
12/12/15 02:55:30 INFO node.Application: Starting new configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:tail,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@137efe53 counterGroup:{ name:null counters:{} } }} channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel{name: MemoryChannel-2}} }
12/12/15 02:55:30 INFO node.Application: Starting Channel MemoryChannel-2
12/12/15 02:55:30 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: MemoryChannel-2, registered successfully.
12/12/15 02:55:30 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: MemoryChannel-2 started
12/12/15 02:55:30 INFO node.Application: Starting Sink HDFS
12/12/15 02:55:30 INFO node.Application: Starting Source tail
12/12/15 02:55:30 INFO source.ExecSource: Exec source starting with command:tail -F /var/log/apache2/access.log.1
12/12/15 02:55:30 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully.
12/12/15 02:55:30 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started
看到最后2行。
第二，代理不会将任何数据，直到新的东西来的文件，这里/var/log/apache2/access.log“。 无论是手工的东西复制到该文件，然后重新启动你的Apache和做一些事情，然后检查你的/ HDFS /水槽目录的内容。
因为/var/log/apache2/access.log是没有大到足以让水槽打印文件的线，所以只能尽力thisas下面，你可以找到在控制台输出
for i in {1..100}; do echo "tail log test$i" >> var/log/apache2/access.log;done
我想你可以使用taildir源 ，如果你正在使用水槽1.7.0
以下是我在我的项目中使用：
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /xxx/env/flume/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /xxx/logs/file_name.*.log
a1.sources.r1.headers.f1.headerKey1 = yyy
a1.sources.r1.fileHeader = true
