channel lock error while configuring flume's m

问题:

Configuring multiple sources for an agent throwing me lock error using FILE channel. Below is my config file.

a1.sources = r1 r2
a1.sinks = k1 k2
a1.channels = c1 c3

#sources
a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=4444

a1.sources.r2.type=exec
a1.sources.r2.command=tail -f /opt/gen_logs/logs/access.log

#sinks
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=/flume201
a1.sinks.k1.hdfs.filePrefix=netcat-
a1.sinks.k1.rollInterval=100
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.callTimeout=100000


a1.sinks.k2.type=hdfs
a1.sinks.k2.hdfs.path=/flume202
a1.sinks.k2.hdfs.filePefix=execCommand-
a1.sinks.k2.rollInterval=100
a1.sinks.k2.hdfs.fileType=DataStream
a1.sinks.k2.hdfs.callTimeOut=100000

#channels
a1.channels.c1.type=file
a1.channels.c1.checkpointDir=/home/cloudera/alpha/001
a1.channels.c3.type=file
a1.channels.c3.checkpointDir=/home/cloudera/beta/001

#bind r1 c1 k1
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

a1.sources.r2.channels=c3
a1.sinks.k2.channel=c3

I am getting below error

Channel closed [channel=c3]. Due to java.io.IOException: Cannot lock /home/cloudera/.flume/file-channel/data. The directory is already locked. [channel=c3]

But when i am using memory channel. Its working fine.

回答1:

My gues is that you need to specify different dataDirs property for each channel, cause both of them use the default value

回答2:

From the docs

By default the File Channel uses paths for checkpoint and data directories that are within the user home as specified above. As a result if you have more than one File Channel instances active within the agent, only one will be able to lock the directories and cause the other channel initialization to fail. It is therefore necessary that you provide explicit paths to all the configured channels, preferably on different disks.

The configuration for the channels should be,

#channels
a1.channels.c1.type=file
a1.channels.c1.checkpointDir=/home/cloudera/alpha/001
a1.channels.c1.dataDirs=/mnt/alpha_data/

a1.channels.c3.type=file
a1.channels.c3.checkpointDir=/home/cloudera/beta/001
a1.channels.c3.dataDirs=/mnt/beta_data/

The dataDirs property defaults to /home/user/.flume/file-channel/data path.