NiFi moveHDFS processor appears to do nothing

Question

Simply trying to automate a move of files from one HDFS dir to another. Doing this with a moveHDFS processor in Apache NiFi, but when starting the processor nothing seems to happen.

The processor metrics remain at zero after long amount of time and looking at the bulletin board shows no errors (logging level set to INFO), the only logging output in the bulletin board is:

14:50:04 HSTINFO1e637d0d-0163-1000-7bde-a7993ae403e8
MoveHDFS[id=1e637d0d-0163-1000-7bde-a7993ae403e8] Initialized a new HDFS File System with working dir: file:/home/mapr/nifi-1.6.0 default block size: 33554432 default replication: 1 config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml, /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hdfs-site.xml

(By the way, why does it say "Initialized a new HDFS File System"? Shouldn't it be using the provided hadoop config files to access the existing HDFS on the hadoop cluster?). Have NiFi installed as a single instance on a hadoop cluster node and added the core- and hdfs-site.xml files to the processor properties as needed. Have never used NiFi before and suspect that I am missing something obvious about properly configuring the flow here. Any advice about what is going on here and how to fix it would be appreciated. Thanks.

Answer 1

If you can share more of your flow, that would be helpful. The MoveHDFS processor requires incoming flowfiles in order to process them, so whatever processor is generating or receiving flowfiles needs to pass them on to this processor. Does the queue show any queued flowfiles? The processor shows that no flowfiles have been read in within the last five minutes. You can use ListHDFS or GenerateFlowFile to create the flowfiles that this processor expects.

If that’s not the issue, you can also check the processor schedule to make sure it is set to run frequently (it should be event driven, but if the timer is set and long for some reason, that could be causing this behavior).

As for the new HDFS file system, did you provide the (correct and fully-pathed) core-site.xml and hdfs-site.xml files in the MoveHDFS processor properties? If existing files are not found, it will create default ones. There is full documentation available at the Apache NiFi site.