Running Hadoop commands in a shell script

Question

I have a simple shell script that I'd like to use to move log files into my Hadoop cluster.

When I invoke the the shell script in my Java program, the script can not be found "Hadoop Commands", and I do not take any action.

For instance, I would like to run the following command:

#!/bin/bash
hadoop fs -put /home/can/workspace2/myWebApp/tweets.txt /user/training/tweetFolder

How can I solve this in a simple manner?

Answer 1

I believe it's a PATH problem. check the path for hadoop with which:

which hadoop

Then put the full path on your script.

Answer 2

If you have installed Hadoop using some executables like CDH, HDP etc, it should work. But if it's a setup through Apache's tarball you need to use the complete path along with bin/, as bin directory contains all the executable files.

HADOOP_HOME/bin/hadoop fs -put /home/can/workspace2/myWebApp/tweets.txt /user/training/tweetFolder

Also, it's always a good practice to set the HADOOP_HOME variable.