Hadoop job fails when invoked by cron

Question

I have created the following shell script for invoking a hadoop job:

#!/bin/bash
/opt/hadoop/bin/hadoop jar /path/to/job.jar com.do.something <param-1> ... <param-n> &
wait %1
STATUS=$?
if [ $STATUS -eq 0 ]
then    
    echo "SUCCESS" | mailx -s "Status: "$STATUS -r "[e-mail]" "[e-mail]"
    exit $STATUS
else
    echo "FAILED" | mailx -s "Status: "$STATUS -r "[e-mail]" "[e-mail]"
    exit $STATUS
fi

When I run the above script manually like this:

$ ./path/to/job.sh

Hadoop job executes sucessfully and returns an exit status "0".

Now, to automate the job execution everyday, I have configured a cron job to run the above script like this:

0 22 * * * /path/to/job.sh

But, now job is not submitted to Hadoop and I get a exit status "1".

Few things to note here:

• The user account under which cron job is configured is UserA
• UserA is also Hadoop System User
• The cluster is dedicated for running this job
• The script is executable

I would like to know why the job is not running when cron invokes it ?

Answer 1

I have also encountered a similar problem. I have used $HOME/.bashrc to set environment variables such as JAVA_HOME, HADOOP_HOME and PATH. I can also run my job.sh manually. But hadoop related commands inside job.sh can't be invoked correctly when job.sh is invoked by cron.

The cause for my problem is that cron will not source $HOME/.bashrc. So environment variables inside it can't be seen by cron. After setting all these environment variables in job.sh, hadoop related commands are invoked correctly by cron.

Answer 2

0 22 * * * /path/to/job.sh

I think you lost the "." in your command.

0 22 * * * ./path/to/job.sh

does it work?

Answer 3

after set JAVA_HOME,HADOOP_HOME in job.sh, then:

0 22 * * * sh -x $HOME/path/to/job.sh > $HOME/job.log 2>&1

see what in your job.log

Answer 4

If you set JAVA_HOME and HADOOP_HOME in your /etc/profile. Add

. /etc/profile

in your job.sh. This will help.

Answer 5

the env of running from cron could be different from your regular shell. You may want to check that, e.g. JAVA_HOME, PATH etc.