Hadoop job fails when invoked by cron

问题:

I have created the following shell script for invoking a hadoop job:

#!/bin/bash
/opt/hadoop/bin/hadoop jar /path/to/job.jar com.do.something <param-1> ... <param-n> &
wait %1
STATUS=$?
if [ $STATUS -eq 0 ]
then    
    echo "SUCCESS" | mailx -s "Status: "$STATUS -r "[e-mail]" "[e-mail]"
    exit $STATUS
else
    echo "FAILED" | mailx -s "Status: "$STATUS -r "[e-mail]" "[e-mail]"
    exit $STATUS
fi

When I run the above script manually like this:

$ ./path/to/job.sh

Hadoop job executes sucessfully and returns an exit status "0".

Now, to automate the job execution everyday, I have configured a cron job to run the above script like this:

0 22 * * * /path/to/job.sh

But, now job is not submitted to Hadoop and I get a exit status "1".

Few things to note here:

• The user account under which cron job is configured is UserA
• UserA is also Hadoop System User
• The cluster is dedicated for running this job
• The script is executable

I would like to know why the job is not running when cron invokes it ?

回答1:

0 22 * * * /path/to/job.sh

I think you lost the "." in your command.

0 22 * * * ./path/to/job.sh

does it work?

回答2:

the env of running from cron could be different from your regular shell. You may want to check that, e.g. JAVA_HOME, PATH etc.

回答3:

I have also encountered a similar problem. I have used $HOME/.bashrc to set environment variables such as JAVA_HOME, HADOOP_HOME and PATH. I can also run my job.sh manually. But hadoop related commands inside job.sh can't be invoked correctly when job.sh is invoked by cron.

The cause for my problem is that cron will not source $HOME/.bashrc. So environment variables inside it can't be seen by cron. After setting all these environment variables in job.sh, hadoop related commands are invoked correctly by cron.

回答4:

If you set JAVA_HOME and HADOOP_HOME in your /etc/profile. Add

. /etc/profile

in your job.sh. This will help.

回答5:

after set JAVA_HOME,HADOOP_HOME in job.sh, then:

0 22 * * * sh -x $HOME/path/to/job.sh > $HOME/job.log 2>&1

see what in your job.log