Hadoop MapReduce Error - /bin/bash: /bin/java: is

Question

I am attempting to run a basic MapReduce program on macOS 10.12 that retrieves the maximum temperature from a log file of weather data. When running the job, I receive the following stack trace:

Stack trace: ExitCodeException exitCode=126:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
    at org.apache.hadoop.util.Shell.run(Shell.java:479)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

The stderr log file for the job in the resource manager contains the following message: "/bin/bash: /bin/java: is a directory".

I was originally receiving a similar error message, "/bin/bash: /bin/java: No such file or directory", but modified the hadoop-config.sh script as suggested by the answer in this post. I modified it as follows:

if [[ -z $JAVA_HOME ]]; then
# On OSX use java_home (or /Library for older versions)
 if [ "Darwin" == "$(uname -s)" ]; then
  if [ -x /usr/libexec/java_home ]; then
   export JAVA_HOME=${JAVA_HOME}
  else
   export JAVA_HOME=${JAVA_HOME}
  fi
 fi

My $JAVA_HOME variable is set to: /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

Is this a result of a configuration issue with my JAVA_HOME variable?

Answer 1

This was resolved by hard-coding the JAVA_HOME variable in hadoop-env.sh (as specified in the highest voted answer to this question)

Changed this:

export JAVA_HOME=${JAVA_HOME}

To this:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

Answer 2

I had my own run-in with this issue, and wanted to throw in my own experience. In my case, I was getting /bin/bash: /bin/java: No such file or directory when trying to run the example detailed on Apache's website.

I figured a good place to start looking how to fix this issue would be to look through all the files to see where the problem might be cropping up. grep -r '/bin/java' ./*

In addition to the various declarations of JAVA=$JAVA_HOME/bin/java, I saw _RUNJAVA=$JRE_ENV/bin/java in $HADOOP_CONF_DIR/share/hadoop/httpfs/tomcat/bin/setclasspath.sh. I did not have this environment variable set, so I simply stopped yarn, set it to equal to $JAVA_HOME, and started it back up.

Answer 3

Or you can try the commands sequence presented in this comment:

hdfs namenode -format;
start-dfs.sh;
yarn-daemon.sh start resourcemanager;
yarn-daemon.sh start nodemanager;

You should not start YARN via the "start-yarn.sh" in pseudo-distributed mode. The issue is with the way "start-yarn.sh" works (though this is fixed in upcoming releases).