Where is the classpath set for hadoop

Question

Where is the classpath for hadoop set? When I run the below command it gives me the classpath. Where is the classpath set?

  bin/hadoop classpath

I'm using hadoop 2.6.0

Answer 1

Open your bash profile (~/.profile or ~/.bash_profile) for editing and add the following:

1. export HADOOP_HOME="/usr/local/Cellar/hadoop" then Replace with your own path

2. export HADOOP_CLASSPATH=$(find $HADOOP_HOME -name '*.jar' | xargs echo | tr ' ' ':') Save the changes and reload.

3. source ~/.profile

Answer 2

As said by almas shaikh it's set in hadoop-config.sh, but you could add more jars to it in hadoop-env.sh

Here is a relevant code from hadoop-env.sh which adds additional jars like capacity-scheduler and aws jar's.

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

# ... some other lines omitted

# Add Aws jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:share/hadoop/tools/lib/*

Answer 3

As per this blog post, it is in an environment variable named HADOOP_CLASSPATH. You can set it as you would any other environment variable, the specifics of which depend on which shell you use. If you use bash, then you can call like export HADOOP_CLASSPATH=/path/to/wherever:/path/to/wherever/else.

Answer 4

When you run hadoop command, it sources a file hadoop-config.sh that resides in $HADOOP_HDFS_HOME/libexec which sets your classpath (CLASSPATH) by picking jars residing in various directories viz.

$HADOOP_HDFS_HOME/share/hadoop/mapreduce 
$HADOOP_HDFS_HOME/share/hadoop/common
$HADOOP_HDFS_HOME/share/hadoop/hdfs etc.