Hadoop Maven Dependency Error

Question

I am trying to build a Hadoop job using Maven. This job works well when I don't use Maven and directly import the Hadoop Jar dependency into eclipse.

I'm also able to build a simple jar (hello world type) using Maven without the Hadoop dependency.

But when I add the Hadoop dependency, and run the jar, I get this error:

dataException in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
    at org.graphhadoop.CreateAdjacency.Adjacency(CreateAdjacency.java:74)
    at org.graphhadoop.Driver.main(Driver.java:12)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 2 more

The pom file changes are these:

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
        </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.203.0</version>
        </dependency>
  </dependencies>

<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <mainClass>org.graphhadoop.Driver</mainClass>
            </manifest>
          </archive>
          <manifestEntries>
          <Class-Path>../target/classes</Class-Path>
        </manifestEntries>
        </configuration>
      </plugin>
    </plugins>
  </build>

Where am I going wrong?

Edit:

For now, I am running the Jar locally - not on Hadoop, but on local data.

java -jar jarname.jar

Answer 1

Just as a solution to somebody encountering the same problem, here's what resolved it:

These dependencies need to be added to pom.xml:

        <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.5.2</version>
        </dependency>

        <dependency>
        <groupId>net.sf.json-lib</groupId>
        <artifactId>json-lib</artifactId>
        <version>2.3</version>
        <classifier>jdk15</classifier>
        </dependency>

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.203.0</version>
        </dependency>
  </dependencies>

and these build plugins:

<build>
    <plugins>
    <plugin>
     <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-assembly-plugin</artifactId>
        <executions>
        <execution>
         <goals>
           <goal>attached</goal>
         </goals>
         <phase>package</phase>
         <configuration>
           <descriptorRefs>
             <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <mainClass>org.lll..</mainClass>
            </manifest>
      <manifestEntries>
          <Class-Path>../target/classes</Class-Path>
            </manifestEntries>
          </archive>
        </configuration>
        </execution>
    </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
          <mainClass>org...</mainClass>
            </manifest>
          </archive>
      <manifestEntries>
          <Class-Path>../target/classes</Class-Path>
        </manifestEntries>
        </configuration>
      </plugin>
    </plugins>  
</build>