loading native libraries in hadoop reducer?

Question

I have a native library I need to load for my reduce method and I added it to the distributed cache but when I call System.loadLibrary(mylib.so) in my map method I get an error and failed map task:

Error: no mylib.so in java.library.path

Even though I added it to the distributed cache. Am I missing a step? In my job configuration I call:

DistributedCache.addCacheFile(uri, job.getConfiguration());

Where the uri is the path to mylib.so on the hadoop file system.

This native library depends on many others, all existing on my /hadoop/fs/mystuff/libs on the hadoop fs. I add them all to the distributed cache and I even tried loading all of them in my reduce task using the System.loadLibrary() call. But I keep getting that same java.library.path error. I also tried adding the libraries as command line arguments to the -files flag but I still get the error above.

Answer 1

Though exactly not sure why this is happening, check the How to Include Third-Party Libraries in Your Map-Reduce Job blog entry from Cloudera.

Answer 2

Do you try our code standalone, not in map-reduce? What I know, System.loadLibrary expect library name without ".so" or ".dll" suffix...