Using map/reduce java classes in hive queries

Question

I read like, in hive queries we can use map/reduce scripts.

Can we use java map/reduce classes in Hive queries. If so, can you please provide me the sample.

Thanks MRK

Answer 1

You can start writing you custom Mapper and Reducer following the Link provided by @Moiz Arafat.

After that make a jar out of it (say myjar.jar).

Working of such custom Mapper and Reducer require hive-contrib.jar, which can be found in the lib folder of your hive installation.

Before running a query which uses these Mapper/Reducer, you should add these two jars to hive.

  hive> add file path/to/hive-contrib.jar;
  hive> add file path/to/myjar.jar;

Please use add file and NOT add jar.

Now you can write your query like :

    hive> FROM (
    FROM src
    MAP inkey, inval
    USING 'java -cp myjar.jar:hive-contrib.jar com.myco.mymr.MyMapper'
    AS k, v
    CLUSTER BY k) mapout
    REDUCE k, v
    USING 'java -cp myjar.jar:hive-contrib.jar com.myco.mymr.MyReducer'

Answer 2

Assume you have created your "MyMapper" and "MyReducer" classes and put them in a "mymr.jar" jar file. You can use them through streaming as such:

FROM (
FROM src
MAP inkey, inval
USING 'java -cp myjar.jar
com.myco.mymr.MyMapper'
AS k, v
CLUSTER BY k) mapout
REDUCE k, v
USING 'java -cp myjar.jar
com.myco.mymr.MyReducer'

Answer 3

Refer the url . It should help . Link