How does Hive decide when to use map reduce and wh

问题:

As a simple example,

select * from tablename;

DOES NOT kick in map reduce, while

select count(*) from tablename;

DOES. What is the general principle used to decide when to use map reduce (by hive)?

回答1:

In general, any sort of aggregation, such as min/max/count is going to require a MapReduce job. This isn't going to explain everything for you, probably.

Hive, in the style of many RDBMS, has an EXPLAIN keyword that will outline how your Hive query gets translated into MapReduce jobs. Try running explain on both your example queries and see what it is trying to do behind the scenes.

回答2:

Whenever we fire a query like select * from tablename, Hive reads the data file and fetches the entire data without doing any aggregation(min/max/count etc.). It'll call a FetchTask rather than a mapreduce task.

This is also an optimization technique in Hive. hive.fetch.task.conversion property can (i.e. FETCH task) minimize latency of map-reduce overhead.

This is like we are reading a hadoop file : hadoop fs -cat filename

But if we use select colNames from tablename, it requires a map-reduce job as it needs to extract the 'column' from each row by parsing it from the file it loads.

回答3:

select * from tablename;

Just reads raw data from files in HDFS, so it is much faster without MapReduce.

回答4:

It is an optimisation technique, hive.fetch.task.conversion property can (FETCH) task minimize latency of mapreduce overhead.

When doing SELECT, LIMIT, FETCH queries this property skips mapreduce and uses the FETCH task.

This property can have 3 values - none, minimal (the default) and more.