Hive Does the order of the data record matters for

问题:

I would like to know if the order of the data records matter (performance wise) when joining two tables?
P.S. I am not using any map-side join or bucket join.

Thank you!

回答1:

On the one hand order should not matter because during shuffle join files are being read by mappers in parallel, also files may be splitted between few mappers or vice-versa, one mapper can read few files, then mappers output passed to each reducer. And even if data was ordered it is being read and distributed not in it's order due to parallelism. On the other hand, ordering data may improve compression depending on the data entropy. Similar rows can be compressed better. Therefore files ordered compressed files can be smaller and they will be read faster during join query execution. This may improve join speed because mappers will read data faster. Also indexes in ORC may work more efficient for filtering if data was ordered during load. It depends on your data entropy and filters you are using.