How hashing works in bucketing for hive?

问题:

I know the hashing principal for HashMap in Java, so wanted to know that how the hashing works for the Hive while we bucketing the data in various bucket.

回答1:

I recently had to dig into some Hive source code to figure this out for myself. Here's what I found:

For an integer field, the hash is just the integer value. For a string, it uses a similar version of Java's String hashCode. When hashing multiple values, the hash is a similar version of Java’s List hashCode.

回答2:

Bucketing is used along with partitioning to have more decomposed structure for future analysis. As more partitions result in more hdfs files which can affect namenode performance, we resort to bucketing. The way bucketing actually works is : The number of buckets is determined by hashFunction(bucketingColumn) mod numOfBuckets numOfBuckets is chose when you create the table with partitioning. The hash function output depends on the type of the column choosen. To accurately set the number of reducers while bucketing and land the data appropriately, we use "hive.enforce.bucketing = true". Please refer to this, for more information