Creating Hive table on top of multiple parquet fil

问题:

We have our dataset in s3 (parquet files) in the below format, data divided as multiple parquet files based on the row number.

data1_1000000.parquet
data1000001_2000000.parquet
data2000001_3000000.parquet
...

We have more than 2000 such files and each file has million records on it. All these files have same number of columns and structure. And one of the column has timestamp in it if we need to partition the dataset in hive. How can we point the dataset and create a single hive external table on top of it for our analysis or may be use spark for analysing it?

Thanks.

回答1:

You can simply point to the path containing the files:

CREATE EXTERNAL TABLE parquet_hive (
  foo string
) STORED AS PARQUET
LOCATION 's3://myBucket/myParquet/';