apache drill bad performance

Question

I tried using apache-drill to run a simple join-aggregate query and the speed wasn't really good. my test query was:

SELECT p.Product_Category, SUM(f.sales)
FROM facts f
JOIN Product p on f.pkey = p.pkey
GROUP BY p.Product_Category

Where facts has about 422,000 rows and product has 600 rows. the grouping comes back with 4 rows.

First I tested this query on SqlServer and got a result back in about 150ms.

With drill I first tried to connect directly to SqlServer and run the query, but that was slow (about 5 sec).

Then I tried saving the tables into json files and reading from them, but that was even slower, so I tried parquet files.

I got the result back in the first run in about 3 sec. next run was about 900ms and then it stabled at about 500ms.

From reading around, this makes no sense and drill should be faster! I tried "REFRESH TABLE METADATA", but the speed didn't change.

I was running this on windows, through the drill command line.

Any idea if I need some extra configuration or something?

Thanks!

Answer 1

Why do you expect Apache Drill to be faster here? Drill is very fast, but it's designed for large distributed queries, potentially across several different data sources... but you're not using distributed data.

SQL Server has decades of code and optimizations that make it one of the fastest relational databases running on a single server. Your data is stored efficiently, cached in memory, and the query runs in a single process so the scan and join will be very fast, especially with such little data.

Apache Drill has much more work to do in comparison. It has to interpret your query into a distributed plan and then send it to all the drillbit processes, which then lookup the data sources, access the data using the connectors, run the query, return the results to the first node for aggregation, and then you have your final output. Depending on the data source, Drill might have to read all the data and filter it separately which adds even more time.

JSON files are slow because they are verbose text files that are parsed line by line. Parquet is much faster because it's a binary compressed column-oriented storage format which is very efficient for scanning.

Any relational database will be faster than Drill on a single-machine. The fact that Drill gets you results in 500ms with Parquet is actually impressive considering how much more work it has to do to give you the flexibility it provides. If you only have a few million rows, stick with SQL server. If you have billions of rows, then use the SQL Server columnstore feature to store data in columnar format with great compression and performance.

Use Apache Drill when you:

• Have (10s of) billions of rows or more
• Have data spread across many machines
• Have unstructured data like JSON stored in files without a standard schema
• Want to split the query across many machines to parallelize it and run faster
• Want to access data from different databases and file systems
• Want to join data across these different data sources

Answer 2

One thing people need to understand about how Drill works is how Drill translates an SQL query to an executable plan to fetch and process data from, theoretically, any source of data. I deliberately didn't say data source so people won't think of databases or any software-based data management system.

Drill uses storage plugins to read records from whatever data the storage plugin supports.

After Drill gets these rows, it starts performing what is needed to execute the query, whats needed may be filtering, sorting, joining, projecting (selecting specific columns)...etc

So drill doesn't by default use any of the source's capabilities of processing the queried data. In fact, the source may not support any capability of such !

If you wish to leverage any of the source's data processing features, you'll have to modify the storage plugin you're using to access this source.

One query I regularly remember when I think about Drill's performance, is this one

Select a.CUST_ID, (Select count(*) From SALES.CUSTOMERS where CUST_ID < a.CUST_ID) rowNum from SALES.CUSTOMERS a Order by CUST_ID

Only because of the > comparison operator, Drill has to load the whole table (i.e actually a parquet file), SORT IT, then perform the join.

This query took around 18 minutes to run on my machine which is a not so powerful machine but still, the effort Drill needs to perform to process this query must not be ignored.

Drill's purpose is not to be fast, it's purpose is to handle vast amounts of data and run SQL queries against structured and semi-structured data. And probably other things that I can't think about at the moment but you may find more information for other answers.