Cassandra write benchmark, low (20%) CPU usage

问题:

I'm building Cassandra 3x m1.large cluster on Amazon EC2. I've used DataStax Auto-Clustering AMI 2.5.1-pv, with Cassandra DataStax Community version 2.2.0-1.

When doing write benchmarks, on 'production' data, it seems that cluster can handle around 3k to 5k write requests per second, without read load. Nearly all the time nodes do:

• Compaction of system.hints
• Compaction of mykeyspace.mybigtable
• Compaction of mybigtable index

However, what worries me is the low CPU usage. All of the 3 nodes have CPU usage ranging from 17% to 24% percents. Isn't the CPU usage too low? Isn't that limiting my write speed? It could be 100% for me.

BTW. How can I check what's limiting (CPU, memory, network, discs) my write performance?

Here are some stats:

Edit:

• I'm inserting data nicely distributed around cluster
• I'm using Consistency level of One

回答1:

First of all, CPU isn't 20%. While the CPU system is at 20%, the user CPU is at around 70%. Here is the explanation between user CPU and system CPU: User CPU time vs System CPU time?

Secondly, iostat invoked without arguments isn't the best way of looking at disc usage. From: Basic I/O Monitoring on Linux

Without a specified interval, iostat displays statistics since the system was up then exits, which is not useful in our case.

For more comprehensive look at the system, use

  dstat -rcdgilmnps 60

Now we see clearly the averages from the last minute. CPU idle is at 1-4%, we have ~340 ios with 15M write speed.

Next usefull tool is nodetool cfstats:

Where we can see some stats for particular table. Write latency statistics are particularly interesting and equals to 1.5ms.

Finally, executing trace for write:

id: 12345 -> host NodeAsked:9042, achieved consistency: LocalOne
Sending MUTATION message to /NodeA on NodeAsked[MessagingService-Outgoing-/NodeA] at 0
Sending MUTATION message to /NodeB on NodeAsked[MessagingService-Outgoing-/NodeB] at 0
REQUEST_RESPONSE message received from /NodeA on NodeAsked[MessagingService-Incoming-/NodeA] at 0
Processing response from /NodeA on NodeAsked[SharedPool-Worker-32] at 0
MUTATION message received from /NodeAsked on NodeA[MessagingService-Incoming-/NodeAsked] at 12
Determining replicas for mutation on NodeAsked[SharedPool-Worker-45] at 114
Appending to commitlog on NodeAsked[SharedPool-Worker-45] at 183
Adding to mytable memtable on NodeAsked[SharedPool-Worker-45] at 241
Appending to commitlog on NodeA[SharedPool-Worker-5] at 5360
Adding to mytable memtable on NodeA[SharedPool-Worker-5] at 5437
Enqueuing response to /NodeAsked on NodeA[SharedPool-Worker-5] at 5527
Sending REQUEST_RESPONSE message to /NodeAsked on NodeA[MessagingService-Outgoing-/NodeAsked] at 5739

Shows that what's limiting us is the storage speed. It's the best to execute several spontaneous writes with enabled tracing on normal write load to see some patterns.

Vote up if you agree.

回答2:

Is the application your using to benchmark available (open source) anywhere? Theres a chance if your application is doing something like serially sending requests that your throughput could be bottlenecked on latency (littles law) over the actual limits of your cluster. CPU should be the limiting factor in write performance so 20% does have the single threaded application look to it.

Theres a tool cassandra-stress that can mimick most kinds of loads which will fully utilize your client.

回答3:

It is a matter of consistency. When you insert data, and consistency level is Quorum in your case, the driver waits until all nodes respond that the data is available, while inserting, do a Consistency of One, which will give you better performance. As for the performance of compaction please see following article: http://www.datastax.com/dev/blog/ec2-series-doc

Another reason why your writes are performing badly might be table design. If you do not set proper partition keys (depending on your data) then you might get long rows, which in general take longer when being compacted. If you want you can provide your table model (schema) and a sample of your data so this question might be answered in more detail.

Also bear in mind that C* was designed to run on commodity hardware. It rarely uses system resources to the full extent of i.e. available processor power. Cassandra can however - on reads - leverage as much memory as you feed it! As for the write throughput, there is a tool called CCM (https://github.com/pcmanus/ccm) which can benchmark your installation...