Handling quorum writies fail on Cassandra

Question

According to Datastax documentation about atomicity in Cassandra: QUORUM write succeeded only on one node will not be rolled back (Check Atomicity chapter there:http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html#cassandra/dml/dml_about_transactions_c.html). So when I am performing a QUORUM write on cluster with RF=3 and one node fails, I will get write error status and one successful write on another node. This produces two cases:

1. write will be propagated to other nodes when they became online;
2. write can be completely lost if the node accepted that write will be completely broken before propagation.

What is the best ways to deal with such kind of fails in let say hypothetical funds transfer logging?

Answer 1

When a QUORUM write fails with a "TimedOut" exception, you don't know if the write succeeded or not. You should retry the write, and treat it as if it failed. If you have multiple writes that you need to be grouped together, you should place them in a "batch", so that the batch succeeds or fails together.

In either case, you also want to be doing QUORUM reads if you care about consistent results coming back. If you had an RF=3, and the QUORUM write only got on one node, the first time a QUORUM read succeeds that includes the new value, it will be repaired on one of the other nodes, and QUORUM read will always give the new value. So even if the value is written at ONE, successive QUORUM reads will never see the value go back in time.