Cassandra Python driver OperationTimedOut issue

I have a python script which is used to interact with cassandra with datastax python driver

It has been running since March 14th, 2016 and had not problem until today.

2016-06-02 13:53:38,362 ERROR ('Unable to connect to any servers', {'172.16.47.155': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})
2016-06-02 13:54:18,362 ERROR ('Unable to connect to any servers', {'172.16.47.155': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})

Below is the function used for creating a session, and shutdown the session (session.shutdown()) every time a query is done.(Every day we only have less than 100 queries from the subscribers side, therefore I chose build connection, do the query and close it instead of leaving the connection alive)

The session is not shared between threads and processes. If i invoke the below function in python console, it connects with the DB properly, but the running script cannot connect to the DB anymore.

Any one can help or shed some light on this issue? Thanks

def get_cassandra_session(stat=None):
    """creates cluster and gets the session base on key space"""
    # be aware that session cannot be shared between threads/processes
    # or it will raise OperationTimedOut Exception
    if config.CLUSTER_HOST2:
        cluster = cassandra.cluster.Cluster([config.CLUSTER_HOST1, config.CLUSTER_HOST2])
    else:
        # if only one address is available, we have to use older protocol version
        cluster = cassandra.cluster.Cluster([config.CLUSTER_HOST1], protocol_version=2)

    if stat and type(stat) == BatchStatement:
        retry_policy = cassandra.cluster.RetryPolicy()
        retry_policy.on_write_timeout(BatchStatement, ConsistencyLevel, WriteType.BATCH_LOG, ConsistencyLevel.ONE,
                                      ConsistencyLevel.ONE, retry_num=0)
        cluster.default_retry_policy = retry_policy
    session = cluster.connect(config.KEY_SPACE)
    session.default_timeout = 30.0
    return session

Specs: python 2.7 Cassandra 2.1.11

Quotes from datastax doc:

The operation took longer than the specified (client-side) timeout to complete. This is not an error generated by Cassandra, only the driver.

The problem is I didn't touch the driver. I set the default timeout to 30.0 seconds but why it timedout in 5 seconds(it is said in the log)

回答1:

The default connect timeout is five seconds. In this case you would need to set Cluster.connect_timeout. The Session default_timeout applies to execution requests.

It's still a bit surprising when any TCP connection takes more than five seconds to establish. One other thing to check would be monkey patching. Did something in the application change patching for Gevent or Eventlet? That could cause a change in default behavior for the driver.

回答2:

I've learned that the gevent module interferes with the cassandra-driver