How to Fix Read timed out in Elasticsearch

Question

I used Elasticsearch-1.1.0 to index tweets. The indexing process is okay. Then I upgraded the version. Now I use Elasticsearch-1.3.2, and I get this message randomly:

Exception happened: Error raised when there was an exception while talking to ES.
ConnectionError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)).

Snapshot of the randomness:

Happened --33s-- Happened --27s-- Happened --22s-- Happened --10s-- Happened --39s-- Happened --25s-- Happened --36s-- Happened --38s-- Happened --19s-- Happened --09s-- Happened --33s-- Happened --16s-- Happened 

--XXs-- = after XX seconds

Can someone point out on how to fix the Read timed out problem?

Thank you very much.

Answer 1

Try this:

es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)

It might won't fully avoid ReadTimeoutError, but it minimalize them.

Answer 2

For what it's worth, I found that this seems to be related to a broken index state.

It's very difficult to reliably recreate this issue, but I've seen it several times; operations run as normal except certain ones which periodically seem to hang ES (specifically refreshing an index it seems).

Deleting an index (curl -XDELETE http://localhost:9200/foo) and reindexing from scratch fixed this for me.

I recommend periodically clearing and reindexing if you see this behaviour.

Answer 3

You also should check if all fine with elastic. Some shard can be unavailable, here is nice doc about possible reasons of unavailable shard https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/

Answer 4

Its hard to give a direct answer since the error your seeing might be associated with the client you are using. However a solution might be one of the following:

1.Increase the default timeout Globally when you create the ES client by passing the timeout parameter. Example in Python

es = Elasticsearch(timeout=30)

2.Set the timeout per request made by the client. Taken from Elasticsearch Python docs below.

# only wait for 1 second, regardless of the client's default
es.cluster.health(wait_for_status='yellow', request_timeout=1)

The above will give the cluster some extra time to respond

Answer 5

Read timeouts can also happen when query size is large. For example, in my case of a pretty large ES index size (> 3M documents), doing a search for a query with 30 words took around 2 seconds, while doing a search for a query with 400 words took over 18 seconds. So for a sufficiently large query even timeout=30 won't save you. An easy solution is to crop the query to the size that can be answered below the timeout.

Answer 6

Increasing various timeout options may immediately resolve issues, but does not address the root cause.

Provided the ElasticSearch service is available and the indexes are healthy, try increasing the the Java minimum and maximum heap sizes: see https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html .

TL;DR Edit /etc/elasticsearch/jvm.options -Xms1g and -Xmx1g