Intermittent connection timeouts to Solr server us

Question

I have a production webserver hosting a search, and another machine which hosts the Solr search server (on a subnet which is in the same room, so no network problems). All is fine >90% of the time, but I consistently get a small number of The operation has timed out errors.

I've increased the timeout in the SolrNet init to 30 seconds (!)

SolrNet.Startup.Init<SolrDataObject>(
    new SolrNet.Impl.SolrConnection(
        System.Configuration.ConfigurationManager.AppSettings["URL"]
    ) {Timeout = 30000}
);

...but all that happened is I started getting this message instead of Unable to connect to the remote server which I was seeing before. It seems to have made no difference to the amount of timeout errors.

I can see nothing in any log (believe me I've looked!) and clearly my configuration is correct because it works most of the time. Anyone any ideas how I can find more information on this problem?

EDIT:

I have now increased the number of HttpRequest connections from 2 to 'a large number' (I see up to 10 connections) - but this has had no discernible effect on this problem.

The firewall is set to allow ANY connections between the two machines.

We've also checked the hardware with our server host and there are no problems on the connections, according to them.

EDIT 2:

We're still seeing this issue.

We're now logging the timeouts and they're mostly just over 30s - which is the SolrNet layer's timeout; some are 20s, though - which is the Tomcat default timeout period - which suggests it's something in the actual connection between the machines.

Not sure where to go from here, though - they're on a VLAN and we're specifically using the VLAN address - response time from pings is ALWAYS <1ms.

Answer 1

Try setting this in .net.

ServicePointManager.Expect100Continue = false;

or this

ServicePointManager.SetTcpKeepAlive(true, 200000, 200000); - this sends requests to the server to keep the connection alive.

Answer 2

Without more information, I can only guess a few possible reasons:

• You're fetching tons of documents per query, and it times out while transferring data.
• You're hitting the ServicePoint.ConnectionLimit. If so, just increase this value. See also How can I programmatically remove the 2 connection limit in WebClient
• You have some very facet-heavy requests or misusing Solr (e.g. not using filter queries). Check the qtime in the response. See the Solr performance wiki for more details.