Read timed out on SonarQube analysis

We previously discussed the problem on the now closed SonarQube Users mailing list.

The problem was solved by exchanging the for about half a week by optimizing the Postgre DB, then the problem reoccured.

We are using Jenkins 1.612 using the SonarQube Jenkins Plugin 2.2.1, Sonar 5.1 using a Postgre 9.1 database.

We are running about 20 JAVA projects/branches using much legacy code and having about 1.2 million LOC and 130k issues.
There are about 30 JAVA projects module projects having 1k-20k LOC and less than 2k issues.
There are about 50 custom JAVA projects having less than 200k LOC and less than 10k issues.
Our projects are written in JAVA (6/7) and using ANT 50% or Gradle 50% as build tools.

Now we are getting this error again:

Gestartet durch vorgelagertes Projekt "Project__Branch__ant__dbTest", Build 86
originally caused by:
 Gestartet durch vorgelagertes Projekt "Project__Branch__ant__build", Build 101
 originally caused by:
  Build wurde durch eine SCM-Änderung ausgelöst.
Baue auf Slave SonarQube (Sonar-Analyse) in Arbeitsbereich D:\Jenkins (prod-test)\workspace\Project__Branch__ant__sonar
Restoring workspace from build #101 of project Project__Branch__ant__build
[Project__Branch__ant__sonar] $ "D:\Jenkins (prod-test)\tools\hudson.plugins.sonar.SonarRunnerInstallation\sonar-runner-2.4\bin\sonar-runner.bat" -e -Dsonar.jdbc.url=jdbc:postgresql://localhost:5432/postgres ******** ******** -Dsonar.host.url=http://sonarqube:9000 ******** ******** "-Dsonar.projectBaseDir=D:\Jenkins (prod-test)\workspace\Project__Branch__ant__sonar" -Dsonar.branch=Branch
D:\Jenkins (prod-test)\tools\hudson.plugins.sonar.SonarRunnerInstallation\sonar-runner-2.4
SonarQube Runner 2.4
Java 1.7.0_45 Oracle Corporation (64-bit)
Windows Server 2008 R2 6.1 amd64
SONAR_RUNNER_OPTS=-Xmx2048m -XX:MaxPermSize=256m
INFO: Error stacktraces are turned on.
INFO: Runner configuration file: D:\Jenkins (prod-test)\tools\hudson.plugins.sonar.SonarRunnerInstallation\sonar-runner-2.4\conf\sonar-runner.properties
INFO: Project configuration file: D:\Jenkins (prod-test)\workspace\Project__Branch__ant__sonar\sonar-project.properties
INFO: Default locale: "de_DE", source code encoding: "windows-1252" (analysis is platform dependent)
INFO: Work directory: D:\Jenkins (prod-test)\workspace\Project__Branch__ant__sonar\.sonar
INFO: SonarQube Server 5.1
02:22:57.131 INFO  - Load global repositories
02:22:59.624 INFO  - Load global repositories (done) | time=2533ms
02:22:59.642 INFO  - Server id: 20150601083800
02:22:59.650 INFO  - User cache: C:\.sonar\cache
02:22:59.768 INFO  - Install plugins
02:23:05.720 INFO  - Install JDBC driver
02:23:05.781 INFO  - Create JDBC datasource for jdbc:postgresql://localhost:5432/postgres
02:23:14.096 INFO  - Initializing Hibernate
02:23:28.270 INFO  - Load project repositories
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 1:05.593s
Final Memory: 38M/269M
INFO: ------------------------------------------------------------------------
ERROR: Error during Sonar runner execution
org.sonar.runner.impl.RunnerException: Unable to execute Sonar
    at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:91)
    at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)
    at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)
    at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:102)
    at org.sonar.runner.api.Runner.execute(Runner.java:100)
    at org.sonar.runner.Main.executeTask(Main.java:70)
    at org.sonar.runner.Main.execute(Main.java:59)
    at org.sonar.runner.Main.main(Main.java:53)
Caused by: java.lang.IllegalStateException: Unable to request: /batch/project?key=Project%3ABranch&preview=false
    at org.sonar.batch.bootstrap.ServerClient.request(ServerClient.java:109)
    at org.sonar.batch.bootstrap.ServerClient.request(ServerClient.java:99)
    at org.sonar.batch.bootstrap.ServerClient.request(ServerClient.java:87)
    at org.sonar.batch.repository.DefaultProjectRepositoriesLoader.load(DefaultProjectRepositoriesLoader.java:55)
    at org.sonar.batch.repository.ProjectRepositoriesProvider.provide(ProjectRepositoriesProvider.java:40)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.picocontainer.injectors.MethodInjector.invokeMethod(MethodInjector.java:129)
    at org.picocontainer.injectors.MethodInjector.access$000(MethodInjector.java:39)
    at org.picocontainer.injectors.MethodInjector$2.run(MethodInjector.java:113)
    at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
    at org.picocontainer.injectors.MethodInjector.decorateComponentInstance(MethodInjector.java:120)
    at org.picocontainer.injectors.CompositeInjector.decorateComponentInstance(CompositeInjector.java:58)
    at org.picocontainer.injectors.Reinjector.reinject(Reinjector.java:142)
    at org.picocontainer.injectors.ProviderAdapter.getComponentInstance(ProviderAdapter.java:96)
    at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
    at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
    at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
    at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
    at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
    at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
    at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
    at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
    at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
    at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
    at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.getComponentInstance(AbstractInjectionFactory.java:56)
    at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
    at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
    at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
    at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
    at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:677)
    at org.sonar.api.platform.ComponentContainer.getComponentByType(ComponentContainer.java:209)
    at org.sonar.batch.scan.ProjectScanContainer.doBeforeStart(ProjectScanContainer.java:92)
    at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:90)
    at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
    at org.sonar.batch.scan.ScanTask.scan(ScanTask.java:57)
    at org.sonar.batch.scan.ScanTask.execute(ScanTask.java:45)
    at org.sonar.batch.bootstrap.TaskContainer.doAfterStart(TaskContainer.java:135)
    at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
    at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
    at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:158)
    at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:95)
    at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:67)
    at org.sonar.runner.batch.IsolatedLauncher.execute(IsolatedLauncher.java:48)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:87)
    ... 9 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1675)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1673)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1671)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1244)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at org.sonar.api.utils.HttpDownloader$BaseHttpDownloader$HttpInputSupplier.getInput(HttpDownloader.java:298)
    at org.sonar.api.utils.HttpDownloader$BaseHttpDownloader$HttpInputSupplier.getInput(HttpDownloader.java:255)
    at org.sonar.batch.bootstrap.ServerClient.request(ServerClient.java:105)
    ... 59 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
    at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2678)
    at java.net.URLConnection.getContentEncoding(URLConnection.java:533)
    at org.sonar.api.utils.HttpDownloader$BaseHttpDownloader$HttpInputSupplier.getInput(HttpDownloader.java:296)
    ... 61 more
ERROR: 
ERROR: Re-run SonarQube Runner using the -X switch to enable full debug logging.
Build step 'Invoke Standalone SonarQube Analysis' marked build as failure
Sending e-mails to: ****@****.**
Notifying upstream projects of job completion
Finished: FAILURE

I hope you may help finding a solution for this problem. Thx.

Edit: Operability restored: Found a workaround if triggered from Jenkins.

At this link there was a similar issue for an older Sonar version. They triggered regularly the URL to keep the query in DB cache. This works for this issue also.

In Jenkins I installed the http Request plugin and before issuing the sonar analysis I trigger a http get on http://mySonarQube:9000/batch/project?key=myProjectKey&preview=false . This works around the real problem (DB request duration and timeout interval don't fit), but it may keep us operable.

In the next few days Sonar 5.2 will occur. We will test this version and I hope there the issues will be gone... I'll report...

https://jira.sonarsource.com/projects/SONAR/versions/11629

The problem can be solved in 4 ways:

add more resources to SonarQube server process and/or server platform (e.g. faster disks)
upgrade SonarQube server to the newest version (e.g. a lot of performance problems have been fixed in SonarQube 5.2 - see release notes)
change timeout value in sources and compile custom server
add indexes on columns in database (not recommended, you can read executed SQL commands from logs in debug mode)