Could not get a resource from the pool(SocketTimeo

Question

I'm running multiple worker threads(around 10) to access the data from the redis Q.
For the i'm using infinte timeout for Jedis Client.

Jedis jedis = pool.getResource();
jedis.getClient().setTimeoutInfinite();  

Still i'm getting the error "Could not get a resource from the pool". The stacktrace is given below.

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:22)
at Workers.Worker1.met1(Worker1.java:124)
at Workers.Worker1.work(Worker1.java:108)
at org.gearman.impl.worker.WorkerConnectionController$3.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)  

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:124)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:54)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1657)
at redis.clients.jedis.JedisPool$JedisFactory.makeObject(JedisPool.java:63)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at redis.clients.util.Pool.getResource(Pool.java:20)
... 6 more  

Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at redis.clients.jedis.Connection.connect(Connection.java:119)
... 11 more

Answer 1

Based on Rick Hanlon's answer, this exception is also thrown if using Redis with Spring Boot.

If you're using Spring Boot, just the dependency of Redis is not enough; you also need to manually download and install Redis on your machine from redis.io, then run it off of a Bash terminal:

me@my_pc:/path/to/redis/dir$ ./src/redis-server ./redis.conf

After running the server, you'll need to add the relevant lines in all your apps that use Redis:

application.properties:

...
spring.redis.host: <yourhost> // usually localhost, but can also be on a LAN
spring.redis.port: <yourport> // usually 6379, but settable in redis.conf

application.yml:

...
spring:
  redis:
    host: <yourhost> // usually localhost, but can also be on a LAN
    port: <yourport> // usually 6379, but settable in redis.conf

Answer 2

Possible causes;

1 - Redis server is down or Redis application not responding.

2 - Application can not connect to Redis server (firewall etc. issues).

3 - Connection to Redis server timed out.

4 - All connections in the (Redis) pool are currently busy, new connection can not be allocated.

The cases 1 and 2 are infra related.

For case 3, connection timeout must be increased ("RedisConnectionTimeout"):

pool = new pool(poolConfig, RedisIp, RedisPort, RedisConnectionTimeout);

For case 4, max connection count must be increased ("RedisMaximumActiveConnectionCount"):

poolConfig.setMaxTotal(RedisMaximumActiveConnectionCount); 

Assuming the following or similar implementation;

private Pool<Jedis> pool =  null;   

private final String RedisIp="10.10.10.11";
private final int RedisPort=6379;
private final String RedisConnectionTimeout=2000;
private final String RedisMaximumWaitTime=1000;
private final String RedisMaximumIdleConnectionCount=20;
private final String RedisMaximumActiveConnectionCount=300;
private final String SentinelActive=false;
private final String SentinelHostList="10.10.10.10:26379,10.10.10.10:26380,10.10.10.10:26381";
private final String SentinelMasterName="sentinel-master-name";

private synchronized void initializePool()
{
    if(pool!=null) return;

    poolConfig poolConfig = new poolConfig();
    poolConfig.setMaxTotal(RedisMaximumActiveConnectionCount); 
    poolConfig.setMaxIdle(RedisMaximumIdleConnectionCount);  
    poolConfig.setMaxWaitMillis(RedisMaximumWaitTime); 

    if(SentinelActive)
    {
        String [] sentinelsArray = SentinelHostList.split(",");

        Set<String> sentinels = new HashSet<>();            
        for(String sentinel : sentinelsArray)
        {
            sentinels.add(sentinel);
        }

        String masterName = SentinelMasterName;

        pool = new JedisSentinelPool(masterName, sentinels, poolConfig, RedisConnectionTimeout);            
    }
    else
    {       
        pool = new pool(poolConfig, RedisIp, RedisPort, RedisConnectionTimeout);
    }

}           

protected Jedis getConnection()
{               
    if(pool==null)
        initializePool();

      Jedis jedis = pool.getResource();

      return jedis;     
}   

Answer 3

I noticed that this exception can and will be thrown if Redis is not running. Just a heads up.

Answer 4

Not sure, but maybe you don't return Jedis objects to the pool, and your redis-server has connection limit.

Every worker thread should return Jedis instances to the pool after its work is complete:

Jedis jedis = jedisPool.getResource();
try {
    jedis.getClient().setTimeoutInfinite();
    // your code here...
    ...
} finally {
    jedisPool.returnResource(jedis);
}

Answer 5

Did it happen always or occasionally? If it was occasional, you may want to check the size of your connection pool.

Default connection pool size, if you are using JedisPoolConfig, is 8. This could be too small for your case.

JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(128);
jedisPool = new JedisPool(poolConfig, HOST, PORT, ...);

Answer 6

If your code is like this:

JedisPoolConfig jedisPoolConfig = initPoolConfig();    
jedisPool = new JedisPool(jedisPoolConfig, "*.*.*.*", 6379);  

You can try this:

JedisPoolConfig jedisPoolConfig = initPoolConfig();    
jedisPool = new JedisPool(jedisPoolConfig, "*.*.*.*", 6379,10*1000); 

This is because for Redis the default timeout time is 2 seconds, but the program may have finished running in this time.