stop Spring Scheduled execution if it hangs after

Question

I have used Spring Framework's Scheduled to schedule my job to run at every 5 mins using cron. But sometime my job waits infinitely for an external resource and I can't put timeout there. I can't use fixedDelay as previous process sometime goes in wait infinitely mode and I have to refresh data at every 5 mins.

So I was looking any option in Spring Framework's Scheduled to stop that process/thread after a fixed-time either it run successfully or not.

I have found below setting which initialized ThreadPoolExecutor with 120 seconds for keepAliveTime which I put in @Configuration class. Can anybody tell me will this work as I expected.

@Bean(destroyMethod="shutdown")
public Executor taskExecutor() {
    int coreThreads = 8;
    int maxThreads = 20;
    final ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
            coreThreads, maxThreads, 120L, 
            TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>()
    );
    threadPoolExecutor.allowCoreThreadTimeOut(true);

    return threadPoolExecutor;
}

Answer 1

allowCoreThreadTimeOut and timeout setting doesn't help cause it just allow work thread to be ended after some time without work (See javadocs)

You say your job waits infinitely for an external resource. I'am sure it's because you (or some third-party library you using) use sockets with time out infinite-by-default. Also keep in mind what jvm ignores Thread.interrupt() when it blocked on socket.connect/read.

So find out witch socket library used in your task (and how exactly it used) and change it's default timeout settings.

As example: there is RestTemplate widely used inside Spring (in rest client, in spring social, in spring security OAuth and so on). And there is ClientHttpRequestFactory implementation to create RestTemplate instances. By default, spring use SimpleClientHttpRequestFactory which use JDK sockets. And by default all it's timeouts are infinite.

So find out where exactly you freeze, read it's docs and configure it properly.

P.S. If you don't have enough time and "feeling lucky" try to run your app with setting jvm properties sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout to some reasonable values (See docs for more details)

Answer 2

I'm not sure this will work as expected. Indeed the keepAlive is for IDLE thread and I don't know if your thread waiting for resources is in IDLE. Furthermore it's only when the number of threads is greater than the core so you can't really know when it happen unless you monitor the threadpool.

keepAliveTime - when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating.

What you can do is the following:

public class MyTask {

    private final long timeout;

    public MyTask(long timeout) {
        this.timeout = timeout;
    }

    @Scheduled(cron = "")
    public void cronTask() {
        Future<Object> result = doSomething();
        result.get(timeout, TimeUnit.MILLISECONDS);
    }

    @Async
    Future<Object> doSomething() {
        //what i should do
        //get ressources etc...
    }
}

Don't forget to add @EnableAsync

It's also possible to do the same without @Async by implementing a Callable.

Edit: Keep in mind that it will wait until timeout but the thread running the task won't be interrupted. You will need to call Future.cancel when TimeoutException occurs. And in the task check for isInterrupted() to stop the processing. If you are calling an api be sure that isInterrupted() is checked.

Answer 3

The keepAliveTime is just for cleaning out worker threads that hasn't been needed for a while - it doesn't have any impact on the execution time of the tasks submitted to the executor.

If whatever is taking time respects interrupts you can start a new thread and join it with a timeout, interrupting it if it doesn't complete in time.

public class SomeService {

    @Scheduled(fixedRate = 5 * 60 * 1000)
    public void doSomething() throws InterruptedException {
        Thread taskThread = new TaskThread();
        taskThread.start();
        taskThread.join(120 * 000);
        if(taskThread.isAlive()) {
            // We timed out
            taskThread.interrupt();
        }
    }

    private class TaskThread extends Thread {

        public void run() {
            // Do the actual work here
        }
    }
}