I have a Spring Boot application with various schedulers that use the @Scheduled annotation. Some of the schedules are setup using fixedRate, others use cron. The fixedRate schedules run fine, but I've noticed that over the past month the actual time at which the cron schedules fire will vary by 1-2 hours each day.
i.e. a job with cron setup of
@Scheduled(cron = "0 0 2 * * *")
will fire at 3:00 AM instead of 2:00 AM. It isn't happening every day either. It will run at the same time for a few days in a row, then something happens and it's off by another hour each time it is run until I restart the application.
The system time on the server is accurate, I'm all out of ideas here. Has anyone encountered this before?
EDIT 1
I put in some logging to determine if the jobs were mis-firing at the same interval, and the results seem random. Here are the start/end times for the aforementioned cron job over three days:
Format: Calendar.getInstance().getTime() (System.currentTimeMillis())
Start: Sat Oct 21 03:14:15 CDT 2017 (1508573655778)
End: Sat Oct 21 03:22:24 CDT 2017 (1508574144708)
Start: Sun Oct 22 02:26:58 CDT 2017 (1508657218774)
End: Sun Oct 22 02:35:12 CDT 2017 (1508657712492)
Start: Mon Oct 23 02:00:03 CDT 2017 (1508742003072)
End: Mon Oct 23 02:08:11 CDT 2017 (1508742491493)