Rescheduling Akka Schedulers

This blog post is an afterthought of myself working on Akka Schedulers in an official project. The objective here is to convey the challenges that one may face while working with scheduling/rescheduling Akka Schedulers.

On this journey of ours to understand scheduling/rescheduling Akka Schedulers, we will be discussing a reactive scheduler implementation based on Akka Schedulers which would respond to configuration file changes reactively. In other words, configuration file changes would not require service restarts. Not only would it respond to configuration file changes reactively but also it would reschedule itself based on the time taken to complete the job reactively. For instance, if the scheduler is scheduled to run every day and it takes 5 days for the job itself to finish. The scheduler would reschedule itself for every 5 days instead. Here, however, ttl would take precedence if the time taken to finish the job is less than the value of ttl. In other words, if the scheduler was scheduled for 5 days and the job takes 2 days to finish; the scheduler would remain scheduled for 5 days.

We would be discussing an expunge service that cleans up data obsoleted by the ttl value defined in the configuration file. Let's take a look at the actor system.

The configuration for the actor system depicted above can be defined something like this:

Every change (ttl/freq) to the configuration file is propagated to the Scheduler via ConfigWatcher and Supervisor which in turn forwards the ttl changes to DataExpunge.

What does it take to reschedule an akka scheduler?

It is the freq value in the configuration file that scheduling/rescheduling in Scheduler is based on. Every new freq value would require Scheduler to reschedule itself. This involves cancellation of already scheduled akka scheduler and subsequent instantiation of new akka scheduler with the new freq value. The reason being, a new instance of akka scheduler that implements Cancellable trait is returned everytime a scheduler is scheduled.

If you look at the documentation explaining Akka Scheduler, you would notice that all scheduler apis return a Cancellable, which is defined like so:

The only way to reschedule an akka scheduler would require cancellation of the same and subsequent instantiation of a new akka scheduler.

States need to be managed carefully when they are schedule dependent!

Responding reactively to the configuration changes by rescheduling the scheduler is not as straight forward as it may look, considering that the scheduler also reacts to the time taken to complete the job as explained above. Let's first see how rescheduling would work in absence of reacting to configuration changes.

Scheduler asks data expunge actor to perform the job by sending it message Do(ttl) which responds with message Done upon completion of the expunge job. It is upon the reception of this message that the scheduler computes the time taken to complete the job and reschedules itself accordingly as shown below:

Now, since scheduler reacts to both configuration changes and time taken by data expunge asynchronously, it could be a problem when the scheduler would attempt to reschedule while a expunge job could already be in progress. Rescheduling while a job already in progress would leave the next Do(ttl) message sit ideal in the DataExpunge actor's mailbox up until the actor finishes the job already in progress. This would result in computation of incorrect time taken by the new job, as shown in the diagram above, resulting in incorrect schedules. The only way out would be to cancel the scheduler as soon as the configuration changes are received while delaying the reschedule up until last Done message is received for the scheduler already cancelled.

You can find a working implementation of an extended version of the actor system discussed above on github.