Scale down ScaledJob Jobs even when they have not completed

[blaizmannen]

We converted our Sidekiq(asynchronous job processor for ruby) deployment to a KEDA ScaledJob to tackle the following problem:

• scale down operations of our deployment causes some long running jobs to terminate prematurely. Solving that by making the terminationGracePeriodSeconds into a large value (i.e 3600) could work, but this would also block any new helm upgrades/new deployments from going through.

Hence we decided to use KEDA ScaledJob as a workaround, since jobs are independent of each other and a long running Job will not block a new helm upgrade of an app.

It works, but since our Sidekiq isn't a Job by nature, it never ever completes/exits and keeps running while waiting for new tasks. This prevents KEDA ScaledJob from scaling the Jobs down. Scaling up works fine.

What we could do is make the ScaledJob Job run a command where it checks if there are no active sidekiq jobs being processed for X consecutive seconds, then terminate the process and complete the Job. But before we do that, we would like to know if there is any native way that KEDA can achieve this, being allow the KEDA ScaledJob to scale down the Jobs even if they have not completed.

[JCMais]

did you figure this out? I need to do something similar with BullMQ

[SpiritZhou]

What about adding a "timeout" feature in ScaledJob and scale down jobs when it exceed the timeout? @tomkerkhove @JorTurFer @zroubalik

[zroubalik]

@SpiritZhou Adding a timetout might be great addition, though I am not sure if it fully solves the issue.

@blaizmannen sorry, I missed this somehow. I suppose you went with the proposed apporach, right?

[JCMais]

the timeout was basically what I did in the application layer. Having this as part of Keda would be great yeah.

The timeout would just scale the number of jobs down, but it should not kill the jobs, just signal for them to exit, as they may be processing a task atm

[blaizmannen]

To solve the problem, we ended up not going with KEDA.

Our sidekiq application remained as a kubernetes deployment. We then created a preStopHook script that would send a TSTP signal to the sidekiq application, and wait X amount of seconds OR wait till there are no more sidekiq jobs (whichever first), before the TERM and then ultimately the KILL signals are sent. This is enough to gracefully terminate our sidekiq pods in the event of any scale down operation/deployment update, and allow the jobs to be rescheduled back into the queue if they get interrupted

[JorTurFer]

you can do exactly this but scaling out/in based on the queue length xD
I mean, the goal of KEDA is driving the autoscaling, if you need the autoscaling feature and you have solved the pod shutdown, you can include a ScaledObject for scaling in/out (if you need it)