KEDA best practice questions

[endejoli]

We manage a central OpenShift cluster in our organization. We are now doing a POC with KEDA. I have a few basic general questions related to KEDA

1. Would it generally be advisable to let KEDA controller monitor only the required namespaces? or is it generally considered safe to leave that value blank so KEDA controller monitors all the namespaces?
2. We will be letting our developers to create scaledObjects in our cluster. If someone creates thousands of scaledObjects CRD in a namespace, Would it bring down KEDA controller ?. Maybe no one does this but just trying to figure out the control measures to prevent such accidental stuff.

Thanks,
Tintu

[tomkerkhove]

@coderanger has used a lot of ScaledObjects in production before so he can talk about 2 but I don't think this should be an issue. We are adding metrics to the operator so total # of SO would be good help here, what do you think @zroubalik? (although I think we are planning this already)

In terms of 1, it's more something you should ask if that's ok for your scenario but I don't see many concerns on that front to be honest.

[endejoli]

@zroubalik Thank you for the response

1. When you say KEDA watches all namespaces, that is only if keep watchNamespace to empty correct?
2. Thank you for the policy link on object quota, that really helps. But just for my awarness, asking the question again, would it really bring down the KEDA controller if there are too many scaledobjects?

[zroubalik]

1. yes
2. 🤷‍♂️ I think that eventually a ton of SOs will slow it down, it's just a piece of software, right? :) But I can't give you the exact numbers, we unfortunately don't have them. It would be really great if you can share with us your observations wrt this topic.

[endejoli]

Sure, I will let you know if I ever brought it down :) .. I was not really looking at a number, to be honest. I was just looking to see if there is a possibility of a production outage if someone does something stupid :)
Anyhow I will explore the policy option to restrict the same. Appreciate the patience and thank you for the support.

[zroubalik]

Sure thing, happy to help :)

Btw KEDA is operator managed by k8s, so if you bring down a particular instance of KEDA, k8s scheduler will spin up another pod with a new KEDA instance and this instance should catch up on deployed SOs. There is no state being stored, so the restart should be pretty trivial.

[coderanger]

Adding more scaled objects won't specifically destabilize the operator though it will increase RAM usage slowly so you may need to increase limits there eventually. It will eventually impact startup time though with ~1000 right now, startup is still only about 30 seconds so I wouldn't worry too much.