Scheduler and Scheduler Extension OOM on clusters with large number of nodes

[chiragjn]

We are observing OOM on scheduler and scheduler extension pods on clusters with large number of nodes (1500+) with only 1 or 2 nodes being inferentia nodes. Our current hypothesis is the scheduler trying to add all nodes in the main memory.

We can keep on increasing the memory but is there any provision in the scheduler config to not scan and add non inferentia nodes?

Maybe a note in the docs will help

--

Also as a general curiosity, is the scheduler and extension source code available somewhere?

[mounchin]

Scheduler extn maintains a pod cache internally, which can grow with the number of pods on your cluster.

• Whats the current memory usage of the scheduler extn?
• How many pods are on your cluster? P50 / P99 size of your pods?

We will also look into not storing the entire pod in the cache and only store certain fields which the scheduer cares about it. For now the only option is to increase the memory.

| Also as a general curiosity, is the scheduler and extension source code available somewhere?
No, the code is not open sourced.