memory leak in ocm controller 0.27.3

[dee0sap]

Context
There is apparently a memory leak in ocm-controller 0.27.3

NOTE: This hasn't been released yet as the current ocm-controller code needs a new release of OCM and there is only an RC available.

Version
v0.27.3 ( not released yet )

To Reproduce
Steps to reproduce the behavior:

Not sure. In our clusters we currently have 100 objects that the ocm-controller must reconcile. That is a combination of cv, localizations, cfg, resources and fluxdeployers.

With 512M requested and a 1G limit the ocm-controller can run for about 10m.

With 1G requested and a 2G limit it ran for 97m before being oom killed.

While the logs show a few errors these aren't themselves indicators or a memory problem. Instead they are about a couple of mutation objects that refer to a cv that don't exist.

Actual behavior
With the original memory settings the ocm-controller pretty quickly makes it way up to the 1G limit. I will attach a graph of this.

Expected behavior
Don't get oom killed

Screenshots (optional)

Environment

• windows
• linux
• [] mac

Additional Comments
Any observation, gut feeling or other information you think help the community to troubleshoot the problem?

[Skarlso]

Hey Dan!

We need a bit more info here. :) What I'm missing is comparable results. I don't have data from the previous version for that graph for example, so I don't know what's happening. I'm sure there is memory usage increase maybe because of various changes but I need a baseline.

[dee0sap]

Hey @Skarlso

I'll pass this on to my team. I am trying to wrap up things before I travel to SAP Walldorf for a workshop next week and then go on vacation after that.

[Skarlso]

Thanks, Dan!

[technovatin222]

Testing on ocm-controller 0.27.1 also shows memory leak, it looks to me the memory leak introduced in older versions:

[Skarlso]

@technovatin222 Hello 👋 Which version of the controller is that now?

[Skarlso]

BTW, did you try increasing the memory allowance and see if it stabilizes at some point or if it just keeps going up forever?