Memory handling on an infinite loop simulation

[pedro-ricardo]

Hello there ... firstly thank you for maintaining this library 👍
I have a question about how I can handle memory correctly.

I'm using PyFMI to set an infinite simulation coupled with an OpcUa Server to keep track and interact with the running model.

Here is some characteristics of my algorithm:

• I've built a simulation using the Master algorithm because I have lots of FMUs to link in my model.
• The infinite loop I've setup contains a simulation of a step and a sleep to make it realtime.
• I've seen that the Master class do not have a do_step function so to do this I'm using the simulate(start_time=time, final_time=time+step) function with {'initialize':False} set in options.

My problem is that running this long enough will lead to a memory increasing until it reaches a maximum allowed by the OS.

I do not need the FMUs to keep track of the history of values of the simulations, I only want the latest values in each step ...
I believe the error is here...

Assuming the error is that ... could you guide me to call the correct functions to clean this history so I can run the simulation indefinitely?

Also if you think that the error is somewhere else, could you give me some advice on how to setup this correctly?

Thank you.

[modelonrobinandersson]

Hi @pedro-ricardo, thank you for reaching out about your question. I have a couple of follow-up questions to understand your situation better.

1. Is your FMU running any external code?
2. Is this an ME FMU or CS FMU?
3. Are you using default options besides initialize: False?
4. Approximately how long time does it take until you see the memory issue?
5. During the simulation a log file is generated. Can you see if the size of the this file increases continuously with the simulation? It is unfortunately very easy to run into very large simulation log files that can take up your entire disk space if given the right circumstances

[pedro-ricardo]

Thanks for the quick response...

1. Is your FMU running any external code?

No, the FMUs don't even have an algorithm part, it's just parameters, input, output and the equation parts.

2. Is this an ME FMU or CS FMU?

All are CS FMUs

3. Are you using default options besides initialize: False?

Yes, I do the following in the loop:

step = 0.5
std_opt = master.simulate_options()
std_opt['step_size'] = 0.01
std_opt['result_handling'] = "none"
if (current_time>0.0):
    std_opt['initialize'] = False

master.simulate( start_time=current_time, final_time=current_time+step, options=std_opt )
current_time = current_time+step

time.sleep(step)

4. Approximately how long time does it take until you see the memory issue?

1000 seconds in real time, but it does happen faster if I fix the sleep step to 0.5s and enlarge the final_time of each simulation

5. During the simulation a log file is generated. Can you see if the size of the this file increases continuously with the simulation? It is unfortunately very easy to run into very large simulation log files that can take up your entire disk space if given the right circumstances

I don't see any file getting large ... but I do set the same log file for all models with: pyfmi.load_fmu(path, log_file_name='./sim_log.txt', log_level=2) maybe that is doing something funny. I've checked folder and disk space .. it is normal.

[pedro-ricardo]

This is the memory behavior I see:

it started with 200Mb and ended up with more than 10Gb before I killed it

[modelonrobinandersson]

Hi @pedro-ricardo, is master here the object returned from the function pyfmi.load_fmu or the returned object from class Master? Even though I don't think this has anything to do with your logging you can omit the log_file_name and let PyFMI use a default (based on the FMU). See also if your memory explodes when you set log_level=0.

[pedro-ricardo]

The master in the code snippet is: master = pyfmi.Master(model_list, connections) and the model_list is filled by appending pyfmi.load_fmu returned objects.

I'll ommit the log_file_name and set log_level=0 to check.

[pedro-ricardo]

See also if your memory explodes when you set log_level=0

The memory still explodes and now I also have a bunch of .txt empty files

[modelonrobinandersson]

@pedro-ricardo I wonder if the memory goes up because when you do time.sleep(step) Python does not not wait step seconds before it continues, since time.sleep does release the GIL.
As an attempt to isolate the behavior, instead of an infinite loop, run it for n steps, after the n steps simply do input('Waiting...') and monitor if the memory consumption goes down after a while (in case there are up to n of time.sleep waiting to finish).

[pedro-ricardo]

Hello there @modelonrobinandersson,
I'm sorry for the delay, the company asked to change our code to another library to check ... and it is running smooth in FMPy now.
But I'm still interested in finding where was my mistake with PyFMI.
The main difference when I switched to FMPy is that instead of the master algorithm they join the FMUs in a larger FMU, and then step it.

About the test you asked for, I've added an 11 iterations loop with 100s in each step (that will get me close to 1000s where the problem occurs) without the sleep and waited for and input after the loop finished.

As you can see, the memory keeps high after the loop ended. That time span after the loop ended is 4 minutes.

[modelonrobinandersson]

@pedro-ricardo can you share more of the code snippet you shared earlier just so we can understand the situation fully, it would be nice to find the issue to avoid more people running into this.
Can you perhaps from where you construct the class Master all the way to time.sleep as you have above? Also so the loop is visible.

[pedro-ricardo]

Ok, I've built a simple debug function, tested it and got the same behaviour.
I've also added some documentation on the parameters so you can see what this function receives

Here it is:

# --------------------
@staticmethod
def debug_start(models:dict, connections:list, initial_params:dict):
    ''' A simplified execution loop for debugging the Master algorithm\n
    * `models`: Dictionary with model name as key and the result of `pyfmi.load_fmu`
        functions as value. Example:\n
        ```
        {'model_name1':<pyfmi.fmi.FMUModelCS2 object at 0x159dba0>,
        'model_name2': <pyfmi.fmi.FMUModelCS2 object at 0x159c650>}
        ```
    * `connections`: List of connection dictionaries in the format:
        `{'from':(model_obj,variable_name),'to':(model_obj,variable_name)}`. Example:\n
        ```
        [ {'from': (<pyfmi.fmi.FMUModelCS2 object at 0x159dba0>, 'Out1'),
            'to': (<pyfmi.fmi.FMUModelCS2 object at 0x15c16e0>, 'Inp1')},
        {'from': (<pyfmi.fmi.FMUModelCS2 object at 0x159dba0>, 'Out2'),
            'to': (<pyfmi.fmi.FMUModelCS2 object at 0x15c16e0>, 'Inp2')} ]
        ```
    * `initial_params`: Dictionary with model names the initial values
        for model inputs and parameters. Example:\n
        ```
        {'model_name1':{'Par1': 500, 'Inp1': 530},
        'model_name2': {'Par1': 50, 'Par2': 40} }
        ```
    '''
    
    # Go though all model objects and set the initial parameters
    for m_name in models.keys():
        for prop, val in initial_params[m_name].items():
            models[m_name].set(prop,val)

    # Parse models and connections to pyfmi Master format.
    model_list = list(models.values())
    connection_list = [c['from']+c['to'] for c in connections]
    
    # Build master algorithm 
    master = pyfmi.Master(model_list, connection_list)

    # Set initial configurations
    current_time = 0.0
    std_opt = master.simulate_options()
    std_opt['step_size'] = 0.01
    std_opt['result_handling'] = "none"

    # Simulate
    for i in range(11):
        
        if (i>0):
            # Disable initialization after first iteration
            std_opt['initialize'] = False

        # Call master simulation
        master.simulate( start_time=current_time,
            final_time=current_time+100, options=std_opt)            
        
        # Update simulation Time
        current_time = current_time+100

    # Wait after finish
    input('Press any Key to end ....')
# --------------------

[pedro-ricardo]

I'm not using the time.sleep nor the infinite loop above because the loop you suggested had the same memory problem and it is simpler.

[pedro-ricardo]

Isn't it possible that this pyfmi.Master class is just accumulating all the simulated data inside it? Maybe I should get the returned object and perform some kind of free/clean.

This coupled simulation I'm doing is pretty big and the step_size is small.
For the purpose of running this as "real time", I will not need this internal history to be kept by pyfmi

[modelonrobinandersson]

@pedro-ricardo thank you a lot. Sorry for the questions but I have a couple of suspicions:

1. Can you see if you get the same behavior if opts['logging'] is set to False (if you have it as True)? Note this by default False, but I see some data is appended to lists if this is True.
2. The class instance Master also has a dict named self.models_dict, can you see if this changes over time? You can try simply using print(master.models_dict). It would be helpful to see if it still looks "reasonable" by the time you see your memory consumption explode. I can also find several places in the code where the following is commented out as #store_communication_point(self.models_dict) except for def simulate(...) and wonder if this is causing the memory to go up.

Let me know your findings.