Describe the bug
Memory usage for parallel findsoln runs increases over time. This is becoming a problem for me during high resolution solves. Can anyone reproduce this?
Expected Result
It should behave as for findsoln compiled in serial: memory constant during GMRES steps and increasing during the hookstep procedure, after which any memory allocated for the hookstep is deleted.
Actual Result
The footprint increases rapidly for the duration of the solve, including during individual time-integrations (calls to f).
Steps to reproduce the issue
It's not necessary to take any GMRES steps to see this happening. Memory starts to leak on the first evaluation of the residual. To reproduce, run findsoln on any state, e.g. if I set the period very high to prolong the computation of f...
$ mpirun -np 28 findsoln -eqb -R 410 -T 5000 eq1.nc
61506 unknowns
Computing G(x)
f^T: ....10...etc
and run top, I can see the memory increasing during the time integration steps, which does not happen for serial builds.
It's worth pointing out that simulateflow does not seem to have this problem. I can run that in parallel and memory usage is constant throughout. This is where I am stuck for the moment. While integrating, findsoln is looping through what is currently lines 749-768 of cfdsi.cpp. Well that seems to me to be essentially the same thing as lines 117-145 of simulateflow.cpp, which works fine. I suppose this must be a red herring and the real leak is somewhere else - my MPI is probably too rusty to debug this.
Information on your system
I am testing this on an HPC node with OpenMPI 2.0.1.
Describe the bug
Memory usage for parallel findsoln runs increases over time. This is becoming a problem for me during high resolution solves. Can anyone reproduce this?
Expected Result
It should behave as for findsoln compiled in serial: memory constant during GMRES steps and increasing during the hookstep procedure, after which any memory allocated for the hookstep is deleted.
Actual Result
The footprint increases rapidly for the duration of the solve, including during individual time-integrations (calls to f).
Steps to reproduce the issue
It's not necessary to take any GMRES steps to see this happening. Memory starts to leak on the first evaluation of the residual. To reproduce, run findsoln on any state, e.g. if I set the period very high to prolong the computation of f...
and run top, I can see the memory increasing during the time integration steps, which does not happen for serial builds.
It's worth pointing out that simulateflow does not seem to have this problem. I can run that in parallel and memory usage is constant throughout. This is where I am stuck for the moment. While integrating, findsoln is looping through what is currently lines 749-768 of cfdsi.cpp. Well that seems to me to be essentially the same thing as lines 117-145 of simulateflow.cpp, which works fine. I suppose this must be a red herring and the real leak is somewhere else - my MPI is probably too rusty to debug this.
Information on your system
I am testing this on an HPC node with OpenMPI 2.0.1.