Results differ with number of processors used and on different machines

[aabrown100-git]

I've found that simulation results are different when using different number of processors. This occurs on my personal computer, as well as on Sherlock (a Stanford computing cluster). Moreover, for the same number of processors, the results are different between my personal computer and Sherlock. The difference is small, but definitely different. The difference seems to be greatest in the beginning of the simulation.

To replicate, run the following simulation with SimVascular/svFSI (master branch) with 1 and 4 procs and observe the difference in B_ST_Velocity_flux.txt.
02-LV-Guccione-passive.zip

Does this have to do with ParMETIS splitting elements, which would effectively change the mesh slightly. Does this mean the mesh is not dense enough?

The difference is small, but I believe in some cases it can be consequential and I'd like to understand why this happens.

[vvedula22]

Yes, the differences are usually very small and if you were to run longer you will see the differences to be even smaller - identical for some time steps in many cases. Differences arise due to ParMetis division but primarily from the iterative linear solver that solves over local domains belonging to that process. Differences on different machines could be due to the different compiler environment and compiler flags that are used for optimizing loops. Generally, these differences are small, but small is subjective.

Mesh refinement may not help with this. Using tighter tolerances in the linear and nonlinear solvers may reduce the differences. Initial transient will further exacerbates these differences compared to when reaching a steady solution (if exists).