MPAS model hangs

[AndersJensen-NOAA]

MPAS hangs randomly (and sometimes not at all) when MPAS global print statements are set to true. The global print statements are config_print_global_minmax_vel, config_print_detailed_minmax_vel, config_print_global_minmax_sca. Additional model hangs have also been reported when setting these global print statements to false.

These print statements use the subroutine mpas_dmpar_maxattributes_real in /src/framework/mpas_dmpar.F, and the subroutine relies on MPI_Allreduce. When using (older) intel compilers, there are reports that MPI_Allreduce and MPI_Allgather can hang.

@SamuelTrahanNOAA recently debugged a hanging version of MPAS and found that one MPI task was stuck in a different part of the code than the other tasks.

Solutions:

1. Set global config print statements to false. This may help, but not completely fix the issue.
2. Use newer intel compilers if possible or gfortran. More testing will follow.

[SamuelTrahanNOAA]

The global MPAS HFIP 2025 runs had a similar problem. My workaround was to automatically detect the hanging job, kill it, and resubmit it. The logic to implement this reliably in mpas_app is somewhat complicated. However, it was a 100% reliable workaround.

[SamuelTrahanNOAA]

I've opened a ticket to the Ursa helpdesk (RDHPCS#2026011454000206) asking for a GNU+OpenMPI build of their parallel-netcdf 1.12.3. Once we have this, we can test without the Intel MPI.

EDIT: The parallel-netcdf version is 1.12.3, not 1.12.4

[SamuelTrahanNOAA]

OpenMPI may not be an option due to this limitation which the developers say won't be fixed:

• TCP BTL tries binding to address already in use open-mpi/ompi#7246

We're investigating using OpenMPI with UCX to bypass that limitation. Otherwise, we may choose a different MPI.

I'm working with the RDHPCS admins to find a way to test this without Intel MPI.