Exchange grid bug when atmosphere and land have different resolutions

[JosephMouallem]

As part of developing a high-resolution coupled SHiELD configuration, we encountered a bug where the exchange grid generation hangs indefinitely when configuring the atmosphere and land at different resolutions (Consider a C96 atmosphere with C48 land and 1deg ocean for simplicity). Ideally, this workflow should work out of the box to be consistent with other coupled model configurations.
At the moment, it only works under a specific workaround, and the implications of that workaround are unclear.

The relevant exchange regression tests are located under: test_fms/exchange.

Failing Configuration:
The following setup consistently hangs during the setup of the exchange grid:

make_hgrid --grid_type gnomonic_ed --nlon 96 --grid_name C48_grid
make_hgrid --grid_type gnomonic_ed --nlon 192 --grid_name C96_grid

make_solo_mosaic --num_tiles 1 --dir . --mosaic_name ocean_mosaic --tile_file ocean_hgrid.nc --periodx 360

make_solo_mosaic --num_tiles 6 --dir . --mosaic_name atmos_mosaic --tile_file C96_grid.tile1.nc,C96_grid.tile2.nc,C96_grid.tile3.nc,C96_grid.tile4.nc,C96_grid.tile5.nc,C96_grid.tile6.nc

make_solo_mosaic --num_tiles 6 --dir . --mosaic_name land_mosaic --tile_file C48_grid.tile1.nc,C48_grid.tile2.nc,C48_grid.tile3.nc,C48_grid.tile4.nc,C48_grid.tile5.nc,C48_grid.tile6.nc

make_coupler_mosaic --atmos_mosaic atmos_mosaic.nc --land_mosaic land_mosaic.nc --ocean_mosaic ocean_mosaic.nc --ocean_topog topog.nc --mosaic_name grid_spec --check --print_memory --verbose

Working workaround:

The exchange test only passes if both of the following changes are applied:
1- The atmosphere grid is generated with --great_circle_algorithm
2- Use --interp_order 1 with make_coupler_mosaic

Example:

make_hgrid --grid_type gnomonic_ed --nlon 96 --grid_name C48_grid
make_hgrid --grid_type gnomonic_ed --nlon 192 --grid_name C96_grid --great_circle_algorithm

Same calls as above:

make_coupler_mosaic --atmos_mosaic atmos_mosaic.nc --land_mosaic land_mosaic.nc --ocean_mosaic ocean_mosaic.nc --ocean_topog topog.nc --mosaic_name grid_spec --interp_order 1 --check --print_memory --verbose

Tested with fre/bronx-23 as well as with the latest main branch of FRE-NCtools on Gaea C6.

[nikizadehgfdl]

@JosephMouallem where is your ocean_hgrid.nc ? I cannot reproduce this with SPEAR ocean_hgrid
/work/Niki.Zadeh/datasets/modelgrids/mosaic_SPEAR.C96.OM360x320.20171213

I tried this exec on PAN
/work/Niki.Zadeh/exchange_grid_toolset/tools/FRE-NCtools_main/builds/gfdlpan/bin/make_coupler_mosaic_parallel

Here's my workdir
/work/Niki.Zadeh/exchange_grid_toolset/workdir/mosaic_ac96lc48oSPEAR.20260226.fixlon.an206/

[JosephMouallem]

@nikizadehgfdl, I am using the same 1deg ocean grid as well: /home/Joseph.Mouallem/grid_to_debug/ocean_files/ocean_hgrid.nc
The make_coupler_mosaic doesn't hang indeed but the exchange grid setup call setup_xmap under test_fms/exchange is the call that hangs depending on how make_coupler_mosaic is called.