Add omega support for single column tests

[cbegeman]

Add omega support for single column tests and reorganize for shared init steps

Checklist

• User's Guide has been updated
  * [ ] Developer's Guide has been updated
• API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
• Documentation has been built locally and changes look as expected
• Testing comment in the PR documents testing used to verify the changes
• New tests have been added to a test suite

[cbegeman]

@hyungyukang This is a starting point when you're ready test vertical mixing tendencies with E3SM-Project/Omega#310. There may be additional work to support Omega.

[hyungyukang]

@cbegeman , I’ve tested this PR with Omega after implementing the implicit vertical mixing, using @katsmith133 's vertical mixing module PR (E3SM-Project/Omega#310).

There are a few things that currently need to be adjusted manually when running Omega, and I also encountered an issue with visualization. I believe both of these can be fixed in a straightforward way. Despite these issues, I was able to run the test successfully and compare the results with those from MPAS-Ocean.
Below is the process I followed to run Omega on Frontier:

git clone git@github.com:cbegeman/polaris.git
cd polaris
git checkout column-variants

cd e3sm_submodules
rm -rf Omega
git clone -b hkang/omega/add-tendency-terms git@github.com:hyungyukang/Omega.git
cd Omega
git submodule update --init --recursive externals/ekat externals/scorpio cime externals/cpptrace components/omega/external/yaml-cpp components/omega/external/GSW-C

cd ../../

module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load cray-parallel-netcdf

./configure_polaris_envs.py --conda /lustre/orion/cli115/proj-shared/hgkang/miniconda3 -m frontier -c craycray -i mpich

source ./load_dev_polaris_0.10.0-alpha.1_frontier_craycray_mpich.sh

polaris setup -t  ocean/single_column/cvmix -m frontier -w /lustre/orion/cli115/proj-shared/hgkang/E3SM/OMEGA/Polaris/work_SC_test_cray --model omega

cd /lustre/orion/cli115/proj-shared/hgkang/E3SM/OMEGA/Polaris/work_SC_test_cray
cd ocean/single_column/cvmix/forward_no_vadv

In this directory, some changes to omega.yml are required in order to run the test. Below is the omega.yml file that I used:

omega.yml

Omega:
  Timing:
    Level: 2
    AutoFence: true
    TimingBarriers: false
    PrintAllRanks: false
  TimeIntegration:
    CalendarType: No Leap
    TimeStepper: Forward-Backward
    TimeStep: 00:10:00
    StartTime: 0001-01-01_00:00:00
    StopTime: none
    RunDuration: 0010_00:00:00
  Decomp:
    HaloWidth: 3
    DecompMethod: MetisKWay
  IO:
    IOTasks: 1
    IOStride: 1
    IOBaseTask: 0
    IORearranger: box
    IODefaultFormat: NetCDF4
  State:
    NTimeLevels: 2
  Advection:
    FluxThicknessType: Center
    FluxTracerType: Center
  WindStress:
    InterpType: Isotropic
  VertCoord:
    Density0: 1026.0
    MovementWeightType: Uniform
  Tendencies:
    ThicknessFluxTendencyEnable: true
    PVTendencyEnable: false
    KETendencyEnable: false
    SSHTendencyEnable: false
    VelDiffTendencyEnable: false
    ViscDel2: 1.0e3
    VelHyperDiffTendencyEnable: false
    ViscDel4: 2.0e14
    DivFactor: 1.0
    VelVertMixTendencyEnable: true
    PresForceTendencyEnable: false
    PresGradForceTendencyEnable: false
    GeoptGradTendencyEnable: false
    WindForcingTendencyEnable: true
    BottomDragTendencyEnable: false
    BottomDragCoeff: 0.0
    TracerHorzAdvTendencyEnable: false
    TracerDiffTendencyEnable: false
    EddyDiff2: 10.0
    TracerHyperDiffTendencyEnable: false
    EddyDiff4: 0.0
    TracerVertMixTendencyEnable: true
    UseCustomTendency: false
    ManufacturedSolutionTendency: false
  Tracers:
    Base: [Temperature, Salinity]
    Debug: [Debug1, Debug2, Debug3]
  Eos:
    EosType: linear
    Linear:
      DRhoDT: -0.2
      DRhoDS: 0.01
      RhoT0S0: 1000.0
  VertMix:
    Background:
      Diffusivity: 1.0e-5
      Viscosity: 1.0e-4
    Convective:
      Enable: true
      Diffusivity: 1.0
      TriggerBVF: 0.0
    Shear:
      Enable: true
      NuZero: 0.005
      Alpha: 5.0
      Exponent: 2.0
  ManufacturedSolution:
    WavelengthX: 5.0e6
    WavelengthY: 4.33013e6
    Amplitude: 1.0
  IOStreams:
    InitialVertCoord:
      UsePointerFile: false
      Filename: init.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
      - InitVertCoord
    # InitialState should only be used when starting from scratch.
    # For restart runs, the frequency units should be changed from
    # "OnStartup" to "never" so that the initial state file is not read.
    InitialState:
      UsePointerFile: false
      Filename: init.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
      - State
      - WindStressZonal
      - WindStressMeridional
      - Tracers
    History:
      UsePointerFile: false
      Filename: output.nc
      Mode: write
      IfExists: append
      Precision: double
      Freq: 1
      FreqUnits: Seconds
      UseStartEnd: false
      Contents:
      - AuxiliaryState
      - State
      - SshCell
      - Tracers
      FileFreq: 9999
      FileFreqUnits: years
    # Restarts are used to initialize for all job submissions after the very
    # first startup job. We use UseStartEnd with a start time just after the
    # simulation start time so that omega does not attempt to use a restart
    # for the first startup job.
    RestartRead:
      UsePointerFile: true
      PointerFilename: ocn.pointer
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: true
      StartTime: 0001-01-01_00:00:01
      EndTime: 99999-12-31_00:00:00
      Contents:
      - Restart
    RestartWrite:
      UsePointerFile: true
      PointerFilename: ocn.pointer
      Filename: ocn.restart.$Y-$M-$D_$h.$m.$s
      Mode: write
      IfExists: replace
      Precision: double
      Freq: 6
      FreqUnits: months
      UseStartEnd: false
      Contents:
      - Restart

In the omega.yml, I had to 1) add Restart IO section, 2) turn off several tendencies, and 3) add Tracers in InitialStates:Contents.

cp omega.yml  ../forward

cd /lustre/orion/cli115/proj-shared/hgkang/E3SM/OMEGA/Polaris/work_SC_test_cray

sbatch job_script.custom.sh

After the test, I was able to see SUCCESS in the log file. However, the visualization step failed due to mislinked output files. It also appears that Polaris attempted to read date information from the Omega output files, whose format has since changed.

I will post the results in another comment.

[hyungyukang]

I quickly visualized the results to compare them with MPAS-O.

In the results, there is a slight difference in Omega when Shear: Enable: true (the last figure). Also, the MPAS-O results seem to be almost unchanged regardless of the options config_use_cvmix_shear and config_use_cvmix_convection. One additional point I wanted to confirm is that MPAS-O is using KPP via config_cvmix_shear_mixing_scheme = 'KPP' (should this instead be set to PP?).

[cbegeman]

Suggested change

	config_cvmix_shear_mixing_scheme: KPP
	config_cvmix_shear_mixing_scheme: PP

[cbegeman]

Suggested change

	config_use_cvmix_kpp: true
	config_use_cvmix_kpp: false

[cbegeman]

Suggested change

	config_cvmix_kpp_matching: MatchBoth
	config_cvmix_kpp_interpolationOMLType: cubic

[cbegeman]

In the results, there is a slight difference in Omega when Shear: Enable: true (the last figure). Also, the MPAS-O results seem to be almost unchanged regardless of the options config_use_cvmix_shear and config_use_cvmix_convection.

For convection, my understanding is that Omega does not yet support evaporative forcing. @katsmith133 and I talked about starting off with unstable stratification but I don't have this included yet in this branch. I'll put that on my to-do list.

For shear, I think we either need to
a. change the magnitude of wind forcing
b. change the parameters used in the shear parameterization (maybe increase config_cvmix_shear_PP_nu_zero)

@katsmith133 let me know if you have recommendations for these values. Otherwise I can play around with them a bit.

[cbegeman]

@hyungyukang In your omega.yml file above, WindForcingTendencyEnable: false and it should be set to true to have a source of shear.

Let's also leave ThicknessFluxTendencyEnable: true so that Omega is ready for those surface forcing options when they are available. I'll make sure that Polaris sets both this and the wind forcing tendency config options

[hyungyukang]

@cbegeman , Thanks for the suggestions!

After the VMix module was merged into the develop branch, I reworked the tendency-term implementation on a new branch (hkang/omega/add-tendency-terms). The branch name has been updated in the procedure I described above.

I tested again (forward_no_vadv) with WindForcingTendencyEnable: true and ThicknessFluxTendencyEnable: true, as you suggested, and I don’t see any difference in the results since the surface stress is zero (not sure this is intended at this moment or not).

For MPAS-Ocean, I also reran the tests using your suggested settings:

# Change
    config_cvmix_shear_mixing_scheme: PP
    config_use_cvmix_kpp: false
 
# Remove
    config_cvmix_kpp_matching: MatchBoth
    config_cvmix_kpp_interpolationOMLType: cubic

It appears that there is no noticeable change compared to the initial state.

If you have any suggestions you want to test, please let me know.

[cbegeman]

@vanroekel @katsmith133 I realized that I didn't set config_disable_thick_vadv to True so that could explain the problem. I'll update here when I've rerun the case.

Here's the namelist file for the wind-driven shear column test case with no vertical advection:
/lcrc/group/e3sm/ac.cbegeman/scratch/MPAS-Ocean-output/chrys/polaris-column-E3SM-master-20260111/ocean/single_column/vmix_stable/forward_no_vadv/namelist.ocean

[vanroekel]

One more change -> config_use_cvmix_kpp = .false.

Also thinking about this I'm fairly certain each column in this case should be BFB with each other for a constant stress + heat + evap flux. I can't think why we would generate vertical velocity in this case.

[cbegeman]

Testing

Both MPAS-Ocean and Omega have been run on chrys with intel, openmpi and the ocean/single_column/vmix_stable and ocean/single_column/vmix_unstable tasks.

MPAS-Ocean issues:

• For MPAS-Ocean's vmix_stable, PP shear mixing does not appear to be active when enabled via config option. @katsmith133 is looking into it, as she has run MPAS-Ocean's PP in the past.

Omega issues:

• SshCell is incorrect. Consistent with -bottomDepth + layerThickness(k), which for these tests is -396. Its dimensions also include NVertLayers
• ZMid output from Omega is not consistent with the geometric zMid, ranges from ZMid(k=0) = 9.8 to ZMid(k=nVertLayers) = -397.9
• ZInterface ranges from ZInterface(k=0) = 10.25 to ZInterface(k=nVertLayersP1) = -BottomDepth for a case with no mass flux

Polaris issues:

• For Omega's forward_no_vadv_constant steps, PP shear mixing must be manually disabled until Config mapping issue with 2-level config options #450 is resolved
• For vmix_unstable visualization, the temperature profile that is plotted does not correspond to the values logged prior to plotting. Cause of the discrepancy unknown.
• For Omega's viz steps, velocity figures will not be produced until the reconstruction capability is added (planned addition first in Polaris, then in Omega)
• For Omega's forward steps, vertical advection is not yet available so these steps (the only forward steps that have vertical advection active) are not run by default for MPAS-Ocean or Omega.

[hyungyukang]

Suggested change

PresForceTendencyEnable: false

This term does not exist anymore after rewriting PGF in Omega.

[hyungyukang]

Suggested change

GeoptGradTendencyEnable: false

This term is included in the PGF.

[hyungyukang]

Suggested change

	PresGradForceTendencyEnable: false
	PressureGradTendencyEnable: false

[hyungyukang]

Suggested change

	config_cvmix_num_ri_smooth_loops: 0
	streams:
	config_cvmix_num_ri_smooth_loops: 0
	debug:
	config_disable_thick_sflux: true
	config_disable_vel_hadv: true
	config_disable_vel_coriolis: true
	config_disable_vel_pgrad: true
	config_disable_tr_sflux: true
	streams:

Turn off some tendencies in MPAS-O to match the tendency settings used in Omega.

[hyungyukang]

Are these supposed to be 0? If not, they should probably be set as shown below:

Suggested change

	config_cvmix_background_diffusion: 0.
	config_cvmix_background_viscosity: 0.
	config_cvmix_background_diffusion: 1.0e-5
	config_cvmix_background_viscosity: 1.0e-4

[hyungyukang]

I retested ocean/single_column/vmix_stable/forward_no_vadv_constant using the new implementation of implicit vertical mixing in Omega (working branch: https://github.com/hyungyukang/Omega/tree/hkang/merge-eos-const-PR).

To match the MPAS-Ocean equations, I cherry-picked E3SM-Project/Omega#370 to enable the constant EOS option.

Using the RungeKutta4 time stepper in Omega and RK4 in MPAS-Ocean, I obtained nearly identical results (~1.0e-11 difference) after applying a few option changes (noted as comments in this PR):

• Vertical profile of normal velocity at day 10 at the first edge

• Vertical profile of temperature at day 10 at the first cell

Note that the Forward-Backward time stepper in Omega appears to require a time step smaller than 150 s to prevent the model from blowing up.

I will run the Ekman test later.