Investigating why mom5 is slower with oneAPI

[manodeep]

This issue tracks my investigation into why mom5 is significantly slower when compiled with oneAPI. For starters, I compared the timings from the logfile.0000.out for two runs - the released config (with ifort, atmosphere layout 16x13, ocean layout 14x14) and one of the faster oneAPI versions that still used 416 cores (atmosphere layout 12x13, ocean layout 19x13)

Two functions (assuming the time for one doesn't include the other) - (Ocean shortwave) and (Ocean shortwave penetrate) together contribute to nearly 200 seconds of slowdown.

Here's the list of the functions that stood out as taking significantly more time with oneAPI:

Comparing the last 204 lines from logfile.00000.out

Generated from vimdiff'ing last_204_lines_ocean_logfile_000000_out_pr98-29-atm-layout-12x13-ocean-layout-19x13-cice4-12.txt and last_204_lines_ocean_logfile_000000_out_Jun20-released-config.txt in /g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids (or, via symlink, as /home/593/ms2335/perf-opt-classic-esm1.6/sapphirerapids)

function	time with ifort	time with ifx
(Ocean update density)	68.753486	130.024669
(Ocean shortwave)	23.124541	117.218480
(Ocean surface flux)	41.250250	189.090736
(Ocean barotropic dynamics)	46.788131	152.962205
(Ocean diagnostics)	62.194185	98.986182
(Ocean tracer halo updates)	275.070814	322.911296
(Ocean sum ocean surface)	88.946095	127.932427
(Ocean tracer tmask limit)	22.219161	82.341906
(Ocean tracer: advection)	233.915603	264.396445
(Ocean generic tracer: source)	781.969017	851.825148
(Ocean neutral: diffuse)	345.828021	418.568721
(Ocean ndiffuse: fx-flux)	64.232973	117.111274
(Ocean ndiffuse: fy-flux)	59.803690	109.397266
(Ocean advect: MDFL-sweby-cui)	37.223727	63.833007
(Ocean shortwave penetrate)	15.296601	103.740304
(Ocean tracer_diag: total water	5.859774	37.781129

[manodeep]

Concatenated the data from the two timings

# Concatenated output:              ifort           ifx
##############################################################
Total runtime                      4318.405117	  5401.304575
Ocean                              3959.762338	  4806.477286
(Ocean initialization)                3.785997	     4.520254
(Ocean ODA)                           0.000000	     0.000000
(Red Sea/Gulf Bay salinity fix)       0.152466	     0.275797
(Ocean advection velocity)           16.619888	    19.078135
(Ocean density diag)                 15.823220	    26.562481
(Ocean update density)               68.753486	   130.024669
(Ocean vertical mixing coeff)       131.185507	   141.752537
(Ocean neutral physics)             928.926576	   980.566021
(Ocean viscous form drag)             2.400294	     1.514246
(Ocean submesoscale restrat)        251.978845	   239.995412
(Ocean shortwave)                    23.124541	   117.218480
(Ocean sponges_eta)                   0.003970	     0.001412
(Ocean sponges_tracer)                0.005306	     0.002551
(Ocean sponges_velocity)              0.005444	     0.003369
(Ocean xlandinsert)                   0.006231	     0.007710
(Ocean xlandmix)                      0.005684	     0.004030
(Ocean rivermix)                     17.155149	    16.872525
(Ocean overexchange)                  0.009112	     0.005447
(Ocean mixdownslope)                101.305000	    99.885226
(Ocean blob update)                   0.003193	     0.002947
(Ocean blob cell update)              0.000000	     0.000000
(Ocean blob diagnose depth)           0.002386	     0.001973
(Ocean overflow)                      0.516753	     3.251078
(Ocean overflow_OFP)                  0.006695	     0.005386
(Ocean sigma transport)              32.223843	    31.152822
(Ocean tracer update)               411.899566	   433.534896
(Ocean surface flux)                 41.250250	   189.090736
(Ocean bottom flux)                   0.510496	     0.488641
(Ocean restoring flux)                0.654708	     0.665941
(Ocean TPM source)                    1.032742	     0.625914
(Ocean TPM bbc)                       0.001546	     0.000639
(Ocean TPM tracer)                  840.582128	   887.539475
(Ocean explicit accel_a)            164.165828	   170.728563
(Ocean explicit accel_b)              3.045653	     1.934835
(Ocean implicit accel)               18.115767	    15.925851
(Ocean eta and pbot tendency)         0.106819	     0.094398
(Ocean eta and pbot update)           0.528586	     0.451239
(Ocean eta and pbot diagnose)        34.682037	    29.096617
(Ocean rho_dzt tendency)              1.829594	     1.342900
(Ocean dzt_dst update)                0.000000	     0.000000
(Ocean surface height smooth)        15.379911	    24.313785
(Ocean bottom pressure smooth)        0.000000	     0.000000
(Ocean mass forcing)                  1.056531	     0.841298
(Ocean barotropic forcing)            2.190875	     2.025157
(Ocean barotropic dynamics)          46.788131	   152.962205
(Ocean velocity update)              15.423497	    16.926787
(Ocean diagnostics)                  62.194185	    98.986182
(Ocean update T-cell thickness)      16.264056	    18.801497
(Ocean update Total thickness)        0.000000	     0.000000
(Ocean update L sys. thickness)       0.000000	     0.000000
(Ocean update E sys. thickness)       0.000000	     0.000000
(Ocean update U-cell thickness)      16.961868	    24.939252
(Ocean tracer halo updates)         275.070814	   322.911296
(Ocean velocity halo update)          1.527814	     1.567438
(Ocean sum ocean surface)            88.946095	   127.932427
(Ocean average state)                 0.029699	     0.050383
(Ocean tracer tmask limit)           22.219161	    82.341906
(Ocean gotm: advection)               0.000000	     0.000000
(Ocean increment eta)                 0.003835	     0.000617
(Ocean increment tracer)              0.003956	     0.002538
(Ocean increment velocity)            0.005122	     0.002106
(Ocean update rho_salinity)           0.201834	     0.152608
(Ocean idealized surface waves)       0.004707	     0.005207
(Ocean open boundaries)               0.000000	     0.000000
(Ocean tracer: bih tracer)            0.017899	     0.015993
(Ocean tracer: lap tracer)            0.016531	     0.018834
(Ocean tracer: vert diffuse)         16.167541	     9.119890
(Ocean tracer: vert diffuse impl     29.204165	    33.321576
(Ocean tracer: advection)           233.915603	   264.396445
(Ocean tracer: frazil)                3.400824	     2.491245
(Ocean tracer: convection)            0.003087	     0.003456
(Ocean tracer: Lagrangian blobs)      0.030037	     0.018968
(Ocean tracer: implicit blobs)        0.006552	     0.005305
(Ocean tracer: adjust the L thic      0.000327	     0.001134
(Ocean tracer: adjust the E thic      0.006619	     0.004147
(Ocean CT2PT: 1)                      0.798708	     1.328300
(Ocean CT2PT: 2)                      0.000000	     0.000000
(Ocean CT2PT: 3)                      0.000000	     0.000000
(Ocean generic tracer: vertdiff)     55.964717	    33.255515
(Ocean generic tracer: source)      781.969017	   851.825148
(Ocean generic tracer: bottom up      0.368527	     0.576816
(Ocean generic tracer: diag)          0.014033	     0.016359
(Ocean generic tracer: set_value      0.000000	     0.000000
(Ocean generic tracer: get_value      0.284771	     0.258744
(Ocean generic tracer: sumsfcset      0.000000	     0.000000
(Ocean vmix: constant)                0.000000	     0.000000
(Ocean vmix: PPvmix)                  0.000000	     0.000000
(Ocean vmix: KPP_test)                0.000000	     0.000000
(Ocean vmix: KPP_mom4p0)              0.000000	     0.000000
(Ocean vmix: KPP_mom4p1)             81.194335	    82.782496
(Ocean vmix: chenvmix)                0.000000	     0.000000
(Ocean vmix: gotmvmix)                0.000000	     0.000000
(Ocean vmix: tidal)                  45.359259	    56.932203
(Ocean vmix: watermass_diag)          0.007820	     0.008299
(Ocean neutral: tracer derivs)       89.173714	    74.342841
(Ocean neutral: slopes)              12.126679	     7.227450
(Ocean neutral: eady rate)           21.248923	    35.185279
(Ocean neutral: baroclinic)           0.049023	     0.031548
(Ocean neutral: rossby radius)        2.289100	     1.572614
(Ocean neutral: bc zone radius)       0.000000	     0.000000
(Ocean neutral: diffusivity)          7.315881	    19.055356
(Ocean neutral: cabbel/thermob)       0.000000	     0.000000
(Ocean neutral: eta_tend_gm90)        0.000000	     0.000000
(Ocean neutral: nrho-trans)           0.000000	     0.000000
(Ocean neutral: rho-trans)            0.000000	     0.000000
(Ocean neutral: theta-trans)          0.000000	     0.000000
(Ocean neutral: diffuse)            345.828021	   418.568721
(Ocean neutral: GM stir)            310.308989	   256.450115
(Ocean neutral: blayer)               3.312345	     7.676325
(Ocean neutral: fz-terms)            42.556532	    58.048823
(Ocean ndiffuse: fx-flux)            64.232973	   117.111274
(Ocean ndiffuse: fy-flux)            59.803690	   109.397266
(Ocean ndiffuse: fz-flux)            77.798731	    75.181703
(Ocean gm: fx-flux)                  38.622291	    32.047321
(Ocean gm: fy-flux)                  40.460363	    34.299820
(Ocean gm: fz-flux)                  64.474251	    53.600265
(Ocean gm: bc_modes)                  9.069349	     9.543167
(Ocean gm: psi_modes)                 0.000000	     0.000000
(Ocean gm: psi_bvp)                  76.884308	    85.841739
(Ocean const   lap frict)             0.000000	     0.000000
(Ocean general lap frict)            44.110204	    62.295514
(Ocean C-grid lap frict)              0.000000	     0.000000
(Ocean const   bih frict)             0.000000	     0.000000
(Ocean general bih frict)            79.353581	    73.806192
(Ocean c-grid  bih frict)             0.000000	     0.000000
(Ocean advect: horz 4th)              0.000000	     0.000000
(Ocean advect: vert 4th)              0.000000	     0.000000
(Ocean advect: horz 6th)              0.000000	     0.000000
(Ocean advect: vert 6th)              0.000000	     0.000000
(Ocean advect: horz quk)              0.000000	     0.000000
(Ocean advect: vert quk)              0.000000	     0.000000
(Ocean advect: horz qukmom3)          0.000000	     0.000000
(Ocean advect: vert qukmom3)          0.000000	     0.000000
(Ocean advect: MDFL-sup-b)            0.000000	     0.000000
(Ocean advect: MDFL-sweby)            0.000000	     0.000000
(Ocean advect: MDFL-sweby-all)      233.852472	   264.337367
(Ocean advect: MDFL-sweby-test)       0.000000	     0.000000
(Ocean advect: DST-linear)            0.000000	     0.000000
(Ocean advect: DST-linear-test)       0.000000	     0.000000
(Ocean advect: MDFL-sweby-mpi)       33.666781	    23.843408
(Ocean advect: MDFL-sweby-cuk)       33.480322	    46.779349
(Ocean advect: MDFL-sweby-cui)       37.223727	    63.833007
(Ocean advect: MDFL-sweby-cuj)       62.110208	    73.118260
(Ocean advect: MDFL-sweby-diag)      24.351281	    33.642747
(Ocean advect: MDPPM)                 0.000000	     0.000000
(Ocean advect: MDPPM-TEST)            0.000000	     0.000000
(Ocean advect: psom total)            0.000000	     0.000000
(Ocean psom advect: psom_x)           0.000000	     0.000000
(Ocean psom advect: psom_y)           0.000000	     0.000000
(Ocean psom advect: psom_z)           0.000000	     0.000000
(Ocean advect: MDMDT-TEST)            0.000000	     0.000000
(Ocean advect: horz up)               0.000000	     0.000000
(Ocean advect: vert up)               0.000000	     0.000000
(Ocean advect: horz 2nd)              0.000000	     0.000000
(Ocean advect: vert 2nd)              0.000000	     0.000000
(Ocean advect: gyre_overturn)         0.000000	     0.000000
(Ocean advect: advect diss)           0.000000	     0.000000
(Ocean shortwave penetrate)          15.296601	   103.740304
(Ocean_OFP_init)                      0.000000	     0.000000
(Ocean_OFP_main)                      0.000000	     0.000000
(ocean_OFP_update_1)                  0.000000	     0.000000
(ocean_OFP_update_2)                  0.000000	     0.000000
(ocean_OFP_update_3)                  0.000000	     0.000000
(Ocean diagnostics: adv_vel)         20.063988	    29.820630
(Ocean diagnostics: tracer)          36.385204	    63.796043
(Ocean diagnostics: velocity)         5.724297	     5.353638
(Ocean adv_vel_diag: numerics)        0.251616	     0.151553
(Ocean adv_vel_diag: s-trans)        10.803571	    20.200450
(Ocean adv_vel_diag: rho-trans)       8.979198	     9.454702
(Ocean adv_vel_diag: nrho-trans)      0.003760	     0.003240
(Ocean adv_vel_diag: theta-trans      0.005350	     0.002097
(Ocean tracer_diag: compute_subd      0.002123	     0.002697
(Ocean tracer_diag: compute_trac      0.004265	     0.002968
(Ocean tracer_diag: mld_dtheta)       0.002506	     0.001682
(Ocean tracer_diag: mld)              5.441176	     5.039607
(Ocean tracer_diag: rho_mld)          0.228143	     0.194540
(Ocean tracer_diag: potrho depth      0.000327	     0.000817
(Ocean tracer_diag: theta depth)      0.000333	     0.000657
(Ocean tracer_diag: tracer on rh      0.004796	     0.005565
(Ocean tracer_diag: tracer_zrho       0.003924	     0.002941
(Ocean tracer_diag: numerical)        0.293686	     0.240460
(Ocean tracer_diag: diag mixing       0.001017	     0.001049
(Ocean tracer_diag: conserve)         0.090672	     0.077636
(Ocean tracer_diag: total water       5.859774	    37.781129
(Ocean tracer_diag: total water       1.056526	     0.962110
(Ocean tracer_diag: total tracer     23.361596	    19.463599
(Ocean tracer_diag: integrals)        0.122878	     0.034277
(Ocean tracer_diag: change)           0.141542	     0.067936
(Ocean tracer_diag: land check)       0.007247	     0.027890
(Velocity diag: energy analysis)      1.501711	     1.159672
(Velocity diag: press convert)        0.131108	     0.125394
(Velocity diag: press energy)         0.000000	     0.000000
(Velocity diag: friction energy)      0.000000	     0.000000
(Velocity diag: vert dissipation      0.005410	     0.006689
(Wave model initialization)           0.000000	     0.000000
(Wave diagnostics)                    0.000000	     0.000000
Override                              1.205577	     1.233548
OASIS init                            0.611238	     0.882900
oasis_recv                          168.942581	   126.069609
oasis_send                            1.303476	     1.258477
oasis_recv1                          21.513922	    33.359416
oasis_send1                           0.864890	     0.802798

[manodeep]

And for an apples-to-apples comparison, here's the one between pr98-18 and the released config (both have the exact same config)

# Concatenated output:              ifort           ifx
##############################################################
Total runtime                      4318.405117	 5871.434355
Ocean                              3959.762338	 5548.512027
(Ocean initialization)                3.785997	    4.409954
(Ocean ODA)                           0.000000	    0.000000
(Red Sea/Gulf Bay salinity fix)       0.152466	    0.258358
(Ocean advection velocity)           16.619888	   24.130160
(Ocean density diag)                 15.823220	   33.286870
(Ocean update density)               68.753486	  367.972312
(Ocean vertical mixing coeff)       131.185507	  169.896063
(Ocean neutral physics)             928.926576	 1113.266948
(Ocean viscous form drag)             2.400294	    2.254454
(Ocean submesoscale restrat)        251.978845	  301.252751
(Ocean shortwave)                    23.124541	  135.143752
(Ocean sponges_eta)                   0.003970	    0.005314
(Ocean sponges_tracer)                0.005306	    0.003716
(Ocean sponges_velocity)              0.005444	    0.003361
(Ocean xlandinsert)                   0.006231	    0.005778
(Ocean xlandmix)                      0.005684	    0.004111
(Ocean rivermix)                     17.155149	   20.544609
(Ocean overexchange)                  0.009112	    0.007900
(Ocean mixdownslope)                101.305000	  122.529088
(Ocean blob update)                   0.003193	    0.004367
(Ocean blob cell update)              0.000000	    0.000000
(Ocean blob diagnose depth)           0.002386	    0.002299
(Ocean overflow)                      0.516753	    3.817419
(Ocean overflow_OFP)                  0.006695	    0.010635
(Ocean sigma transport)              32.223843	   31.694642
(Ocean tracer update)               411.899566	  514.668082
(Ocean surface flux)                 41.250250	  170.869394
(Ocean bottom flux)                   0.510496	    0.546963
(Ocean restoring flux)                0.654708	    0.702409
(Ocean TPM source)                    1.032742	    0.583875
(Ocean TPM bbc)                       0.001546	    0.002469
(Ocean TPM tracer)                  840.582128	  998.799832
(Ocean explicit accel_a)            164.165828	  206.738762
(Ocean explicit accel_b)              3.045653	    3.359721
(Ocean implicit accel)               18.115767	   22.291192
(Ocean eta and pbot tendency)         0.106819	    0.103299
(Ocean eta and pbot update)           0.528586	    0.331578
(Ocean eta and pbot diagnose)        34.682037	   33.988874
(Ocean rho_dzt tendency)              1.829594	    1.752537
(Ocean dzt_dst update)                0.000000	    0.000000
(Ocean surface height smooth)        15.379911	   16.403004
(Ocean bottom pressure smooth)        0.000000	    0.000000
(Ocean mass forcing)                  1.056531	    1.069225
(Ocean barotropic forcing)            2.190875	    4.598512
(Ocean barotropic dynamics)          46.788131	   89.310285
(Ocean velocity update)              15.423497	   20.502249
(Ocean diagnostics)                  62.194185	  119.125983
(Ocean update T-cell thickness)      16.264056	   22.902288
(Ocean update Total thickness)        0.000000	    0.000000
(Ocean update L sys. thickness)       0.000000	    0.000000
(Ocean update E sys. thickness)       0.000000	    0.000000
(Ocean update U-cell thickness)      16.961868	   22.041542
(Ocean tracer halo updates)         275.070814	  334.391999
(Ocean velocity halo update)          1.527814	    1.537316
(Ocean sum ocean surface)            88.946095	  143.525484
(Ocean average state)                 0.029699	    0.078722
(Ocean tracer tmask limit)           22.219161	  105.134348
(Ocean gotm: advection)               0.000000	    0.000000
(Ocean increment eta)                 0.003835	    0.003805
(Ocean increment tracer)              0.003956	    0.002937
(Ocean increment velocity)            0.005122	    0.003847
(Ocean update rho_salinity)           0.201834	    0.178211
(Ocean idealized surface waves)       0.004707	    0.005579
(Ocean open boundaries)               0.000000	    0.000000
(Ocean tracer: bih tracer)            0.017899	    0.016261
(Ocean tracer: lap tracer)            0.016531	    0.021999
(Ocean tracer: vert diffuse)         16.167541	   10.857745
(Ocean tracer: vert diffuse impl     29.204165	   41.730564
(Ocean tracer: advection)           233.915603	  315.687941
(Ocean tracer: frazil)                3.400824	    3.062851
(Ocean tracer: convection)            0.003087	    0.003709
(Ocean tracer: Lagrangian blobs)      0.030037	    0.025518
(Ocean tracer: implicit blobs)        0.006552	    0.010078
(Ocean tracer: adjust the L thic      0.000327	    0.000737
(Ocean tracer: adjust the E thic      0.006619	    0.006478
(Ocean CT2PT: 1)                      0.798708	    2.001941
(Ocean CT2PT: 2)                      0.000000	    0.000000
(Ocean CT2PT: 3)                      0.000000	    0.000000
(Ocean generic tracer: vertdiff)     55.964717	   47.794889
(Ocean generic tracer: source)      781.969017	  948.199304
(Ocean generic tracer: bottom up      0.368527	    0.548626
(Ocean generic tracer: diag)          0.014033	    0.014604
(Ocean generic tracer: set_value      0.000000	    0.000000
(Ocean generic tracer: get_value      0.284771	    0.223790
(Ocean generic tracer: sumsfcset      0.000000	    0.000000
(Ocean vmix: constant)                0.000000	    0.000000
(Ocean vmix: PPvmix)                  0.000000	    0.000000
(Ocean vmix: KPP_test)                0.000000	    0.000000
(Ocean vmix: KPP_mom4p0)              0.000000	    0.000000
(Ocean vmix: KPP_mom4p1)             81.194335	  105.206635
(Ocean vmix: chenvmix)                0.000000	    0.000000
(Ocean vmix: gotmvmix)                0.000000	    0.000000
(Ocean vmix: tidal)                  45.359259	   59.886581
(Ocean vmix: watermass_diag)          0.007820	    0.009338
(Ocean neutral: tracer derivs)       89.173714	   95.930599
(Ocean neutral: slopes)              12.126679	    8.926513
(Ocean neutral: eady rate)           21.248923	   27.232284
(Ocean neutral: baroclinic)           0.049023	    0.041964
(Ocean neutral: rossby radius)        2.289100	    2.481390
(Ocean neutral: bc zone radius)       0.000000	    0.000000
(Ocean neutral: diffusivity)          7.315881	    7.380261
(Ocean neutral: cabbel/thermob)       0.000000	    0.000000
(Ocean neutral: eta_tend_gm90)        0.000000	    0.000000
(Ocean neutral: nrho-trans)           0.000000	    0.000000
(Ocean neutral: rho-trans)            0.000000	    0.000000
(Ocean neutral: theta-trans)          0.000000	    0.000000
(Ocean neutral: diffuse)            345.828021	  468.340973
(Ocean neutral: GM stir)            310.308989	  310.672825
(Ocean neutral: blayer)               3.312345	    9.905124
(Ocean neutral: fz-terms)            42.556532	   70.645529
(Ocean ndiffuse: fx-flux)            64.232973	  122.913312
(Ocean ndiffuse: fy-flux)            59.803690	  122.104899
(Ocean ndiffuse: fz-flux)            77.798731	   89.647065
(Ocean gm: fx-flux)                  38.622291	   38.087659
(Ocean gm: fy-flux)                  40.460363	   38.560185
(Ocean gm: fz-flux)                  64.474251	   76.318427
(Ocean gm: bc_modes)                  9.069349	   10.505511
(Ocean gm: psi_modes)                 0.000000	    0.000000
(Ocean gm: psi_bvp)                  76.884308	   91.931388
(Ocean const   lap frict)             0.000000	    0.000000
(Ocean general lap frict)            44.110204	   65.998234
(Ocean C-grid lap frict)              0.000000	    0.000000
(Ocean const   bih frict)             0.000000	    0.000000
(Ocean general bih frict)            79.353581	   92.792880
(Ocean c-grid  bih frict)             0.000000	    0.000000
(Ocean advect: horz 4th)              0.000000	    0.000000
(Ocean advect: vert 4th)              0.000000	    0.000000
(Ocean advect: horz 6th)              0.000000	    0.000000
(Ocean advect: vert 6th)              0.000000	    0.000000
(Ocean advect: horz quk)              0.000000	    0.000000
(Ocean advect: vert quk)              0.000000	    0.000000
(Ocean advect: horz qukmom3)          0.000000	    0.000000
(Ocean advect: vert qukmom3)          0.000000	    0.000000
(Ocean advect: MDFL-sup-b)            0.000000	    0.000000
(Ocean advect: MDFL-sweby)            0.000000	    0.000000
(Ocean advect: MDFL-sweby-all)      233.852472	  315.623691
(Ocean advect: MDFL-sweby-test)       0.000000	    0.000000
(Ocean advect: DST-linear)            0.000000	    0.000000
(Ocean advect: DST-linear-test)       0.000000	    0.000000
(Ocean advect: MDFL-sweby-mpi)       33.666781	   25.242759
(Ocean advect: MDFL-sweby-cuk)       33.480322	   56.465580
(Ocean advect: MDFL-sweby-cui)       37.223727	   73.757554
(Ocean advect: MDFL-sweby-cuj)       62.110208	   88.187185
(Ocean advect: MDFL-sweby-diag)      24.351281	   42.558050
(Ocean advect: MDPPM)                 0.000000	    0.000000
(Ocean advect: MDPPM-TEST)            0.000000	    0.000000
(Ocean advect: psom total)            0.000000	    0.000000
(Ocean psom advect: psom_x)           0.000000	    0.000000
(Ocean psom advect: psom_y)           0.000000	    0.000000
(Ocean psom advect: psom_z)           0.000000	    0.000000
(Ocean advect: MDMDT-TEST)            0.000000	    0.000000
(Ocean advect: horz up)               0.000000	    0.000000
(Ocean advect: vert up)               0.000000	    0.000000
(Ocean advect: horz 2nd)              0.000000	    0.000000
(Ocean advect: vert 2nd)              0.000000	    0.000000
(Ocean advect: gyre_overturn)         0.000000	    0.000000
(Ocean advect: advect diss)           0.000000	    0.000000
(Ocean shortwave penetrate)          15.296601	  118.361038
(Ocean_OFP_init)                      0.000000	    0.000000
(Ocean_OFP_main)                      0.000000	    0.000000
(ocean_OFP_update_1)                  0.000000	    0.000000
(ocean_OFP_update_2)                  0.000000	    0.000000
(ocean_OFP_update_3)                  0.000000	    0.000000
(Ocean diagnostics: adv_vel)         20.063988	   35.376108
(Ocean diagnostics: tracer)          36.385204	   76.859497
(Ocean diagnostics: velocity)         5.724297	    6.869305
(Ocean adv_vel_diag: numerics)        0.251616	    0.275093
(Ocean adv_vel_diag: s-trans)        10.803571	   23.532312
(Ocean adv_vel_diag: rho-trans)       8.979198	   11.547501
(Ocean adv_vel_diag: nrho-trans)      0.003760	    0.003164
(Ocean adv_vel_diag: theta-trans      0.005350	    0.003328
(Ocean tracer_diag: compute_subd      0.002123	    0.001165
(Ocean tracer_diag: compute_trac      0.004265	    0.003429
(Ocean tracer_diag: mld_dtheta)       0.002506	    0.001313
(Ocean tracer_diag: mld)              5.441176	    6.416748
(Ocean tracer_diag: rho_mld)          0.228143	    0.189011
(Ocean tracer_diag: potrho depth      0.000327	    0.000359
(Ocean tracer_diag: theta depth)      0.000333	    0.000316
(Ocean tracer_diag: tracer on rh      0.004796	    0.004484
(Ocean tracer_diag: tracer_zrho       0.003924	    0.003673
(Ocean tracer_diag: numerical)        0.293686	    0.282044
(Ocean tracer_diag: diag mixing       0.001017	    0.000526
(Ocean tracer_diag: conserve)         0.090672	    0.101493
(Ocean tracer_diag: total water       5.859774	   47.049313
(Ocean tracer_diag: total water       1.056526	    1.052814
(Ocean tracer_diag: total tracer     23.361596	   21.728327
(Ocean tracer_diag: integrals)        0.122878	    0.042285
(Ocean tracer_diag: change)           0.141542	    0.093119
(Ocean tracer_diag: land check)       0.007247	    0.034567
(Velocity diag: energy analysis)      1.501711	    1.515072
(Velocity diag: press convert)        0.131108	    0.157040
(Velocity diag: press energy)         0.000000	    0.000000
(Velocity diag: friction energy)      0.000000	    0.000000
(Velocity diag: vert dissipation      0.005410	    0.004736
(Wave model initialization)           0.000000	    0.000000
(Wave diagnostics)                    0.000000	    0.000000
Override                              1.205577	    1.411534
OASIS init                            0.611238	    0.624738
oasis_recv                          168.942581	  124.792559
oasis_send                            1.303476	    1.726678
oasis_recv1                          21.513922	   38.172943
oasis_send1                           0.864890	    0.969744