SMIOL error

[gsketefian]

Hi,

I'd like to report an error message I'm getting when running certain hashes of MPAS-Model's gsl/develop branch. The error message is related to SMIOL and generally has this form:

ERROR: SMIOLf_define_dim failed with error -2
ERROR: invalid subroutine argument
ERROR: MPAS IO Error: SMIOL error -2: invalid subroutine argument

I started seeing this after upgrading from hash 9ce949e (PR #161) to hash 4d97971 (PR #168). There are several commits between these, so I ran with different ones to see where the error message first shows up. It turns out it's caused by the commit immediately after 9ce949e, which is 3cb859c (PR #121).

In all cases, I am building and running MPAS via the mpas_app. The base directory I'm working under is

/scratch4/BMC/fv3lam/Gerard.Ketefian/DTC_MPAS_stoch/gsk_code_Ning_latest

Under this, you can find the code and a sample experiment for each of the two hashes in the following locations:

Hash 9ce949ec (this does not generate the error)
------------------------------------------------
mpas_app location:                   mpas_app.Anders_9ce949ec
MPAS-Model location:                 mpas_app.Anders_9ce949ec/src/MPAS-Model
mpas_app experiment location:        expt_dirs/conus_15km.forecast_only.Anders_9ce949ec
forecast location under experiment:  expt_dirs/conus_15km.forecast_only.Anders_9ce949ec/2023091500/forecast

Hash 3cb859cb (this does generate the error)
--------------------------------------------
mpas_app location:                   mpas_app.Haiqin_3cb859cb
MPAS-Model location:                 mpas_app.Haiqin_3cb859cb/src/MPAS-Model
mpas_app experiment location:        expt_dirs/conus_15km.forecast_only.Haiqin_3cb859cb
forecast location under experiment:  expt_dirs/conus_15km.forecast_only.Haiqin_3cb859cb/2023091500/forecast

You can see the error in these log files:

/scratch4/BMC/fv3lam/Gerard.Ketefian/DTC_MPAS_stoch/gsk_code_Ning_latest/expt_dirs/conus_15km.forecast_only.Haiqin_3cb859cb/2023091500/forecast/log.atmosphere.00*.err

Also, I noticed that the history*.nc files in these two runs have very different sizes. The ones from hash 9ce949e (no error message) are 942 mb, while the ones from hash 3cb859c (with error message) are only 67 mb. I suspect this is likely due to the error message, but I haven't dug into it to find out for sure.

I wonder if anyone else has seen this error and whether the latest commit fixes it (or whether there is a fix in the works).

Thanks.
@clark-evans @JeffBeck-NOAA I could not tag Will and Ning in this issue. They'll probably be interested in this. Do we need to/Can we add them to the ufs-community organization?

[clark-evans]

Thanks for opening this issue, @gsketefian. I'm tagging @jordanschnell and @haiqinli for their awareness.

A few thoughts:
(1) Our only experience with MPAS App this year has been with the 2025 HFIP configuration, which uses an earlier model hash. I don't believe @SamuelTrahanNOAA encountered any of these SMIOL issues with that hash. I'm not sure if Sam has the time/capacity to update the MPAS-Model hash on disk on ursa and run a quick test with the last real-time forecast cycle to see if the issue reproduces there.
(2) Our only real-time system (at least on the physics end) that is at or beyond the offending hash is dev2, which is currently running v8.3.1-2.1 plus PR 169. I don't believe we're seeing these issues in those runs. Though I don't think PR 169 should affect this, as a quick test, could you try adding in PR 169 to your code base and seeing if the issue persists?
(3) Out of curiosity, have you tried running this model hash outside of MPAS App (e.g., by manually setting the namelist and submitting the job through slurm)? If so, does the issue persist there?

I can add Will and Ning to this repo; could you please pass along their Github usernames?

[gsketefian]

@clark-evans Thank you for the suggestions. I will try adding PR #169.

I have not tried outside of mpas_app, but I think Will runs it without mpas_app, so I would like to ask him if he also sees the error. Also, I assume after Ning made his final set of changes for SPPT, he ran it, and I forgot to ask if he saw this error message. I'd like to ask him that (he runs with mpas_app).

Will's and Ning's github usernames are @willmayfield and @NingWang325. Thanks.

[SamuelTrahanNOAA]

To switch hashes, I may have to update the namelist and change the physics. Experimental physics changes probably won't be in that hash. Also, namelist options often change from version to version, and may need to be updated.

[clark-evans]

Tagging myself and @gsketefian as a reminder to revisit this at the Jan 15 stochastic physics meeting.

[gsketefian]

Here's an update.

I found that changing the name of the stream named "output" in registry.chemistry.xml (in the directory MPAS-Model/src/core_atmosphere/physics/), e.g. to "smoke_output", gets rid of the SMIOL error. Just to be sure, I also renamed the output file for what is now the smoke_output" stream to "history_smoke.$Y-$M-$D_$h.$m.$s.nc" (originally it was "history.$Y-$M-$D_$h.$m.$s.nc").

I think the issue is that the "output" stream in registry.chemistry.xml conflicts with the existing "output" stream in Registry.xml (in the directory MPAS-Model/src/core_atmosphere/Registry.xml`).

My testing was with hash 3cb859c. I'm still not sure why others don't see this error since I don't think it's been corrected as of the writing of this message. To move on with the DTC stochastic physics work, I will likely commit this stream name change to the our (the DTC Stochastic Physics project's) working branch in the dtcenter fork of MPAS-Model. It seems like a good idea to change it here in the authoritative repo as well, but we can discuss that later when we merge our branch into the authoritative repo.

@haiqinli Any thoughts on this? Thanks.

@JeffBeck-NOAA @willmayfield @NingWang325 @clark-evans @haiqinli

[clark-evans]

Nudging for @haiqinli

[haiqinli]

Nudging for @haiqinli

We (me, @jordanschnell and @guoqing-noaa ) will discuss about this, and will update soon. Thanks!

[guoqing-noaa]

My understanding is that MPAS-Model will combine two "output" sections together automatically.
We tested this in our retros and realtime-like experiments (conus3km, conus12km, 1km, gaea and ursa) and did not find any issues so far.

@gsketefian Could you test whether it works if you manually combine the chem output section into the regular output section in the Registry.xml files? We would like to understand which caused this issue, the chem variables or the stream setting?

Writing out chem variables into a separate file will induce lots of downstream changes, including a needed major update to MPASSIT. I suggest we try our best to identify the root cause before we move to that route. Thanks!