Update EKAT to require C++20

[tcclevenger]

Update EKAT to require C++20

Motivation

Needed to update to Kokkos 5.0.

Testing

CI already uses C++20.

[bartgol]

Asking the obv question: are we sure that E3SM will build everywhere with C++20 required? If not, this my hold back any ekat update in e3sm.

[rljacob]

It will not build with all of our current default compilers. In particular, the "intel" compiler on Chrysalis. Possibly others.

[bartgol]

Looks good. Only possible concern is whether e3sm is ready for c++20 across all machines.

[tcclevenger]

@bartgol @rljacob Good to know. I created this PR while I'm testing E3SM on the target machines, but since we already know there will be some that are not compatible, we will need to find a different way to handle this.

Kokkos 5 is going to require C++20, what would the process be to eventually update compilers on Chrysalis (and maybe others) so that we can upgrade? Or is this going to be a bigger roadblock? I guess the first step is for me to test exactly which compilers do not work.

[bartgol]

One solution would be to mis with a cmake2.0 way of specifying the std, via something like

if (NOT CMAKE_CXX_STANDARD)
  target_compile_features(ekat_core PUBLIC cxx_std_20)
endif()

Then, in the mach files for old compilers we keep lines like

set (CMAKE_CXX_STANDARD 17)

which will force ekat to honor the required standard. Of course, for machines where we do set 17 in the mach files, we won't be able to build anything that needs kokkos 5.0 (e.g. any F case, since EAM still use kokkos for the SL transport). This means that we cannot switch to kokkos 5 until we can abandon compilers that only support c++17.

[tcclevenger]

On Chrysalis:

• gnu: default compiler is compatible
• oneapi-ifx: version is technically compatible, but we get a build error about not being able to find "concepts" file. Output indicates oneAPI is not correctly configured.
• intel: Kokkos no longer supports INTEL's classic compilers (as of Kokkos 4.6, we currently use Kokkos 4.5). Maybe they are C++20 compliant, but we will need to change anyways to update Kokkos.

[bartgol]

• oneapi-ifx: version is technically compatible, but we get a build error about not being able to find "concepts" file. Output indicates oneAPI is not correctly configured.

Ugh, that's underwhelming. I wonder if this is on the chrys IT folks' radar...

[tcclevenger]

Ugh, that's underwhelming. I wonder if this is on the chrys IT folks' radar...

Yeah, I want to try and make a small reproducer outside of E3SM. Make sure it's not something we are doing.

[tcclevenger]

Update:

• PM-CPU/GPU and Aurora: Ran some tests from nightly, C++20 builds and runs BFB with master.
• Chrysalis_oneapi-ifx: Az update the oneapi-ifx compiler and I've tested against master baselines and C++20 builds and runs BFB (yay!)
• Chrysalis_intel: No longer supported in Kokkos 5, I think current one we are using isn't c++20 compatible.
• Chrysalis_gnu: technically compliant gnu version, but internal compiler error with C++20 (somewhere in ekat_pack)

@rljacob @bartgol How should we proceed? Can we require users of Chrysalis start switching to oneapi? Do we need to have a frozen version of Kokkos as a tpl for the other compilers? Could we require anyone using ekat (running with EAMxx) use oneapi?

[bartgol]

• PM-CPU/GPU and Aurora: Ran some tests from nightly, C++20 builds and runs BFB with master.

Did you run both gnu and intel on pm-cpu? I think @ndkeen is also using nvidia as compiler on pm-gpu, so maybe we should check that too.

@rljacob @bartgol How should we proceed? Can we require users of Chrysalis start switching to oneapi? Do we need to have a frozen version of Kokkos as a tpl for the other compilers? Could we require anyone using ekat (running with EAMxx) use oneapi?

I think we can keep both intel and oneapi. But we won't be able to run anything that uses kokkos with the intel compiler. I think this includes also any EAM case though, since EAM uses hommexx's SL transport by default, so this may impact a lot of ppl.

@jgfouca wild option: can CIME update a submodule conditionally on the compiler (or any other XML var)? E.g., can we do something like

if compiler is "intel":
   update_submodule(f'{SRCROOT}/externals/ekat', 'v1.1', recursive=True)  

to switch the submodule to an older tag?

[rljacob]

We will require Chrysalis users to use oneapi-ifx. It will be made the default. ifort will still be an option for anyone who needs it and they have to use an earlier hash.

We should make a plan on when this goes to master since its a breaking change for chrysalis.

[bartgol]

We will require Chrysalis users to use oneapi-ifx. It will be made the default. ifort will still be an option for anyone who needs it and they have to use an earlier hash.

We should make a plan on when this goes to master since its a breaking change for chrysalis.

We should prob make this change first, and then update ekat/kokkos (and the required cxx standard). Maybe mid-late March? I think 2 months may be a good enough window for ppl to switch to the new compiler.

[jgfouca]

@bartgol , CIME does not support that currently. We could, in theory, add support for configuration that allows for customizing submodules, but that seems kinda hacky.

[tcclevenger]

@rljacob @bartgol I'll do some runs with EAM cases to confirm that EAM will still need to compile Kokkos, then make a PR updating the default compiler.

[tcclevenger]

Testing update

• PM-CPU
  • intel: SMS_Lh4.ne4pg2_ne4pg2.F2010-SCREAMv1.pm-cpu_intel.eamxx-output-preset-1--eamxx-prod is BFB with master baseline
  • gnu: SMS_Lh4.ne4pg2_ne4pg2.F2010-SCREAMv1.eamxx-output-preset-1--eamxx-prod is BFB with master baseline
  • nvidia: SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.eamxx-output-preset-3 runs, but no baselines exist (I'm generating my own master baseline)
• PM-GPU:
  • gnugpu: SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.eamxx-output-preset-3 is BFB with master baseline
  • nvidiagpu: No baselines, CDASH only runs e3sm_gpuacc_next_nvidiagpu, which passes

I will next go to frontier, then I think we will have all the info we need to move forward.

[tcclevenger]

Re: intel compiler on chrysalis

Confirmed, the F2010 test case still requires ekat and does not work with C++20. I'll create a PR in E3SM switching the default compiler to oneapi on chrysalis. Not sure how we want to deal with trying to have an old version of Kokkos available vs. trying to require everyone switch compilers.

[bartgol]

Re: intel compiler on chrysalis

Confirmed, the F2010 test case still requires ekat and does not work with C++20. I'll create a PR in E3SM switching the default compiler to oneapi on chrysalis. Not sure how we want to deal with trying to have an old version of Kokkos available vs. trying to require everyone switch compilers.

I'm in favor of requiring ppl to switch. The master branch should always follow the latest development. If ppl need older compiler, they can use tags and swtich to older tags.

@rljacob may have other thoughts though...

[rljacob]

In your switch, will you also make a older pre-built version of Kokkos availble for ifort users? There are in-progress production runs with ifort and its not clear yet if they can switch. Also CCing @amametjanov

[bartgol]

In your switch, will you also make a older pre-built version of Kokkos availble for ifort users? There are in-progress production runs with ifort and its not clear yet if they can switch. Also CCing @amametjanov

Are the in-progress prod runs using version-of-the-day master? Or are they working on a particular tag? Imho, every production campaign should work on a tag, rather than use current master (which can also undergo non-bfb changes).

[tcclevenger]

In your switch, will you also make a older pre-built version of Kokkos availble for ifort users? There are in-progress production runs with ifort and its not clear yet if they can switch. Also CCing @amametjanov

I agree with Luca that any campaigns I think should be on a tag, and so my preference would be to not have a separate Kokkos, unless there was major pushback.

[tcclevenger]

As far as different compilers, intel won't work for F2010 or F2010-SCREAMv1, and right now gnu has an internal compiler error (ice) inside EKAT. Can we just put oneapi-ifx as the default and say "good luck" to anyone using gnu like we would with intel, or do we need to try and update gnu or solve the ice for gnu?

[rljacob]

the gnu compiler on Chrysalis is used for ice sheet model (MALI) development. Either they need to also switch to oneapi or yes the gnu issue has to be solved.

[bartgol]

the gnu compiler on Chrysalis is used for ice sheet model (MALI) development. Either they need to also switch to oneapi or yes the gnu issue has to be solved.

One big reason to bump to c++20 is b/c MALI would like to follow trilinos more closely. So I think they'd be ok with switching. Pinging @mperego.

Another option could be to bump gnu's version on chrys from 11 to something like 13 (which I think is what pm-cpu uses?).

[rljacob]

OMEGA also gets its Kokkos from EKAT so may need to chime in on this. @amametjanov should know what compilers/machines matter for OMEGA.

[mperego]

the gnu compiler on Chrysalis is used for ice sheet model (MALI) development. Either they need to also switch to oneapi or yes the gnu issue has to be solved.

One big reason to bump to c++20 is b/c MALI would like to follow trilinos more closely. So I think they'd be ok with switching. Pinging @mperego.

Another option could be to bump gnu's version on chrys from 11 to something like 13 (which I think is what pm-cpu uses?).

I support moving to c++20, but it might be a problem for MALI if we are moving to c++20 but not updating Kokkos at the same time. Also, while in principle it should be possible to build MALI with oneapi we haven't tried yet. What's the timeline for this change?

[bartgol]

I support moving to c++20, but it might be a problem for MALI if we are moving to c++20 but not updating Kokkos at the same time.

Why?

Also, while in principle it should be possible to build MALI with oneapi we haven't tried yet. What's the timeline for this change?

No precise timeline. Soon-ish? I would say a month or two at most. If MALI only uses gcc, we can work on making gcc work. But we may have to upgrade the compiler version, which requires some work (it will inevitably diff).

[mperego]

@bartgol Trilinos switched to c++20 essentially at the same time it switched to Kokkos 5.0. If E3SM is using an older version of Kokkos, chances are that the version of Trilinos that is compatible with E3SM Kokkos does not supprt c++20. Everything is doable, but it could create some extra hiccups.

A month should be enough time, assuming we have an E3SM branch working reasonably welll with the new compilers.

[rljacob]

The new compiler is on master already. Add "--compiler oneapi-ifx" to your create_newcase on Chrysalis.

[tcclevenger]

@bartgol Trilinos switched to c++20 essentially at the same time it switched to Kokkos 5.0. If E3SM is using an older version of Kokkos, chances are that the version of Trilinos that is compatible with E3SM Kokkos does not supprt c++20. Everything is doable, but it could create some extra hiccups.

I will do a round of testing with Kokkos 5 and C++20 update on the various machines, and if needed we could do that all in one go. Considering that we will be using the legacy implementation of views in Kokkos 5 initially, I don't think we should have many other issues updating Kokkos since 4.5 and 5.0 are only 2 minor versions apart, with c++20 being the biggest other change.

[bartgol]

@mperego that makes sense. As Conrad said, we can do both std and kk upgrades in one sweep. I thought it was simpler to do the cxx20 upgrade first, but for Trilinos it may indeed cause more hiccups. Given that, I am in favor of a combo upgrade, where we both upgrade kokkos and cxx standard (well, the latter is implied by kk 5 anyways).

[tcclevenger]

I've created a new PR #410 for C++20 and Kokkos5 combo update. I'll give machine testing updates there.

[rljacob]

What we should do is first update compilers to C++20 and work out any kinks with existing code/testing. Then do the updates to Ekat/Kokkos/Trilinos/MALI/OMEGA. A test branch that does both is fine but I think this should be 2 PRs to E3SM master.

[rljacob]

Made a page to summarize status: https://e3sm.atlassian.net/wiki/spaces/IPD/pages/5917081627/Upgrade+to+C+20+OneAPI

[bartgol]

What we should do is first update compilers to C++20 and work out any kinks with existing code/testing. Then do the updates to Ekat/Kokkos/Trilinos/MALI/OMEGA. A test branch that does both is fine but I think this should be 2 PRs to E3SM master.

What mauro suggested is that using C++20 with existing trilinos/albany installations may not work. We're not 100% sure, so I guess we could try, but it is entirely possible that building an old trilinos (like the one we currently have installed on tgt machines) while forcing c++20 std may not work. This is why Mauro suggested to do both at the same time. I think it's fine to do the jump while we also upgrade the std.

[rljacob]

When we introduce the new compiler suites, the old ones will still be around and, because Kokkos hasn't been updated yet, they could still build any case with the old ones. Then we introduce the new Kokkos and any other library updates plus a switch to the new compiler for anyone that hasn't done it yet.

[tcclevenger]

When we introduce the new compiler suites, the old ones will still be around and, because Kokkos hasn't been updated yet, they could still build any case with the old ones. Then we introduce the new Kokkos and any other library updates plus a switch to the new compiler for anyone that hasn't done it yet.

Sorry for the confusion, this PR is now out of date, and the new PR is just from a branch that I have to update Kokkos and C++20 in EKAT. I'm using it to test all the E3SM compilers so that I know exactly which ones will work with Kokkos 5, but my plan would still be to merge any compiler default updates/testing updates in E3SM before merging Kokkos 5 update.

Made a page to summarize status: https://e3sm.atlassian.net/wiki/spaces/IPD/pages/5917081627/Upgrade+to+C+20+OneAPI

I'll update that page with Kokkos 5 status