Refactor hierarchical parallelism wrappers

[mwarusz]

This PR refactors Omega wrappers of Kokkos hierarchical parallelism. Specifically, it:

• Adds a new way of specifying ranges of inner loops through a helper struct Range:

   parallelForInner(Team, Range{Start, End}, INNER_LAMBDA ...
  

  This allows us to remove most uses of vertRange(KMin, KMax) with manual offsetting of loop indices.
• Changes how team scratch memory is requested. This is now done through helper structs LaunchConfig and TeamScratch. For example

   parallelForOuter(LaunchConfig({N}, TeamScratch<Real>(NScratch)), KOKKOS_LAMBDA ...
  

  This has the following benefits:
  • Use of scratch memory is more visible, compared to a trailing int parameter.
  • You can request scratch memory of different types: TeamScratch<Real, int>(NScratchReal, NScratchInt).
  • You can use scratch memory in parallelReduceOuter.
  • LaunchConfig serves as a general customization point for outer loops. For example, you can also request a different team size.
• For better code organization, moves the wrappers out of OmegaKokkos.h into their own header files: OmegaKokkosFlatPar.h for flat parallelism and OmegaKokkosHiPar.h for hierarchical parallelism. OmegaKokkos.h now mostly contains general purpose Kokkos utilities.
• Adds parallelSearchInner for searching an index range in parallel at the inner level of hierarchical parallelism.
• Adds documentation and tests for the new features.
• With help from copilot, changes existing code to make use of the new features.

Checklist

• Documentation:
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
• Linting
  • Pre-commit run locally
• Building
  • CMake build does not produce any new warnings from changes in this PR
• Testing
  • Add a comment to the PR titled Testing with the following:
    • Which machines CTest unit tests
      have been run on and indicate that are all passing.
    • The Polaris omega_pr test suite
      has passed, using the Polaris e3sm_submodules/Omega baseline
    • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
    • Indicate "All tests passed" or document failing tests
  • New tests:
    • CTest unit tests for new features have been added per the approved design.

[mwarusz]

Testing

CTest unit tests

• Machine: aurora
• Compiler: oneapi-ifx
• Build type: Release
• Result: All 39 tests passed
• Log: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/hipar-update/build-ifx/Testing/Temporary/LastTest.log

• Machine: frontier
• Compiler: craygnu-mphipcc
• Build type: Release
• Result: All 39 tests passed
• Log: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/hipar-update/build-craygnu-mphipcc/Testing/Temporary/LastTest.log

Polaris omega_pr suite

• Baseline workdir: /pscratch/sd/m/mwarusz/omega-pr-testing/hipar-update/baseline-gnugpu/
• Baseline build: /pscratch/sd/m/mwarusz/omega-pr-testing/hipar-update/baseline-gnugpu/build
• PR build: /pscratch/sd/m/mwarusz/omega-pr-testing/hipar-update/build-gnugpu
• PR workdir: /pscratch/sd/m/mwarusz/omega-pr-testing/hipar-update/pr-gnugpu
• Machine: pm-gpu
• Compiler: gnugpu
• Build type: <Debug|Release>
• Log: /pscratch/sd/m/mwarusz/omega-pr-testing/hipar-update/pr-gnugpu/polaris_omega_pr.o50955477
• Result: All tests passed