diff --git a/README.md b/README.md index 6e6f78e6..d1a1bc0c 100644 --- a/README.md +++ b/README.md @@ -1,83 +1,194 @@ -# EKAT Version 1.0: E3SM Kokkos Applications Toolkit +# EKAT: E3SM Kokkos Applications Toolkit -EKAT is a small library, containing tools and libraries for writing Kokkos-enabled HPC C++ in the E3SM ecosystem. -Tools include C++/Fortran code (data structures, functions, macros) as well as CMake scripts. +EKAT is a C/C++ library providing tools and utilities for writing Kokkos-enabled +HPC applications in the E3SM ecosystem. It includes data structures, algorithms, +logging, YAML parsing, and CMake scripts that simplify building and testing +performance-portable code. -Please, read our [guidelines](https://github.com/E3SM-Project/EKAT/blob/master/.github/CONTRIBUTING.md) +Please read our [guidelines](https://github.com/E3SM-Project/EKAT/blob/master/.github/CONTRIBUTING.md) if you are interested in contributing to EKAT. -This project is licensed under the terms of the BOOST license. See LICENSE.txt for details. +This project is licensed under the terms of the BOOST license. See LICENSE.txt +for details. -# Overview +# Sub-packages -Despite being first created with the goal of being a common implementation for common Kokkos-related kernels, -EKAT evolved to contain a variety of C++ and CMake utilities. Here is a list of some of the utilities provided -in EKAT. +EKAT is organized into the following sub-packages. **Core** is always enabled; +the remaining packages are opt-in (see [Configuring EKAT](#configuring-ekat)). -- C++ structures to enhance vectorization (Pack and Mask). -- C++ structures for kernel-local scratch memory (Workspace). -- C++ interface to read and store content of yaml files. -- C++ routines for team-level solvers for diagonally dominant tridiagonal systems. -- C++ routines for team-level efficient linear interpolation of 1d data. -- Backport of some C++17 features (such as std::any), and several std-like meta-utilities. -- CMake utilities for creating a suite of unit tests for multiple MPI/OpenMP rank/threads combinations. -- CMake machine files to configure Kokkos for common architectures and threading framework. +| Package | CMake target | Description | +|---------|--------------|-------------| +| Core | `ekat::Core` | Foundational utilities: assertions, MPI communicator wrapper, parameter lists, string utilities, scalar traits, std-like meta-utilities, units, and more. Always enabled. | +| KokkosUtils | `ekat::KokkosUtils` | Kokkos-specific utilities: team policies, views, subviews, workspace (kernel-local scratch memory), math helpers, and kernel-level assertions. | +| Pack | `ekat::Pack` | SIMD-friendly `Pack` and `Mask` types to enhance auto-vectorization of loops over short arrays. | +| Algorithm | `ekat::Algorithm` | Kokkos team-level algorithms: linear interpolation of 1D data and solvers for diagonally dominant tridiagonal systems. | +| Expression | `ekat::Expression` | Expression-template infrastructure for lazy evaluation of array operations over Kokkos views. | +| Logging | `ekat::Logging` | spdlog-based logging utilities with MPI-aware and file-sink policies. | +| Parser | `ekat::Parser` | YAML file reader built on top of yaml-cpp, exposing parsed content as an `ekat::ParameterList`. | + +There is also a **TestingSupport** component (always built alongside the enabled +packages) that provides Catch2-based test helpers used by EKAT's own test suite +and available to downstream projects. # Configuring EKAT -EKAT uses CMake to generate the build system. To make life easier, we provide some machine files -for the setup of Kokkos properties, located in cmake/machine-files/kokkos. In particular, we provide -machine files for the architecture specs, as well as machine files for the threading framework. -In cmake/machine-files you can see some example o f how to combine architecture/threading machine -files into a single one, which can later be used in your cmake configuration script. +EKAT uses CMake (version 3.18 or higher) to generate its build system. Only C +and C++ (C++17) are required; no Fortran compiler is needed. -For instance, the following `my-mach.cmake` file combines intel Skylake arch specs with a OpenMP theading backend. +## Kokkos machine files -``` +To make it easier to configure Kokkos for a specific machine, EKAT ships a +collection of cmake *machine files* under `cmake/machine-files/`: + +- `cmake/machine-files/kokkos/` — architecture and threading back-end files + (e.g., `openmp.cmake`, `cuda.cmake`, `nvidia-v100.cmake`, `intel-skx.cmake`, …) +- `cmake/machine-files/` — full per-machine files that combine architecture and + threading choices for common E3SM platforms (e.g., `pm-cpu.cmake`, + `pm-gpu.cmake`, `weaver.cmake`, …) + +A custom `my-mach.cmake` file that targets an Intel Skylake CPU with OpenMP +threading could look like: + +```cmake set (EKAT_MACH_FILES_PATH /path/to/ekat-src/cmake/machine-files/kokkos) include (${EKAT_MACH_FILES_PATH}/openmp.cmake) include (${EKAT_MACH_FILES_PATH}/intel-skx.cmake) - ``` -And this `do-cmake.sh` script would configure EKAT for a debug build using the machine file above -``` +## Example cmake configuration script + +The following `do-cmake.sh` script shows how to configure EKAT as a standalone +build, enabling some optional sub-packages and the test suite: + +```bash #!/bin/bash -SRC_DIR=${WORK_DIR}/libs/ekat/ekat-src/branch -INSTALL_DIR=${WORK_DIR}/libs/ekat/ekat-install/branch +SRC_DIR=/path/to/ekat-src +BUILD_DIR=/path/to/ekat-build +INSTALL_DIR=/path/to/ekat-install -rm -rf CMakeFiles -rm -f CMakeCache.txt +rm -rf ${BUILD_DIR} && mkdir -p ${BUILD_DIR} +cd ${BUILD_DIR} cmake \ - -C /path/to/my-mach.cmake \ + -C /path/to/my-mach.cmake \ \ - -D CMAKE_BUILD_TYPE:STRING=DEBUG \ - -D CMAKE_CXX_COMPILER:STRING=g++ \ - -D CMAKE_Fortran_COMPILER:STRING=gfortran \ - -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \ + -D CMAKE_BUILD_TYPE:STRING=Release \ + -D CMAKE_CXX_COMPILER:STRING=g++ \ + -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \ \ - -D EKAT_DISABLE_TPL_WARNINGS:BOOL=ON \ + -D EKAT_ENABLE_ALL_PACKAGES:BOOL=ON \ \ - -D EKAT_ENABLE_TESTS:BOOL=ON \ - -D EKAT_TEST_DOUBLE_PRECISION:BOOL=ON \ - -D EKAT_TEST_SINGLE_PRECISION:BOOL=ON \ - -D EKAT_TEST_MAX_THREADS:STRING=8 \ + -D EKAT_DISABLE_TPL_WARNINGS:BOOL=ON \ + \ + -D EKAT_ENABLE_TESTS:BOOL=ON \ + -D EKAT_TEST_MAX_THREADS:STRING=4 \ \ ${SRC_DIR} ``` -Here are some of EKAT's cmake config options: +Once CMake configuration has completed, build and test EKAT with: + +```bash +make -j$(nproc) +ctest --output-on-failure +``` + +## CMake configuration options + +| Option | Default | Description | +|--------|---------|-------------| +| `EKAT_ENABLE_ALL_PACKAGES` | `OFF` | Enable all optional sub-packages at once. | +| `EKAT_ENABLE_ALGORITHM` | `OFF` | Enable the Algorithm sub-package. | +| `EKAT_ENABLE_EXPRESSION` | `OFF` | Enable the Expression sub-package. | +| `EKAT_ENABLE_KOKKOS` | `OFF` | Enable the KokkosUtils sub-package. | +| `EKAT_ENABLE_LOGGING` | `OFF` | Enable the Logging sub-package. | +| `EKAT_ENABLE_PACK` | `OFF` | Enable the Pack sub-package. | +| `EKAT_ENABLE_YAML_PARSER` | `OFF` | Enable the Parser sub-package. | +| `EKAT_ENABLE_MPI` | `ON` | Enable MPI support in Core. | +| `EKAT_ENABLE_TESTS` | `ON` | Build the test suite. | +| `EKAT_TEST_MAX_THREADS` | `1` | Maximum number of OpenMP threads used in tests. | +| `EKAT_TEST_LAUNCHER_BUFFER` | `OFF` | Buffer test output and print it all at once (avoids interleaving with concurrent tests). | +| `EKAT_DISABLE_TPL_WARNINGS` | `OFF` | Suppress compiler warnings from third-party libraries. | +| `EKAT_ENABLE_VALGRIND` | `OFF` | Run tests through Valgrind. | +| `EKAT_ENABLE_COMPUTE_SANITIZER` | `OFF` | Run tests through NVIDIA compute-sanitizer. | +| `EKAT_ENABLE_COVERAGE` | `OFF` | Enable code-coverage instrumentation (Debug builds only). | + +# CMake utility scripts + +EKAT installs several CMake utility scripts that downstream projects can use. +After installing EKAT, add `/share/cmake/Modules` to +`CMAKE_MODULE_PATH` to access them. + +## `EkatCreateUnitTest.cmake` + +Provides three functions for creating Catch2-based unit tests: + +- **`EkatCreateUnitTestExec(name srcs ...)`** — Creates a test executable from + source files. Accepts options to set include directories, compiler + definitions/flags, linked libraries, and linker flags. +- **`EkatCreateUnitTestFromExec(test_name exec_name ...)`** — Takes an existing + executable and registers one CTest test per requested MPI-rank / OpenMP-thread + combination (e.g., `MPI_RANKS 1 4` × `THREADS 1 2` produces four tests). +- **`EkatCreateUnitTest(name srcs ...)`** — Convenience wrapper that calls both + functions above with a combined argument list. + +## `EkatSetCompilerFlags.cmake` + +Provides macros to initialize and configure compiler flags in a portable way: + +- **`ResetFlags()`** — Clears CMake's default language flags for the current + CMake directory scope, giving you a clean slate. +- **`SetFlags(...)`** / **`SetCommonFlags()`** — Apply recommended flags for + debug and release builds. +- **`SetProfilingFlags(PROFILER ... COVERAGE ...)`** — Add profiling or coverage + instrumentation flags. + +## `EkatUtils.cmake` + +General-purpose CMake helpers: + +- **`IsDebugBuild(OUT)`** — Sets `OUT` to `TRUE` when `CMAKE_BUILD_TYPE` is + `Debug`. +- **`CheckMacroArgs(...)`** — Validates parsed arguments for a macro and emits + author warnings for unrecognized or missing-value keywords. +- **`EkatDisableAllWarning(target)`** — Suppresses all compiler warnings for a + given target (useful for third-party code). +- **`dump_cmake_variables([REGEX ...])`** — Prints all visible CMake variables + to the console, with optional regex filtering. + +## `EkatMpiUtils.cmake` + +MPI-related helpers: + +- **`GetMpiDistributionName(OUT)`** — Detects the MPI distribution (OpenMPI, + MPICH, Cray, …) by inspecting the MPI header. +- **`DisableMpiCxxBindings()`** — Removes deprecated MPI C++ bindings from + the link flags to avoid link-time issues. +- **`SetMpiRuntimeEnvVars()`** — Sets runtime environment variables needed by + the detected MPI distribution. + +## `tpls/EkatBuildEkat.cmake` + +Lets downstream projects embed EKAT as a subdirectory without worrying about +double-inclusion: + +```cmake +include(EkatBuildEkat) +BuildEkat(PREFIX MY_PROJECT) +``` + +This macro checks a global property to ensure `add_subdirectory` is only called +once, even if multiple sub-components of a project all try to pull in EKAT. + +## `tpls/EkatBuildKokkos.cmake` / `tpls/EkatFindKokkos.cmake` + +Locate or fetch-and-build Kokkos. `EkatFindKokkos` tries `find_package(Kokkos)` +first; if that fails, `EkatBuildKokkos` fetches Kokkos from the bundled +submodule via `FetchContent`. -- EKAT_ENABLE_TESTS: enables ekat testing -- EKAT_TEST_[DOUBLE|SINGLE]_PRECISION: whether to test single and/or double precision. -- EKAT_TEST_MAX_[THREADS|RANKS]: maximum number of threads/ranks to use in unit tests -- EKAT_TEST_THREADS_INC: increment in number of threads for each test. -- EKAT_DEFAULT_BFB: in certain kernels, whether to default to a BFB, serialized, implementation. -- EKAT_DISABLE_TPL_WARNINGS: whether warnings from TPLs should be disabled. -- EKAT_ENABLE_VALGRIND: whether tests should be run through valgrind. -- EKAT_ENABLE_CUDA_MEMCHECK: whether tests should be run through cuda-memcheck. -- EKAT_ENABLE_COVERAGE: whether to enable code coverage in the compiler. +## `tpls/EkatBuildYamlCpp.cmake` / `tpls/EkatBuildCatch2.cmake` / `tpls/EkatBuildSpdlog.cmake` -Once CMake configuration has completed, you can build and test EKAT with the usual `make` and `ctest` commands. +Fetch and build the yaml-cpp, Catch2, and spdlog third-party libraries +respectively via `FetchContent`, using E3SM-Project forks pinned to specific +commits for reproducibility.