From c2eb1c7aa38b91699f55b09ad277bad405f382f8 Mon Sep 17 00:00:00 2001 From: Andrew Ho Date: Mon, 15 Dec 2025 12:07:28 -0800 Subject: [PATCH 1/5] added base Laghos benchmarking docs --- docs/12_laghos/laghos.rst | 206 +++++++++++++++++++++++++++++++++++++- 1 file changed, 203 insertions(+), 3 deletions(-) diff --git a/docs/12_laghos/laghos.rst b/docs/12_laghos/laghos.rst index 22313ab..e450ac4 100644 --- a/docs/12_laghos/laghos.rst +++ b/docs/12_laghos/laghos.rst @@ -2,11 +2,12 @@ Laghos ****** -https://github.com/CEED/Laghos - Purpose ======= +**Laghos** (LAGrangian High-Order Solver) is a miniapp that solves the time-dependent Euler equations of compressible gas dynamics in a moving Lagrangian frame using unstructured high-order finite element spatial discretization and explicit high-order time-stepping. +It is available at https://github.com/CEED/Laghos . +It requires an installation of Hypre, Metis, and MFEM. Characteristics =============== @@ -14,42 +15,241 @@ Characteristics Problems -------- +The test problems to be run are the triple point problem (problem 3) in 2D and 3D. +These are to be run with a conforming mesh. + +The problem sizes and partitioning scheme for both problems can be set by the user from the command line. + Figure of Merit --------------- +Each time step in Laghos contains 3 major distinct computations: + +1. The inversion of the global kinematic mass matrix (CG H1). +2. The force operator evaluation from degrees of freedom to quadrature points (Forces). +3. The physics kernel in quadrature points (UpdateQuadData). + +Laghos is instrumented to report the total execution times and rates, in terms of millions of degrees of freedom per second (megadofs), for each of these computational phases. (The time for inversion of the local thermodynamic mass matrices (CG L2) is also reported, but that takes a small part of the overall computation.) +Rates are averaged over all RK stages taken and for the purposes of benchmarking are configured to take 100 RK4 timesteps. + +Laghos also reports the total rate for these major kernels, which is the **Figure of Merit (FOM)** for benchmarking purposes. Source code modifications ========================= -Please see :ref:`GlobalRunRules` for general guidance on allowed modifications. +Please see :ref:`GlobalRunRules` for general guidance on allowed modifications. + +For Laghos we define the following restrictions on source code modifications: + +- Laghos must use MFEM and Hypre as the solver library, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre respectively. Hypre must be built with `HYPRE_ENABLE_MIXEDINT=ON`. +- The listed command line options shown in :ref:`RunningLaghos` must be used without modification. A few additional command line options may be added: + - `-d gpu` or `-d raja-gpu` for GPU acceleration (note: the latter requires MFEM to be built with RAJA). + - `-dev` for specifying which GPU to run on for a multi-GPU system + - `-gam` for GPU-aware MPI +- Hypre/MFEM/Laghos may optionally be built with Umpire (https://github.com/LLNL/Umpire). The host and device memory allocators may be changed to any available allocator in MFEM. Building ======== +Prerequisites: +- CMake 3.24.0+ +- C compiler +- C++17 compiler +- MPI + +These instructions install all dependencies to a user-defined `$INSTALLDIR` using a user-defined `$CC` C compiler, `$CXX` C++-17 compiler, `$CUDACC` CUDA compiler (for CUDA acceleration), and `$HIPCC` HIP compiler (for HIP acceleration). Both `nvcc` and `clang` are supported as the CUDA compiler. + +Metis (required) +---------------- + +TODO: only if not doing cartesian partitioning, need to decide on problem size configurations. + +``` +git clone https://github.com/KarypisLab/METIS.git +cd METIS +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=$CC -DCMAKE_INSTALL_PREFIX=$INSTALLDIR +make -j install +``` + +Umpire (optional) +----------------- + +It is only recommended to use Umpire for GPU-accelerated configurations. + +CUDA: +``` +git clone https://github.com/LLNL/Umpire.git +cd Umpire +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=native -DENABLE_CUDA=ON -DUMPIRE_ENABLE_C=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC +make -j install +``` + +HIP: +``` +git clone https://github.com/LLNL/Umpire.git +cd Umpire +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_HIP_ARCHITECTURES=native -DENABLE_HIP=ON -DUMPIRE_ENABLE_C=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC +make -j install +``` + +Hypre (required) +---------------- + +CPU-only: +``` +git clone https://github.com/hypre-space/hypre.git +cd hypre/build +cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX +make -j install +``` + +CUDA: +``` +git clone https://github.com/hypre-space/hypre.git +cd hypre/build +cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON +make -j install +``` +`HYPRE_ENABLE_GPU_AWARE_MPI` and `HYPRE_ENABLE_UMPIRE` may be optionally turned off. + +HIP: +``` +git clone https://github.com/hypre-space/hypre.git +cd hypre/build +cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON +make -j install +``` +`HYPRE_ENABLE_GPU_AWARE_MPI` and `HYPRE_ENABLE_UMPIRE` may be optionally turned off. + +MFEM (required) +--------------- + +CPU-only: +``` +git clone https://github.com/mfem/mfem.git +cd mfem +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DCMAKE_CXX_COMPILER=$CXX +make -j install +``` + +CUDA: +``` +git clone https://github.com/mfem/mfem.git +cd mfem +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DMFEM_USE_CUDA=ON -DMFEM_USE_UMPIRE=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DUMPIRE_DIR=$INSTALLDIR +make -j install +``` +`MFEM_USE_UMPIRE` may be optionally turned off. + +HIP: +``` +git clone https://github.com/mfem/mfem.git +cd mfem +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DMFEM_USE_HIP=ON -DMFEM_USE_UMPIRE=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DUMPIRE_DIR=$INSTALLDIR +make -j install +``` +`MFEM_USE_UMPIRE` may be optionally turned off. + +Laghos (required) +----------------- + +CPU-only: +``` +https://github.com/CEED/Laghos.git +cd Laghos +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX +make -j +``` + +CUDA: +``` +https://github.com/CEED/Laghos.git +cd Laghos +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DCMAKE_CUDA_ARCHITECTURES=native +make -j +``` + +HIP: +``` +https://github.com/CEED/Laghos.git +cd Laghos +mkdir build +cd build +cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DCMAKE_HIP_ARCHITECTURES=native +make -j +``` + +.. _RunningLaghos: Running ======= +2D: +``` +# 2D Q1Q0 +laghos -dim 2 -p 3 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 +# 2D Q2Q1 +laghos -dim 2 -p 3 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 +# 2D Q3Q2 +laghos -dim 2 -p 3 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 +``` +3D: +``` +# 3D Q1Q0 +laghos -dim 3 -p 1 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 +# 3D Q2Q1 +laghos -dim 3 -p 1 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 +# 3D Q3Q2 +laghos -dim 3 -p 1 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 +``` + +TODO: problem sizes and partitioning options Validation ========== +TODO Example Scalability Results =========================== +TODO Memory Usage ============ +TODO Strong Scaling on El Capitan ============================ +TODO Weak Scaling on El Capitan ========================== +TODO References ========== + +V. Dobrev, Tz. Kolev and R. Rieben +High-order curvilinear finite element methods for Lagrangian hydrodynamics +SIAM Journal on Scientific Computing, (34) 2012, pp. B606–B641. +https://doi.org/10.1137/120864672 From 40327617ba1c69c44b27b6bff78fc53197b82314 Mon Sep 17 00:00:00 2001 From: Andrew Ho Date: Tue, 16 Dec 2025 16:29:06 -0800 Subject: [PATCH 2/5] formatting --- docs/12_laghos/laghos.rst | 253 +++++++++++++++++++------------------- 1 file changed, 128 insertions(+), 125 deletions(-) diff --git a/docs/12_laghos/laghos.rst b/docs/12_laghos/laghos.rst index 6cb7964..8185495 100644 --- a/docs/12_laghos/laghos.rst +++ b/docs/12_laghos/laghos.rst @@ -43,37 +43,38 @@ Please see :ref:`GlobalRunRules` for general guidance on allowed modifications. For Laghos we define the following restrictions on source code modifications: -- Laghos must use MFEM and Hypre as the solver library, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre respectively. Hypre must be built with `HYPRE_ENABLE_MIXEDINT=ON`. -- The listed command line options shown in :ref:`RunningLaghos` must be used without modification. A few additional command line options may be added: - - `-d gpu` or `-d raja-gpu` for GPU acceleration (note: the latter requires MFEM to be built with RAJA). - - `-dev` for specifying which GPU to run on for a multi-GPU system - - `-gam` for GPU-aware MPI -- Hypre/MFEM/Laghos may optionally be built with Umpire (https://github.com/LLNL/Umpire). The host and device memory allocators may be changed to any available allocator in MFEM. +* Laghos must use MFEM and Hypre as the solver library, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre respectively. Hypre must be built with ``HYPRE_ENABLE_MIXEDINT=ON``. +* The listed command line options shown in :ref:`RunningLaghos` must be used without modification. A few additional command line options may be added: + + * ``-d gpu`` or ``-d raja-gpu`` for GPU acceleration (note: the latter requires MFEM to be built with RAJA). + * ``-dev`` for specifying which GPU to run on for a multi-GPU system + * ``-gam`` for GPU-aware MPI + +* Hypre/MFEM/Laghos may optionally be built with Umpire (https://github.com/LLNL/Umpire). The host and device memory allocators may be changed to any available allocator in MFEM. Building ======== Prerequisites: -- CMake 3.24.0+ -- C compiler -- C++17 compiler -- MPI +* CMake 3.24.0+ +* C compiler +* C++17 compiler +* MPI -These instructions install all dependencies to a user-defined `$INSTALLDIR` using a user-defined `$CC` C compiler, `$CXX` C++-17 compiler, `$CUDACC` CUDA compiler (for CUDA acceleration), and `$HIPCC` HIP compiler (for HIP acceleration). Both `nvcc` and `clang` are supported as the CUDA compiler. +These instructions install all dependencies to a user-defined ``$INSTALLDIR`` using a user-defined ``$CC`` C compiler, ``$CXX`` C++-17 compiler, ``$CUDACC`` CUDA compiler (for CUDA acceleration), and ``$HIPCC`` HIP compiler (for HIP acceleration). Both ``nvcc`` and ``clang`` are supported as the CUDA compiler. Metis (required) ---------------- TODO: only if not doing cartesian partitioning, need to decide on problem size configurations. -``` -git clone https://github.com/KarypisLab/METIS.git -cd METIS -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=$CC -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -make -j install -``` +.. code-block:: sh + git clone https://github.com/KarypisLab/METIS.git + cd METIS + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=$CC -DCMAKE_INSTALL_PREFIX=$INSTALLDIR + make -j install Umpire (optional) ----------------- @@ -81,121 +82,125 @@ Umpire (optional) It is only recommended to use Umpire for GPU-accelerated configurations. CUDA: -``` -git clone https://github.com/LLNL/Umpire.git -cd Umpire -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=native -DENABLE_CUDA=ON -DUMPIRE_ENABLE_C=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -make -j install -``` + +.. code-block:: sh + git clone https://github.com/LLNL/Umpire.git + cd Umpire + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=native -DENABLE_CUDA=ON -DUMPIRE_ENABLE_C=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC + make -j install HIP: -``` -git clone https://github.com/LLNL/Umpire.git -cd Umpire -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_HIP_ARCHITECTURES=native -DENABLE_HIP=ON -DUMPIRE_ENABLE_C=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -make -j install -``` + +.. code-block:: sh + git clone https://github.com/LLNL/Umpire.git + cd Umpire + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_HIP_ARCHITECTURES=native -DENABLE_HIP=ON -DUMPIRE_ENABLE_C=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC + make -j install Hypre (required) ---------------- CPU-only: -``` -git clone https://github.com/hypre-space/hypre.git -cd hypre/build -cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -make -j install -``` + +.. code-block:: sh + git clone https://github.com/hypre-space/hypre.git + cd hypre/build + cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX + make -j install CUDA: -``` -git clone https://github.com/hypre-space/hypre.git -cd hypre/build -cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON -make -j install -``` -`HYPRE_ENABLE_GPU_AWARE_MPI` and `HYPRE_ENABLE_UMPIRE` may be optionally turned off. + +.. code-block:: sh + git clone https://github.com/hypre-space/hypre.git + cd hypre/build + cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON + make -j install + +``HYPRE_ENABLE_GPU_AWARE_MPI`` and ``HYPRE_ENABLE_UMPIRE`` may be optionally turned off. HIP: -``` -git clone https://github.com/hypre-space/hypre.git -cd hypre/build -cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON -make -j install -``` -`HYPRE_ENABLE_GPU_AWARE_MPI` and `HYPRE_ENABLE_UMPIRE` may be optionally turned off. + +.. code-block:: sh + git clone https://github.com/hypre-space/hypre.git + cd hypre/build + cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON + make -j install + +``HYPRE_ENABLE_GPU_AWARE_MPI`` and ``HYPRE_ENABLE_UMPIRE`` may be optionally turned off. MFEM (required) --------------- CPU-only: -``` -git clone https://github.com/mfem/mfem.git -cd mfem -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DCMAKE_CXX_COMPILER=$CXX -make -j install -``` + +.. code-block:: sh + git clone https://github.com/mfem/mfem.git + cd mfem + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DCMAKE_CXX_COMPILER=$CXX + make -j install CUDA: -``` -git clone https://github.com/mfem/mfem.git -cd mfem -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DMFEM_USE_CUDA=ON -DMFEM_USE_UMPIRE=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DUMPIRE_DIR=$INSTALLDIR -make -j install -``` -`MFEM_USE_UMPIRE` may be optionally turned off. + +.. code-block:: sh + git clone https://github.com/mfem/mfem.git + cd mfem + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DMFEM_USE_CUDA=ON -DMFEM_USE_UMPIRE=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DUMPIRE_DIR=$INSTALLDIR + make -j install + +``MFEM_USE_UMPIRE`` may be optionally turned off. HIP: -``` -git clone https://github.com/mfem/mfem.git -cd mfem -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DMFEM_USE_HIP=ON -DMFEM_USE_UMPIRE=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DUMPIRE_DIR=$INSTALLDIR -make -j install -``` -`MFEM_USE_UMPIRE` may be optionally turned off. + +.. code-block:: sh + git clone https://github.com/mfem/mfem.git + cd mfem + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_DIR=$INSTALLDIR -DMETIS_DIR=$INSTALLDIR -DMFEM_USE_MPI=ON -DMFEM_USE_METIS=ON -DMFEM_USE_HIP=ON -DMFEM_USE_UMPIRE=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DUMPIRE_DIR=$INSTALLDIR + make -j install + +``MFEM_USE_UMPIRE`` may be optionally turned off. Laghos (required) ----------------- CPU-only: -``` -https://github.com/CEED/Laghos.git -cd Laghos -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -make -j -``` + +.. code-block:: sh + git clone https://github.com/CEED/Laghos.git + cd Laghos + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX + make -j CUDA: -``` -https://github.com/CEED/Laghos.git -cd Laghos -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DCMAKE_CUDA_ARCHITECTURES=native -make -j -``` + +.. code-block:: sh + git clone https://github.com/CEED/Laghos.git + cd Laghos + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DCMAKE_CUDA_ARCHITECTURES=native + make -j HIP: -``` -https://github.com/CEED/Laghos.git -cd Laghos -mkdir build -cd build -cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DCMAKE_HIP_ARCHITECTURES=native -make -j -``` + +.. code-block:: sh + git clone https://github.com/CEED/Laghos.git + cd Laghos + mkdir build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DCMAKE_HIP_ARCHITECTURES=native + make -j .. _RunningLaghos: @@ -203,23 +208,24 @@ Running ======= 2D: -``` -# 2D Q1Q0 -laghos -dim 2 -p 3 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 -# 2D Q2Q1 -laghos -dim 2 -p 3 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 -# 2D Q3Q2 -laghos -dim 2 -p 3 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 -``` + +.. code-block:: sh + # 2D Q1Q0 + laghos -dim 2 -p 3 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 + # 2D Q2Q1 + laghos -dim 2 -p 3 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 + # 2D Q3Q2 + laghos -dim 2 -p 3 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 + 3D: -``` -# 3D Q1Q0 -laghos -dim 3 -p 1 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 -# 3D Q2Q1 -laghos -dim 3 -p 1 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 -# 3D Q3Q2 -laghos -dim 3 -p 1 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 -``` + +.. code-block:: sh + # 3D Q1Q0 + laghos -dim 3 -p 1 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 + # 3D Q2Q1 + laghos -dim 3 -p 1 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 + # 3D Q3Q2 + laghos -dim 3 -p 1 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 TODO: problem sizes and partitioning options @@ -253,7 +259,4 @@ TODO References ========== -V. Dobrev, Tz. Kolev and R. Rieben -High-order curvilinear finite element methods for Lagrangian hydrodynamics -SIAM Journal on Scientific Computing, (34) 2012, pp. B606–B641. -https://doi.org/10.1137/120864672 +.. [Laghos] V. Dobrev, Tz. Kolev and R. Rieben 'High-order curvilinear finite element methods for Lagrangian hydrodynamics', SIAM Journal on Scientific Computing, (34) 2012, pp. B606–B641. https://doi.org/10.1137/120864672 From 33d61678ae3d68c506666debab3f95519cf51267 Mon Sep 17 00:00:00 2001 From: Andrew Ho Date: Tue, 16 Dec 2025 16:35:50 -0800 Subject: [PATCH 3/5] codeblocks --- docs/12_laghos/laghos.rst | 42 ++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/docs/12_laghos/laghos.rst b/docs/12_laghos/laghos.rst index 8185495..0a3ec08 100644 --- a/docs/12_laghos/laghos.rst +++ b/docs/12_laghos/laghos.rst @@ -68,7 +68,8 @@ Metis (required) TODO: only if not doing cartesian partitioning, need to decide on problem size configurations. -.. code-block:: sh +.. code-block:: console + git clone https://github.com/KarypisLab/METIS.git cd METIS mkdir build @@ -83,7 +84,8 @@ It is only recommended to use Umpire for GPU-accelerated configurations. CUDA: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/LLNL/Umpire.git cd Umpire mkdir build @@ -93,7 +95,8 @@ CUDA: HIP: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/LLNL/Umpire.git cd Umpire mkdir build @@ -106,7 +109,8 @@ Hypre (required) CPU-only: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/hypre-space/hypre.git cd hypre/build cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX @@ -114,7 +118,8 @@ CPU-only: CUDA: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/hypre-space/hypre.git cd hypre/build cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_CUDA_COMPILER=$CUDACC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON @@ -124,7 +129,8 @@ CUDA: HIP: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/hypre-space/hypre.git cd hypre/build cmake ../src -DCMAKE_BUILD_TYPE=Release -DHYPRE_ENABLE_MIXEDINT=ON -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DHYPRE_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=native -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_HIP_COMPILER=$HIPCC -DHYPRE_ENABLE_GPU_AWARE_MPI=ON -DHYPRE_ENABLE_UMPIRE=ON @@ -137,7 +143,8 @@ MFEM (required) CPU-only: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/mfem/mfem.git cd mfem mkdir build @@ -147,7 +154,8 @@ CPU-only: CUDA: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/mfem/mfem.git cd mfem mkdir build @@ -159,7 +167,8 @@ CUDA: HIP: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/mfem/mfem.git cd mfem mkdir build @@ -174,7 +183,8 @@ Laghos (required) CPU-only: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/CEED/Laghos.git cd Laghos mkdir build @@ -184,7 +194,8 @@ CPU-only: CUDA: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/CEED/Laghos.git cd Laghos mkdir build @@ -194,7 +205,8 @@ CUDA: HIP: -.. code-block:: sh +.. code-block:: console + git clone https://github.com/CEED/Laghos.git cd Laghos mkdir build @@ -209,7 +221,8 @@ Running 2D: -.. code-block:: sh +.. code-block:: console + # 2D Q1Q0 laghos -dim 2 -p 3 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 # 2D Q2Q1 @@ -219,7 +232,8 @@ Running 3D: -.. code-block:: sh +.. code-block:: console + # 3D Q1Q0 laghos -dim 3 -p 1 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 # 3D Q2Q1 From 112177389a6fa2215391c62d2508611ad32387c8 Mon Sep 17 00:00:00 2001 From: Andrew Ho Date: Thu, 18 Dec 2025 11:22:18 -0800 Subject: [PATCH 4/5] updates --- docs/12_laghos/laghos.rst | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/12_laghos/laghos.rst b/docs/12_laghos/laghos.rst index 0a3ec08..973b56d 100644 --- a/docs/12_laghos/laghos.rst +++ b/docs/12_laghos/laghos.rst @@ -43,7 +43,7 @@ Please see :ref:`GlobalRunRules` for general guidance on allowed modifications. For Laghos we define the following restrictions on source code modifications: -* Laghos must use MFEM and Hypre as the solver library, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre respectively. Hypre must be built with ``HYPRE_ENABLE_MIXEDINT=ON``. +* Laghos must use MFEM and Hypre as the solver library, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre respectively. Hypre must be built with ``HYPRE_ENABLE_MIXEDINT=ON``. The final validated results must match or exceed the results of double precision accuracy shown in :ref:`ValidateLaghos`. * The listed command line options shown in :ref:`RunningLaghos` must be used without modification. A few additional command line options may be added: * ``-d gpu`` or ``-d raja-gpu`` for GPU acceleration (note: the latter requires MFEM to be built with RAJA). @@ -56,6 +56,7 @@ Building ======== Prerequisites: + * CMake 3.24.0+ * C compiler * C++17 compiler @@ -224,25 +225,27 @@ Running .. code-block:: console # 2D Q1Q0 - laghos -dim 2 -p 3 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 + laghos -dim 2 -p 3 -ok 1 -ot 0 -oq -1 -pa -no-nc -ms 250 -tf 100000 # 2D Q2Q1 - laghos -dim 2 -p 3 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 + laghos -dim 2 -p 3 -ok 2 -ot 1 -oq -1 -pa -no-nc -ms 250 -tf 100000 # 2D Q3Q2 - laghos -dim 2 -p 3 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 + laghos -dim 2 -p 3 -ok 3 -ot 2 -oq -1 -pa -no-nc -ms 250 -tf 100000 3D: .. code-block:: console # 3D Q1Q0 - laghos -dim 3 -p 1 -ok 1 -ot 0 -pa -no-nc -ms 100 -tf 100000 + laghos -dim 3 -p 1 -ok 1 -ot 0 -oq -1 -pa -no-nc -ms 250 -tf 100000 # 3D Q2Q1 - laghos -dim 3 -p 1 -ok 2 -ot 1 -pa -no-nc -ms 100 -tf 100000 + laghos -dim 3 -p 1 -ok 2 -ot 1 -oq -1 -pa -no-nc -ms 250 -tf 100000 # 3D Q3Q2 - laghos -dim 3 -p 1 -ok 3 -ot 2 -pa -no-nc -ms 100 -tf 100000 + laghos -dim 3 -p 1 -ok 3 -ot 2 -oq -1 -pa -no-nc -ms 250 -tf 100000 TODO: problem sizes and partitioning options +.. _ValidateLaghos: + Validation ========== From d7d2f46540cf7ac00e7a7bad3341a32f33484b7e Mon Sep 17 00:00:00 2001 From: Andrew Ho Date: Wed, 7 Jan 2026 16:22:17 -0800 Subject: [PATCH 5/5] Update to focus on just 3D sedov --- docs/12_laghos/laghos.rst | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/docs/12_laghos/laghos.rst b/docs/12_laghos/laghos.rst index 973b56d..64a56d6 100644 --- a/docs/12_laghos/laghos.rst +++ b/docs/12_laghos/laghos.rst @@ -17,7 +17,7 @@ Characteristics Problems -------- -The test problems to be run are the triple point problem (problem 3) in 2D and 3D. +The test problems to be run are the Sedov shock (problem 1) in 3D. These are to be run with a conforming mesh. The problem sizes and partitioning scheme for both problems can be set by the user from the command line. @@ -49,6 +49,7 @@ For Laghos we define the following restrictions on source code modifications: * ``-d gpu`` or ``-d raja-gpu`` for GPU acceleration (note: the latter requires MFEM to be built with RAJA). * ``-dev`` for specifying which GPU to run on for a multi-GPU system * ``-gam`` for GPU-aware MPI + * ``-dev-pool-size`` for specifying an initial Umpire device memory pool size. * Hypre/MFEM/Laghos may optionally be built with Umpire (https://github.com/LLNL/Umpire). The host and device memory allocators may be changed to any available allocator in MFEM. @@ -220,19 +221,6 @@ HIP: Running ======= -2D: - -.. code-block:: console - - # 2D Q1Q0 - laghos -dim 2 -p 3 -ok 1 -ot 0 -oq -1 -pa -no-nc -ms 250 -tf 100000 - # 2D Q2Q1 - laghos -dim 2 -p 3 -ok 2 -ot 1 -oq -1 -pa -no-nc -ms 250 -tf 100000 - # 2D Q3Q2 - laghos -dim 2 -p 3 -ok 3 -ot 2 -oq -1 -pa -no-nc -ms 250 -tf 100000 - -3D: - .. code-block:: console # 3D Q1Q0