From 26542743f4a1c31fd0e0279dadfc9a980f96eba3 Mon Sep 17 00:00:00 2001 From: Vladimir Tomov Date: Mon, 29 Dec 2025 12:22:41 -0800 Subject: [PATCH 1/4] WIP Remhos documentation Updated documentation for Remhos, including purpose, characteristics, source code modifications, building instructions, and running details. --- docs/40_remhos/remhos.rst | 71 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-) diff --git a/docs/40_remhos/remhos.rst b/docs/40_remhos/remhos.rst index c37cc1f..54c5bf5 100644 --- a/docs/40_remhos/remhos.rst +++ b/docs/40_remhos/remhos.rst @@ -2,6 +2,8 @@ Remhos ****** +Documentation for Remhos (REMap High-Order Solver). + https://github.com/CEED/Remhos Remhos source code is not finalized at this point. The problems to run are yet to be defined. @@ -9,6 +11,7 @@ Remhos source code is not finalized at this point. The problems to run are yet t Purpose ======= +Remhos serves as a proxy-app for the advection-based remap methods used in LLNL's MARBL code. Characteristics =============== @@ -16,9 +19,35 @@ Characteristics Problems -------- +The sample runs of interest are listed below. Those represent a bounded high-order remap of a simple scalar finite element field between two computational meshes. + +2D cpu: + +``lrun -n 8 remhos -dim 2 -epm 1024 -o 3 -p 14 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cpu`` + +2D gpu: + +``lrun -n 8 remhos -dim 2 -epm 1024 -o 3 -p 14 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cuda`` + +3D cpu: + +``lrun -n 8 remhos -dim 3 -epm 512 -o 2 -p 10 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cpu`` + +3D gpu: + +``lrun -n 8 remhos -dim 3 -epm 512 -o 2 -p 10 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cuda`` + Figure of Merit --------------- +Remhos reports several FOMs in the terminal, based on the distinct phases of an advection-based remap calculation. All FOMs are reported in ``(megaDOFs x time steps) per second``, reflecting the throughput of the calculation. + +* FOM RHS: construction of the right-hand side of the system. +* FOM INV: inverting the high-order operator, which is used to obtain a high-order unbounded (HO) solution. +* FOM LO: computation of the low-order bounded (LO) approximation of the solution. +* FOM FCT: computation of the FCT solution, combining the LO and HO solutions to obtain a bounded high-order solution. +* **FOM**: performance metric combining all the above phases. + Source code modifications ========================= @@ -26,7 +55,7 @@ Source code modifications Please see :ref:`GlobalRunRules` for general guidance on allowed modifications. For Remhos, we define the following restrictions on source code modifications: -* Remhos uses MFEM and Hypre as libraries, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre . While source code changes to MFEM and Hypre can be proposed, MFEM and Hypre in Laghos may not be replaced with any other libraries. +* Remhos uses MFEM and Hypre as libraries, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre . While source code changes to MFEM and Hypre can be proposed, MFEM and Hypre in Remhos may not be replaced with any other libraries. * Solver parameters should remain unchanged (smoothers, coarsening, etc.). Remhos uses the default MFEM and Hypre parameters appropriate for each platform. @@ -34,6 +63,46 @@ For Remhos, we define the following restrictions on source code modifications: Building ======== +Remhos has the following external dependencies: + +- **hypre**, used for parallel linear algebra, we recommend version 2.10.0b
+ https://computation.llnl.gov/casc/hypre/software.html + +- **MFEM**, used for (high-order) finite element discretization, its GitHub master branch
+ https://github.com/mfem/mfem + +To build the miniapp, first download **hypre** from the links above +and put everything on the same level as the `Remhos` directory:: + + ~> ls + Remhos/ hypre.tar.gz + +Build **hypre**:: + + ~> tar -zxvf hypre.tar.gz + ~> cd hypre/src/ + ~/hypre/src> ./configure --disable-fortran + ~/hypre/src> make -j + ~/hypre/src> cd ../.. + +For large runs (problem size above 2 billion unknowns), add the +``--enable-bigint`` option to the above ``configure`` line. + +Clone and build the parallel version of MFEM:: + + ~> git clone https://github.com/mfem/mfem.git ./mfem + ~> cd mfem/ + ~/mfem> make parallel -j + ~/mfem> cd .. + +The above uses the `master` branch of MFEM. See the [MFEM building page](http://mfem.org/building/) for additional details. + +Build Remhos:: + + ~> cd Remhos/ + ~/Remhos> make + +See ``make help`` for additional options. Running ======= From 65a0f5071514b1248ccdeeb9618621be0bbea4c1 Mon Sep 17 00:00:00 2001 From: Vladimir Tomov Date: Mon, 29 Dec 2025 17:21:46 -0800 Subject: [PATCH 2/4] Revise Remhos documentation building and running Updated Build instructions and Running. --- docs/40_remhos/remhos.rst | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/40_remhos/remhos.rst b/docs/40_remhos/remhos.rst index 54c5bf5..b686c8d 100644 --- a/docs/40_remhos/remhos.rst +++ b/docs/40_remhos/remhos.rst @@ -55,9 +55,9 @@ Source code modifications Please see :ref:`GlobalRunRules` for general guidance on allowed modifications. For Remhos, we define the following restrictions on source code modifications: -* Remhos uses MFEM and Hypre as libraries, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre . While source code changes to MFEM and Hypre can be proposed, MFEM and Hypre in Remhos may not be replaced with any other libraries. +* Remhos uses MFEM and Hypre as libraries, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre . While source code changes to MFEM and Hypre can be proposed, MFEM and Hypre in Remhos may not be replaced with any other libraries. -* Solver parameters should remain unchanged (smoothers, coarsening, etc.). Remhos uses the default MFEM and Hypre parameters appropriate for each platform. +* Solver parameters should remain unchanged (smoothers, coarsening, etc.). Remhos uses the default MFEM and Hypre parameters appropriate for each platform. Building @@ -65,19 +65,22 @@ Building Remhos has the following external dependencies: -- **hypre**, used for parallel linear algebra, we recommend version 2.10.0b
- https://computation.llnl.gov/casc/hypre/software.html +- **hypre**, used for parallel linear algebra, we recommend version 2.24.0 + + https://github.com/hypre-space/hypre + +- **MFEM**, used for (high-order) finite element discretization, its GitHub master branch -- **MFEM**, used for (high-order) finite element discretization, its GitHub master branch
https://github.com/mfem/mfem To build the miniapp, first download **hypre** from the links above and put everything on the same level as the `Remhos` directory:: + ~> mkdir remhos ~> ls - Remhos/ hypre.tar.gz + remhos/ hypre.tar.gz -Build **hypre**:: +Build **hypre** (note that the folder must be named ``hypre``):: ~> tar -zxvf hypre.tar.gz ~> cd hypre/src/ @@ -92,21 +95,24 @@ Clone and build the parallel version of MFEM:: ~> git clone https://github.com/mfem/mfem.git ./mfem ~> cd mfem/ - ~/mfem> make parallel -j + ~/mfem> make parallel -j MFEM_USE_METIS=NO ~/mfem> cd .. The above uses the `master` branch of MFEM. See the [MFEM building page](http://mfem.org/building/) for additional details. Build Remhos:: - ~> cd Remhos/ - ~/Remhos> make + ~> git clone https://github.com/CEED/Remhos.git ./remhos + ~> cd remhos/ + ~/remhos> make See ``make help`` for additional options. Running ======= +The main performance-related options are the device ``(-d)``, number of tasks ``(-n)``, the elements per task ``(-epm)``, and the finite element order ``(-o)``. Appropriate mesh and partitioning are generated automatically. The product of ``(-n)`` and ``(-epm)`` determines the mesh size. The order ``(-o)`` can also be increased, resulting in more work per mesh element. For example, for weak scaling, vary ``(-n)`` and fix ``(-epm)``; for strong scaling, make sure the product of ``(-n)*(-epm)`` is constant. + Validation ========== From 04f165a1afc909376439adc431615b1f9936a575 Mon Sep 17 00:00:00 2001 From: Vladimir Tomov Date: Thu, 8 Jan 2026 09:01:57 -0800 Subject: [PATCH 3/4] Update memory usage and scaling sections in remhos.rst Added details about memory usage and scaling in Remhos documentation. --- docs/40_remhos/remhos.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/40_remhos/remhos.rst b/docs/40_remhos/remhos.rst index b686c8d..3703725 100644 --- a/docs/40_remhos/remhos.rst +++ b/docs/40_remhos/remhos.rst @@ -121,10 +121,17 @@ Validation Example Scalability Results =========================== +Not available. Memory Usage ============ +For each task, the memory usage is determined by the number of elements per task ``(-n)`` and the finite-element order ``(-o)``. +The dominant contributors are the mesh data structures and the finite-element operator storage. + +Let ``d`` denote the spatial dimension and ``r`` the number of uniform refinements in one dimension (``r ≈ (n)^(1/d)``). +The number of elements per task then scales as ``O(r^d)``. The storage required per element (mesh plus operators) depends on the polynomial order ``o``. +Because Remhos employs partial assembly, the per-element operator storage is optimal, scaling as ``O(o^d)``. Consequently, the total memory consumption per task scales as ``O(r^d o^d) = O((r o)^d)``. Strong Scaling on El Capitan ============================ @@ -135,6 +142,7 @@ Please see :ref:`ElCapitanSystemDescription` for El Capitan system description. Weak Scaling on El Capitan ========================== +Not available. References ========== From bca4afda00873c9522f0390c3ba003d5fd85f992 Mon Sep 17 00:00:00 2001 From: Vladimir Tomov Date: Thu, 8 Jan 2026 09:39:37 -0800 Subject: [PATCH 4/4] Update remhos.rst - references and validation References and validation --- docs/40_remhos/remhos.rst | 40 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/docs/40_remhos/remhos.rst b/docs/40_remhos/remhos.rst index 3703725..0f974a3 100644 --- a/docs/40_remhos/remhos.rst +++ b/docs/40_remhos/remhos.rst @@ -117,11 +117,19 @@ The main performance-related options are the device ``(-d)``, number of tasks `` Validation ========== +Code correctness is validated by running the two tests above and comparing the final solution mass. The following quantities must agree between the CPU and GPU runs: + +.. code-block:: console + + lrun -n 8 remhos -dim 2 -epm 1024 -o 3 -p 14 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa + Final mass u: 0.0930949258 + lrun -n 8 remhos -dim 3 -epm 512 -o 2 -p 10 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa + Final mass u: 0.1160152403 Example Scalability Results =========================== -Not available. +TODO. Memory Usage ============ @@ -138,11 +146,39 @@ Strong Scaling on El Capitan Please see :ref:`ElCapitanSystemDescription` for El Capitan system description. +TODO. Weak Scaling on El Capitan ========================== -Not available. +TODO. References ========== + +Remhos combines discretization methods described in the following articles: + +R. Anderson, V. Dobrev, Tz. Kolev and R. Rieben, +Monotonicity in high-order curvilinear finite element arbitrary Lagrangian-Eulerian remap +(https://doi.org/10.1002/fld.3965), +International Journal for Numerical Methods in Fluids 77(5), 2015, pp. 249-273. + +R. Anderson, V. Dobrev, Tz. Kolev, D. Kuzmin, M. Quezada de Luna, R. Rieben and V. Tomov, +High-order local maximum principle preserving (MPP) discontinuous Galerkin finite element method for the transport equation +(https://doi.org/10.1016/j.jcp.2016.12.031), +Journal of Computational Physics 334, 2017, pp. 102-124. + +R. Anderson, V. Dobrev, Tz. Kolev, R. Rieben and V. Tomov, +High-order multi-material ALE hydrodynamics +(https://doi.org/10.1137/17M1116453), +SIAM Journal on Scientific Computing 40(1), 2018, pp. B32-B58. + +H. Hajduk, D. Kuzmin, Tz. Kolev and R. Abgrall, +Matrix-free subcell residual distribution for Bernstein finite element discretizations of linear advection equations +(https://doi.org/10.1016/j.cma.2019.112658), +Computer Methods in Applied Mechanics and Engineering 359, 2020. + +H. Hajduk, D. Kuzmin, Tz. Kolev, V. Tomov, I. Tomas and J. Shadid, +Matrix-free subcell residual distribution for Bernstein finite elements: Monolithic limiting +(https://doi.org/10.1016/j.compfluid.2020.104451), +Computers and Fluids 200, 2020.