[Trilinos] Migration to new package stack meta-issue

[maxfirmbach]

Description

This issue is for organizing and documenting progress of the migration from the old Epetra stack to the modern Tpetra stack. Specific tasks to be done are organized in issues with the respective label marking them. An overview of the progress can be found in Migration from Epetra to Tpetra.

Goals:

• Migrate the entire linear algebra stack to Tpetra
• Switch 4C's Trilinos dependencies to the Tpetra stack

Background

4C relies on Trilinos for the implementation of sparse linear algebra capabilities.
We currently use the Epetra stack, which will be phased-out from Trilinos by the end of 09/2025 (at the latest) -- and even right now, the Epetra stack is not actively developed anymore, so we cannot easily benefit from new features. Finally, Epetra is limited to MPI-only mode, so we cannot use on-node parallelism or accelerator devices (as they appear in more and more of todays computing clusters).

The modern Tpetra stack is the successor of Epetra. It is actively developed, it forms the basis for a modern solver stack in Trilinos, and comes with a hand-in-hand Kokkos integration for MPI+X parallelism.

The switch to Tpetra is mandatory, such that we can benefit from developments in Trilinos in the future.

The current strategy is to divide the transition into tree phases to ensure an orderly transition. The first phase includes preliminary steps in which the dependency on Epetra is reduced. If possible, the functionalities which still remains from Epetra are encapsulated by using wrappers.
In the second phase, the conversion of the sparse matrices will take place.
In the third and final phase, all remaining dependencies on Epetra that were not removed in the previous phases will be removed.

A presentation about packages, which will be archived can be found HERE (as presented at TUG 2023). Those include the following relevant packages for 4C:

• Epetra / EpetraExt
• Isorropia
• Intrepid
• Amesos
• AztecOO
• Ifpack
• ML
• NOX

In addition, Epetra code from all other packages will be removed.

Functionality is still ported from old to new solver stack till October 2024. If we use features from the old stack, which are not yet in the new one, there's still time to get them moved. After October 2024, the old solver stack out phasing will start till October 2025, to give stakeholders and users some final months for moving to the new stack. Afterwards all Epetra related code will be archived and thus won't be part of the repository anymore.

First Phase I

Before the actual migration to Tpetra can take place, some packages that support only Epetra have to be replaced by newer versions supporting both solver stacks. In addition this will be connected to heavy cleanup actions reducing complexity and thus simplifying the code paths taken in 4C.

Linear solver

• Amesos2: Direct solver package. The direct solver package is quite regularly used for prototyping and problems using difficult physics. As a lot of tests depend on this, it's quite time consuming to replace everything related to Amesos. (see [Trilinos] Move direct solver from Amesos to Amesos2 #661)

• Belos: Iterative solver package. The iterative linear solvers come into play for big problem sizes and are critical for performance running large parallel simulations. All the code already runs the new package, despite some parts in the partitioned fsi still depending on AztecOO, but mainly due to NOX. (see Remove instantiation of ::NOX::Epetra::LinearSystemAztecOO #1353)

• Muelu: Multigrid preconditioner package. We use multigrid for all kinds of problem types as preconditioner either for single-fields or multi-fields, which means this is an integral part of the linear solvers in 4C. Thus we need to switch and remove the old multigrid package ML. The plan is to use template xml-files tailored to certain problem types, which other developers can just use for their own simulations.

In the current state optional, but maybe beneficial:

• Ifpack: Incomplete factorization and additive Schwarz preconditioning package. We use it for all kinds of preconditioning, but have a quite rigid and hard coded entry point to it in our code. Moving the related parameters to an external file and reading this one in would allow us to be more flexible and make the switch to the successor Ifpack2 a bit more trivial (see [Ifpack] Move parameters and options to xml-file #365 Ifpack move parameters and options to xml file #400 [Ifpack] Move backend to Stratimikos #766).

• Teko: Block preconditioning package. Used to remove self implemented block preconditioners and thus simplify the overall code structure and also relevant for the switch to MueLu and Belos for the self implemented block-iterative preconditioner in fsi. It also allows us to delete our own internal amgnxn implementation (see Move remaining scatra to teko #972 [Teko] Remove amgnxn framework #754).

• Stratimikos: Trilinos interface package to linear solvers (Belos, Amesos2) and preconditioners (Ifpack, MueLu, Teko). Would likely reduce the amount of interface/adapter files currently used for different linear solvers (iterative, direct) and preconditioners (incomplete factorization, multigrid, block). This would also force us to think about our implementation of the linear solver class (see [Ifpack] Move backend to Stratimikos #766 [MueLu] Move backend to Stratimikos #1401 [solver] Move internals to Thyra objects #1400).

Nonlinear solver

• Remove Epetra dependency from NOX (see [NOX] Replace NOX::Epetra::Group with Tpetra counterpart #198 [Structure] Use NOX for non-linear solution procedure in new solid time integration #300 Reducing number of NOX-based solvers #359 [NOX] Clean up of NOX::Solver::SingleStep #379 [NOX] Remove "Modified" parameter list #406)

Discretization

• Zoltan/Zoltan2: Packages for graph coloring, graph and spatial domain partitioning and matrix reordering. We currently make use of Isorropia as old interface to Zoltan, the actual partitioning package. Zoltan2 is a mix of being an interface to Zoltan, but also implementing algorithms by itself. (see [Trilinos] Move from Isorropia to Zoltan2 #596 [Trilinos] Switch from Isorropia/Zoltan to Zoltan2/Zoltan #594)

• Intrepid2: Quadrature and basis function package. The code currently uses Intrepid to obtain quadrature rules for a certain cell type, to do the element local integration. We don't use anything else the package provides (which is strange, because getting local Jacobians, it's transposes or inverses is very easy at this point, but that's how it is). Besides that, we also provide a set of own quadrature rules not depending on Intrepid (this is again a classic, why have one thing doing quadrature when you can have several). We need to refactor the implementation based on Intrepid. (see [Trilinos] Refactor Gauss points and move to Intrepid2 #191 [Trilinos] Switch from Intrepid to Intrepid2 #192 [Intrepid2] Switch from Intrepid to Intrepid2 #222)

Linear Algebra Objects

Avoid the usage of Epetra in the physics modules by providing appropriate wrapper classes

• Matrix:
  • Remove occurrences of Epetra_CrsMatrix and replace by Core::LinAlg::SparseMatrix (see [Epetra] Work on Epetra_CrsMatrix replacement #761 [Epetra] Remove calls to epetra_matrix() #764 [Solver] Use SparseOperator instead of Epetra_Operator in linear solver #723)
• Vector:
  • Remove occurrences of Epetra_Vector and replace by Core::LinAlg::Vector
  • Remove occurrences of Epetra_IntVector and replace by Core::LinAlg::Vector
  • Remove occurrences of Epetra_FEVector and replace by Core::LinAlg::Vector (see [Epetra] Integrate Epetra_FEVector classes into our vector wrapper #84)
  • Remove occurrences of Epetra_MultiVector and replace by Core::LinAlg::Vector
• Graph:
  • Remove occurrences of Epetra_CrsGraph and replace by Core::LinAlg::Graph (see Create Wrapper for Epetra_CrsGraph #397 [epetra] Wrapper for Epetra_CrsGraph #403 [epetra] Apply clang-tidy in graph wrapper #498)
  • Remove occurrences of Epetra_FECrsGraph and replace by Core::LinAlg::Graph ([linalg] Make LinAlg::Graph use Epetra_FECrsGraph #404 [LinAlg] Add FE_Graph capability to Core::LinAlg::Graph #759)
• Map:
  • Remove occurrences of Epetra_Map and replace by Core::LinAlg::Map (see [LinAlg] Reduce Epetra_Map and Epetra_BlockMap occurances #552 Provide Wrapper for the Epetra map  #554)
  • Remove occurrences of Epetra_BlockMap and replace by Core::LinAlg::Map (see [LinAlg] Reduce Epetra_Map and Epetra_BlockMap occurances #552 Replace Epetra_BlockMap with Map #799)
• Misc:
  • Remove occurrences of Epetra_Import \ Epetra_Export and replace by Core::LinAlg::Transfer::Export \ Core::LinAlg::Transfer::Import (see [LinAlg] Create wrapper classes for Epetra_Import and Epetra_Export #755 Implement wrapper for Epetra import #827 Implement wrapper for Epetra Export #877)
  • Create a view for for LinAlg objects by using Core::LinAlg::View (see Create LinAlg::View for our wrappers #557 Enable LinAlg::View for Map #787)

Centralize calls to EpetraExt:

• Refactor matrix-matrix multiplication
• Refactor/evaluate matrix-matrix addition (see [EpetraExt] Evaluate the sparse matrix addition methods #136 [LinAlg] Refactor matrix add and multiply methods #1152)
• Refactor matrix transpose
• Remove matrix reindexing (see [EpetraExt] Remove reindexing of linear problem out of direct solver #675)

Framework

Start reducing Epetra dependent utilities with respective calls from Teuchos or remove Epetra dependency:

• Direct Blas/Lapack calls
• Time measurements
• MPI communicator (see Remove LinAlg::Map constructors taking Epetra_Comm #788)

First Phase II

There will be some preparation necessary related to further cleanup to make things go through smoothly (see #764 #763). Everything Epetra related needs to be reduced as much as possible, such that we end up with a minimal set of occurrences that are actually relevant for the switch.

Further preparation steps necessary:

• Assert every Epetra based call if a warning or error occurs (see [LinAlg] Move FOUR_C_THROW into respective wrapper classes #1190)
• Use static matrix profiles without dynamic allocations (see [LinAlg] Static sparsity layout for sparse matrices #1544)
• Remove all inheritance from Epetra based objects (see [LinAlg] Remove inheritance to Epetra objects #1557)
• Preparations to adjust our interface for the second phase (see [LinAlg] Clean-up of sparse algebra wrappers #875 [LinAlg] Refactor sparse operator, matrix and block-matrix wrapper #863)

This is also a good point to get some side quests done, that are not directly related to the switch, but to usability and Trilinos in general:

• Refactor linear solver and preconditioner input in 4C and update the respective documentation (see Update documentation of our solvers in readthedocs #68 [solver] Configuration and input procedure #1577 [solver] Reduce linear solver related input file parameters #1576 [solver] Unify preconditioner configuration file input variable #1575)

Second Phase

After everything is ready to be able to work with Tpetra we can think about migrating the functionality from Epetra sparse linear algebra to Tpetra sparse linear algebra.

• Discuss how to do the switch and integrate Tpetra: We decided to wrap the sparse linear algebra objects, remove utils functions as much as possible and try to move as many packages to the new ones by still depending on Epetra for now.

Linear solver

Call the respective linear solver with the relevant Tpetra objects.

Some packages only support either only Epetra or only Tpetra, we can only adapt them with the actual switch to Tpetra:

• Ifpack2: Additive-Schwarz preconditioner package.

Discretization

We need to feed Zoltan2 with the relevant Tpetra objects, thus we need to call the respective adapters with Tpetra

  using GraphAdapter = Zoltan2::XpetraCrsGraphAdapter<Tpetra::CrsMatrix, Tpetra::MultiVector>;
  using VectorAdapter = Zoltan2::XpetraMultiVectorAdapter<Tpetra::MultiVector>;

Linear Algebra Objects

Replace the Epetra members from sparse algebra wrappers by Tpetra. An overview issue is given in #875.

• Matrix:
  • Replace Epetra_CrsMatrix -> Tpetra::CrsMatrix
• Vector:
  • Replace Epetra_Vector -> Tpetra::Vector
  • Replace Epetra_MultiVector -> Tpetra::MultiVector
• Graph:
  • Replace Epetra_CrsGraph -> Tpetra::CrsGraph
• Map:
  • Replace Epetra_Map -> Tpetra::Map
• Misc:
  • Replace Epetra_Importer -> Tpetra::Importer and Epetra_Exporter -> Tpetra::Exporter

Replace the calls from EpetraExt by Tpetra:

• Replace EpetraExt::MatrixMatrix::Multiply -> Tpetra::MatrixMatrix::Multiply
• Replace EpetraExt::MatrixMatrix::Add -> Tpetra::MatrixMatrix::Add
• Replace EpetraExt::RowMatrix_Transpose -> Tpetra::RowMatrixTransposer

Framework

We need to replace as_epetra_comm() with a call to as_teuchos_comm() as Tpetra works with the communicator objects from Teuchos.

Third Phase

TBD

Quick links

• Open Issues
• Open MRs
• Milestone

How to participate?

If you want to participate in this effort (even temporarily), please assign yourself to this meta-issue, so that we have an overview of possible contributors. Every helping hand is very welcome!

[maxfirmbach]

I took the freedom to mark Intrepid2 as done. The remaining things to do are mainly related to backwards compatibility and Trilinos config cleanup.

[maxfirmbach]

Teko is basically done, the only thing that is "missing" is the deletion of amgnxn (see #754), such that all block systems are going through Teko, this is the main reason it's not marked as done yet.

[maxfirmbach]

I took the freedom to mark Amesos2 as done. The remaining things to do are mainly related to backwards compatibility and Trilinos config cleanup.