WIP: Feat instrument catalyst by julienfausty · Pull Request #2250 · GEOS-DEV/GEOS

julienfausty · 2023-01-24T15:01:06Z

Goal

This pull request should enable GEOSX to perform in-situ and in-transit (co-/post-)processing using the catalyst library and notably its newest implementation based on ADIOS2.

A little about the `catalyst` library

Catalyst is an API specification developed for simulations (and other scientific data producers) to analyze and visualize data in situ.

The Catalyst API uses ‘C’ and is binary compatible with different implementations of the API making it easier to change the implementation at runtime.

Relationship with ParaView: Starting with 5.9, ParaView releases come with an implementation of the Catalyst API. This implementation can be used in lieu of the stub to analyze and visualize simulation results using ParaView’s data-processing and visualization capabilities

Development Steps

Add new dependencies into thirdPartyLibs repository: Add catalyst and adios thirdPartyLibs#214
- catalyst itself version 2.0.0-rc3
- adios/catalyst implementation on commit af3b35f
- adios2 v2.8.3
- update conduit from 0.8.2 to 0.8.6
- update VTK to master branch 49361a2 for ABI namespace mangling
Refactor blueprint mapping in src/coreComponents/fileIO/Outputs/BlueprintOutput to be re-usable in derived classes
Create a src/coreComponents/fileIO/Outputs/CatalystOutput class that inherits from BlueprintOutput in order to use the already existing conduit blueprint mapping
Implement the adios specific meta-data in the conduit node
Updates for GCC13 compatibility
Write unit tests
Design example case

Conduit compatibility

When trying to input the conduit::Node used by GEOSX into the conduit_cpp::Node needed by catalyst we run into an incompatibility between the two conduit libraries which cannot coexist in the same translation unit. As of now and to my knowledge, catalyst does not have any ready-made solution for this type of collision with simulation codes.

This has been remedied in GEOSX with a facade design pattern in the Catalyst folder.

VTK compatibility with ParaView's Catalyst implementation

When trying to use the paraview implementation of catalyst (which can be found here: ParaViewCatalyst) there is a dlopen performed at runtime on the shared library where the implementation is compiled. This shared library has a dependence on the VTK that gets compiled with ParaView. However, when the program goes looking for VTK objects it finds the symbols in the native GEOSX-VTK library before the ParaView VTK. This leads to crashes due to the version differences between the two loaded VTKs.

This has been soloved by mangling GEOSX' VTK version to avoid collisions.

julienfausty · 2023-01-24T15:02:01Z

@sframba

This commit introduces a new output class for running in-situ co-processing using the catalyst API. It also introduces a new `Catalyst` module in the fileIO folder with specific code for dealing with `conduit` collisions.

This commit onward should be compatible with `thirdPartyLibrary` commit acf1e3836ec1cc89996222fabe9458f1a56c1f52 and onward

CharlesGueunet · 2023-09-15T14:49:49Z

src/coreComponents/fileIO/Outputs/CatalystOutput.cpp

+    std::string topoName = topology.name();
+    auto& iMesh = blueprintNode[topoName];
+
+    iMesh["state"].update(blueprintNode["state"]);


is this a large copy ?

CharlesGueunet · 2023-09-15T14:51:46Z

src/coreComponents/fileIO/Outputs/CatalystOutput.cpp

+  DispatchNameToType("uint32", std::uint32_t);
+  DispatchNameToType("uint64", std::uint64_t);
+  DispatchNameToType("float32", float);
+  DispatchNameToType("float64", double);


is this compiler / os dependant ?

CharlesGueunet · 2023-09-15T14:54:50Z

src/coreComponents/fileIO/Outputs/CatalystOutput.cpp

+                          ScopedDataContainer& dataContainer) {
+  std::string dtype = src.dtype().name();
+  conduit_index_t size = src.dtype().number_of_elements();
+#define DispatchNameToType(name, type)                                  \


in cpp 14 can't we use auto lambda instead of this macro switch case ?

CharlesGueunet · 2023-09-15T14:56:55Z

src/coreComponents/fileIO/Outputs/CatalystOutput.cpp

-  ::SanitizeNode(meshGEOSRoot);
+
+  bool isInTransit =
+      this->internal->implementation == "adios" ||


implementation do not have a unique name / id we could use ?

…talyst

untereiner · 2024-01-26T16:37:42Z

If you want to avoid conflicts handling / force push give a try to « external components » like this one and it’s tpl

…talyst

CharlesGueunet · 2024-02-15T14:15:53Z

@sframba I added a small change to allow for multiple simulations to use different adios files through a CATALYST_SST_FILENAME env variable.

…austy/GEOSX into feat-instrument-catalyst

rrsettgast · 2024-12-17T17:37:11Z

@CharlesGueunet Can you migrate the description on this PR to the superseding PR #3459

julienfausty mentioned this pull request Jan 24, 2023

Add catalyst and adios GEOS-DEV/thirdPartyLibs#214

Open

julienfausty marked this pull request as draft January 24, 2023 16:03

julienfausty force-pushed the feat-instrument-catalyst branch 2 times, most recently from 8bc8363 to 7a12751 Compare January 25, 2023 16:28

julienfausty force-pushed the feat-instrument-catalyst branch 4 times, most recently from a48620d to 101fcf6 Compare March 14, 2023 08:33

julienfausty force-pushed the feat-instrument-catalyst branch 4 times, most recently from c837d17 to 309e668 Compare April 5, 2023 16:00

julienfausty force-pushed the feat-instrument-catalyst branch from 309e668 to f831f78 Compare April 26, 2023 07:33

Julien Fausty added 2 commits July 20, 2023 13:21

build: add catalyst optional dependency

1ac86c4

BlueprintOutput: refactor conduit blueprint creation for reuse

f2cea4c

julienfausty force-pushed the feat-instrument-catalyst branch from f831f78 to 1235964 Compare July 20, 2023 15:03

Julien Fausty added 2 commits July 20, 2023 18:04

CatalystOutput: new output for in-situ capability

ae895f7

This commit introduces a new output class for running in-situ co-processing using the catalyst API. It also introduces a new `Catalyst` module in the fileIO folder with specific code for dealing with `conduit` collisions.

VTK: add mangling compatibility

69912ff

This commit onward should be compatible with `thirdPartyLibrary` commit acf1e3836ec1cc89996222fabe9458f1a56c1f52 and onward

julienfausty force-pushed the feat-instrument-catalyst branch from 1235964 to a3d549c Compare July 20, 2023 16:05

GCC13: fixes for warnings and errors

e9bcd67

julienfausty force-pushed the feat-instrument-catalyst branch from a3d549c to e9bcd67 Compare July 21, 2023 12:42

julienfausty force-pushed the feat-instrument-catalyst branch 2 times, most recently from 6880b62 to f9b5095 Compare August 9, 2023 17:15

julienfausty force-pushed the feat-instrument-catalyst branch from f9b5095 to 25d6bac Compare August 29, 2023 16:18

Catalyst: improve blueprint formatting for multi-region support

9c9eccf

julienfausty force-pushed the feat-instrument-catalyst branch from 25d6bac to 9c9eccf Compare August 30, 2023 12:29

CatalystOutput: remove now unrelevant shape information for adios

e8354fd

CharlesGueunet reviewed Sep 15, 2023

View reviewed changes

Merge remote-tracking branch 'public/develop' into feat-instrument-ca…

577732e

…talyst

CharlesGueunet and others added 4 commits February 14, 2024 17:10

Add python wrapping for Catalyst (attempt)

3482d36

Merge remote-tracking branch 'origin/develop' into feat-instrument-ca…

02cb389

…talyst

removed to python function declaration that are not really wrapped

cea54ad

Add multi-writer sst filename

7826b1c

sframba and others added 11 commits February 15, 2024 16:14

added Catalyst output to pygeosx.cpp

14b7f11

added python type

ab86833

Merge branch 'develop' into feat-instrument-catalyst

ace066c

Removed spurious parenthesis

34c6b25

VTK --> Catalyst leftover

a422203

Merge branch 'feat-instrument-catalyst' of https://github.com/julienf…

58bec57

…austy/GEOSX into feat-instrument-catalyst

Added leftover python include

6e1ec04

corrected bug in stub implementation

4f21bca

added correction from PR 2990 to test elastic workflow

f21bfbd

Merge branch 'develop' into feat-instrument-catalyst

76396a3

Merge branch 'develop' into feat-instrument-catalyst

f7f77e2

CharlesGueunet mentioned this pull request Nov 22, 2024

feat: instrument catalyst #3459

Open

7 tasks

rrsettgast closed this Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Feat instrument catalyst#2250

WIP: Feat instrument catalyst#2250
julienfausty wants to merge 23 commits intoGEOS-DEV:developfrom
julienfausty:feat-instrument-catalyst

julienfausty commented Jan 24, 2023 •

edited

Loading

Uh oh!

julienfausty commented Jan 24, 2023

Uh oh!

CharlesGueunet Sep 15, 2023

Uh oh!

CharlesGueunet Sep 15, 2023

Uh oh!

CharlesGueunet Sep 15, 2023

Uh oh!

CharlesGueunet Sep 15, 2023

Uh oh!

untereiner commented Jan 26, 2024

Uh oh!

CharlesGueunet commented Feb 15, 2024 •

edited

Loading

Uh oh!

rrsettgast commented Dec 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

julienfausty commented Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goal

A little about the catalyst library

Development Steps

Conduit compatibility

VTK compatibility with ParaView's Catalyst implementation

Uh oh!

julienfausty commented Jan 24, 2023

Uh oh!

CharlesGueunet Sep 15, 2023

Choose a reason for hiding this comment

Uh oh!

CharlesGueunet Sep 15, 2023

Choose a reason for hiding this comment

Uh oh!

CharlesGueunet Sep 15, 2023

Choose a reason for hiding this comment

Uh oh!

CharlesGueunet Sep 15, 2023

Choose a reason for hiding this comment

Uh oh!

untereiner commented Jan 26, 2024

Uh oh!

CharlesGueunet commented Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rrsettgast commented Dec 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

julienfausty commented Jan 24, 2023 •

edited

Loading

A little about the `catalyst` library

CharlesGueunet commented Feb 15, 2024 •

edited

Loading