Open
Conversation
* related to darshan-hpcgh-729 and darshan-hpcgh-692 * draft infrastructure for skipping the processing of DXT data above a certain module compressed size threshold, in cases where runtime `HEATMAP` data is available * note that for the vast majority of log files that have been provided/problematic in this regard, including the large ones from NERSC, this is of no help, because there is no `HEATMAP` data to fall back on * for a case where this does help, on this branch: `time python -m darshan summary e3sm_io_heatmap_and_dxt.darshan` `real 0m12.415s vs. `main`: `real 0m47.470s` * so, that's not a bad improvement, but there are still many things to decide/do here: - [ ] test the size threshold empirically with more appropriate logs - [ ] decide if we want to use the current approach of summing the sizes of each DXT module together vs. having per-DXT module thresholds - [ ] decide if we'd also want a way to disable DXT handling even if `HEATMAP` is not available (otherwise, all the sample NERSC logs will use > 100 GB memory and be unusable with current report generation machinery) - [ ] add a warning mechanism/message somewhere on the report when the threshold is reached to disable DXT parsing - [ ] add a command line argument to force an override of the disable (if i.e., the user is working on a high memory node and really wants to see DXT results) - [ ] add regression tests for the new machinery
Collaborator
Author
|
Feedback from meeting:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related to ENH: use threshold to avoid plotting DXT heatmaps in logs with DXT and HEATMAP data #729 and ENH, CI:
e3sm_io_heatmap_and_dxt.darshanmemory usage causes CI error #692draft infrastructure for skipping the processing
of DXT data (for Python html report generation) above a certain module compressed size
threshold, in cases where runtime
HEATMAPdata isavailable
note that for the vast majority of log files that
have been provided/problematic in this regard, including
the large ones from NERSC, this is of no help, because there
is no
HEATMAPdata to fall back onfor a case where this does help, on this branch:
time python -m darshan summary e3sm_io_heatmap_and_dxt.darshanreal 0m12.415svs.
main:real 0m47.470sso, that's not a bad improvement, but there are still many
things to decide/do here:
the sizes of each DXT module together vs. having per-DXT
module thresholds
if
HEATMAPis not available (otherwise, all the sample NERSClogs will use > 100 GB memory and be unusable with current report
generation machinery)
threshold is reached to disable DXT parsing
(if i.e., the user is working on a high memory node and really
wants to see DXT results)