Save basic run metadata and add indexing script#420
Save basic run metadata and add indexing script#420Antovigo wants to merge 13 commits intogoodfire-ai:devfrom
Conversation
- Extract _read_last_jsonl_line() to deduplicate backwards-seek JSONL pattern - Remove git_dirty fallback in _read_metadata() - Delete unused groups_cached variable - Skip reading metrics for uncompleted runs
Read duration from run_metadata.json and display it in the TSV index. Also round duration to 2 decimal places instead of 1.
Antovigo
changed the title
There was a problem hiding this comment.
Right now I wouldn't be willing to have this separate logging system for runs. I think run logging preferences are very different for different users, and supporting this would likely open up a lot of other issues/PRs which I don't think will be worth spending the time on now.
I think users can also manage their own run logging without too much work. I'd be open to adding a "notes" field in the main config. If you wanted specific labels you could use things like "label<>my_label" and search for that pattern in the notes field in your logging setup.
I suppose you'd also add a --notes flag to spd-run which pipes to all of the experiment configs that are part of that spd-run call (which may be one or more experiments).
I think this notes field should also be uploaded to the wandb "notes" field that appears in the "overview" tab.
Up to you whether you think this will help you enough to warrant making those changes.
2 participants
Description
A bare-bones way to manage a collection of SPD decomposition runs.
labelandnotes, to the SPD config. These have no effect on the decomposition, and are simply used to save arbitrary notes for future reference.labelis meant to identify a series of experiments (for example, side-by-side comparisons of hyperparameters).notescan be anything.run_metadata.jsonfile to the output folder. This file contains:labelandnotesoptional entries from configspd/scripts/index_spd_runs.py.This script scans all the runs in the SPD output directory, and generates a tab-separated list, sorted most-recent first, with relevant metadata from the
run_metadata.jsonfiles. In addition, the script compares the configs for all runs that share the samelabelvalue, finds which hyperparameters differ between them, and shows them in the "hyperparameters" column.The index looks like this:
The folder that contains the SPD runs can be overridden with the
-iargument, and the location of the index file can be set with-o.In the absence of
run_metadata.json, the scripts figures out whether the run was completed based onmetrics.jsonland the number of steps. The rest is left as NA.There is an optional argument,
--metrics, to add the final values of metrics of interest (e.g.loss/PGDReconLoss,l0/0.0_total) to the index file.By default, the script checks for a previous version of the index, and only re-processes the new runs. Use
--forceto re-process all the runs.Related Issue
N/A
Motivation and Context
By default, the outputs of SPD runs are all stored in the same folder, with uninformative folder names. This makes it difficult to keep track of which is which.
How Has This Been Tested?
- make check passes (basedpyright + ruff)
- I have been using it for a week so far. I find it pretty useful.
Does this PR introduce a breaking change?
No. It shouldn't disrupt anyone's workflow.