What should logging the provenance of the a codelist look like

[CarolineMorton]

In an ideal world, the end product here would let a user recreate a codelist by following the log as a set of instructions. This would include where the codelist came from (i.e. loaded from file, manually created etc), what codes were added and when, what codes were removed, any manipulation of codes (like truncating or mapping - when we have that) etc.

If we start from the python api, i am imagining that we would have something like:

c = CodeList("Pneumonia", "ICD10", "Manually created", logging=True)

The logging would have a timestamp to allow us to order them.

[CarolineMorton]

@oylenshpeegul i'd be super interested to hear about your thoughts on this.

The first step would be to think about what the codelist.rs would look like with "logging" on. Specifically, I'm imagining a lightweight internal logger that captures key actions during the lifecycle of a codelist - something like a Vec of structured log lines that we could later write to a file, or expose via bindings.

As a first step, I'm picturing output along the lines of:

2025-05-26T14:03:12Z: Loaded Codelist from "codelist/pneumonia.txt"
2025-05-26T14:03:12Z: Added A09 Bacterial Pneumonia
2025-05-26T14:03:13Z: Removed A10 Viral Pneumonia

Initially this would just live inside the Codelist struct as a log field, but I expect it to grow over time. A few questions I’m thinking about:

Logging

• Would it be better to use an actual logging framework like tracing or log, or stick with something custom and internal for now? I am leaning towards a bespoke system to begin with, but since I want to call this from a web app eventually, there might be a case for adopting tracing or at least structuring the logging in a way that can be upgraded later.
• Where should the responsibility for formatting log messages sit? Inside the Codelist struct, or delegated to a helper or logger struct? Should the data structure just be a Vec, or something more structured that could support grouping, filtering, and output formats later?
• If we use an internal log, should we expose it with something like .get_log() or .drain_log()?
• Saving logs is straightforward, but loading is harder. One scenario is that you create a codelist, save both the codes and the log, and later want to reload both to continue working. That raises questions about how to load logs back in.
• Do we want to group log entries? For example, instead of separate lines for each code added

2025-05-26T14:03:12Z: Added 3 codes to Codelist: A09, A10, A11

Saving more widely

This also connects to something broader: I think we need to clarify and formalise the saving model. At the moment we have “save to JSON” and “save to CSV”, but these are very format-specific and slightly ambiguous about what’s actually being saved.

It might be better to distinguish between two types of saving:

• Saving codes: Just outputs the codes themselves, optionally with terms, comments, or other annotations. This is what you’d want if you were preparing something for use in an analysis or sharing a code list in a publication.
• Saving codelist: Serialises the full metadata, including provenance, categories, purposes, and potentially the internal log. This would be intended for reloading later into the same software and preserving the full context of the work.

That split would let us define clearer and more consistent APIs for saving and loading, and reduce confusion about what each method is actually doing.

Would love to know your thoughts - especially if you have any instinct on how to keep this ergonomic while still being flexible and future-proof givn that we want the python API to be pretty simple.

[oylenshpeegul]

I don't have any experience with tracing, but I've heard nothing but good things about it. If we do anything async, like a web app, that's surely the way to go. In the mean time, I'm guessing it would be fine to experiment with a Vec of lines. I don't think it would be hard to switch to log later.

On the other hand, I guess if we want a separate provenance file, then maybe something like log4rs is the way to go. This sort of reminds me of a binlog in a MySQL file. We could "replay" a provenance file to restore a codelist. If the log weren't part of the codelist, we wouldn't have to worry about reloading it. I guess this would apply to your second type of saving.

I think "save to JSON" and "save to CSV" would apply to your first type of saving.

[CarolineMorton]

Thanks for that @oylenshpeegul

I have had a think as well. I have split this project into a number of tasks:

• Logging Project: Design internal logging for Codelist lifecycle events #58
• Logging Project: Add get_log() and drain_log to rust API #59
• Logging Project: Define Python API for logging-enabled Codelist #60
• Logging Project: Split codelist save into save_codes, save_codelist and save_log #61
• Logging Project: Group log entries for bulk actions #62
• Logging Project: Design a structured logging format (optional for now) #63