Result stores multiple times the case base when running apply_queries

[gjimenezUCM]

When running cbrkit.retrieval.apply_queries(...) it returns a Result object containing the results for each query (in result.queries).

For each query it stores a QueryResultStep not only the similarities but also the whole case base. For this reason, the same case base is stored multiple times. When dumping the results using a dumper it produces large files due to the repetition of this information.

Potential solutions

• Store the case base outside the QueryResultStep once. If several case bases can be introduced in one result (due to apply_batches) then store a Mapping of case bases and keep the casebase attribute in QueryResultStep as a reference to that mapping.

[mirkolenz]

Yes, that behavior is actually intentional to support retrieval pipelines in the fashion of MAC/FAC (many are called, few are chosen) where a simple pre-filter is applied to the entire casebase and only the best k cases are passed to the next stage. The case base in the result only stores those k cases to make it simpler to access them. But I also see the issue of unnecessary data that is produced when dumping them. As the result classes are pydantic models with fine-granular field options, I'll check if it's possible to add some way of easily removing the case base for exports.

[mirkolenz]

Seems to be more complex than anticipated. While easily possible to exclude the case base from serialization, I haven't found an easy way to make this configurable for users of the library without touching many parts of cbrkit. Before continuing with this I'd like to clarify your intended use case: Would the option to exclude the case base from the export help you at all or are you more interested in saving some storage by de-duplicating the exported case bases? As said earlier, the main issue here is that cbrkit supports retrieval pipelines where the case base changes between each step depending on the query, making it rather difficult to have an elegant solution for this scenario. If you'd like we could also set up a quick call to discuss that.

[mirkolenz]

Does the new function remove_cases in https://github.com/wi2trier/cbrkit/releases/tag/v0.28.8 help you?