[SC-8445] fix: Apply post_process_fn to individual results in comparison tests

[juanmleng]

Internal Notes for Reviewers

When running comparison tests with input_grid or param_grid and a post_process_fn, the post-processing function is applied to the combined results rather than to individual test results. This can lead to an unexpected or unintuitive display of outputs, as post-processing functions are not associated with a specific input_id but are instead applied to the aggregated results.

External Release Notes

[github-actions]

PR Summary

This pull request introduces a new feature to the _run_comparison_test function in the validmind/tests/run.py file. The enhancement allows for an optional post_process_fn parameter, which is a callable function that can be applied to each TestResult after the test is run. This provides additional flexibility for users to modify or analyze test results after they are generated. The changes include:

• Addition of the post_process_fn parameter to the _run_comparison_test function signature.
• Modification of the test execution loop to apply the post_process_fn to each result if it is provided.
• Passing the post_process_fn parameter to the run_test function, ensuring that it is available for further processing.

These changes enhance the test framework by allowing custom post-processing of test results, which can be useful for logging, result transformation, or additional validation steps.

Test Suggestions

• Test the _run_comparison_test function with a valid post_process_fn to ensure it correctly modifies the TestResult.
• Verify that _run_comparison_test behaves as expected when post_process_fn is None.
• Check the integration of post_process_fn with run_test to ensure it is passed and utilized correctly.
• Test edge cases where post_process_fn might raise exceptions to ensure they are handled gracefully.

[johnwalz97]

I am not sure that I agree with the choice to run post-process functions for each individual result in a comparison test instead of on the final result. Although it does make sense to have the ability to do so. Maybe we should discuss in vm-library channel to see what the team thinks?

[AnilSorathiya]

This interesting point raised by @johnwalz97. Somewhat I agree with him. It would be clean if we keep post-process outside in the hand of users to process using the results/output of a test.

We have rawData available in the output of a test. @juanmleng can we use it in post-proces function to get desirable results?

[juanmleng]

@johnwalz97 raised a very valid point here, and made me think about this a bit more. I totally agree @AnilSorathiya If we add the input_id to the RawData in our tests and leave the user to handle the combined and individual results in the post-processing, we can indeed get to the desired results.

[juanmleng]

Closing this PR as we've decided that maintaining the current API is the best approach.

Next steps:

1. Create a new PR to add input IDs to RawData in our tests
2. This will better support post-processing in comparison tests
3. Document post-processing and RawData for comparison tests in our how-to guides

The existing RawData + processing_fn architecture already provides the flexibility we need, we just need to leverage it better.

[juanmleng]

Closing this PR as we've decided that maintaining the current API is the best approach.

Next steps:

1. Create a new PR to add input IDs to RawData in our tests
2. This will better support post-processing in comparison tests
3. Document post-processing and RawData for comparison tests in our how-to guides

The existing RawData + processing_fn architecture already provides the flexibility we need, we just need to leverage it better.