[SC-7588] Explore dynamic llm context injection in test descriptions

[juanmleng]

Internal Notes for Reviewers

We aim to add test-specific context into the LLM-based result descriptions for individual tests to improve the quality of the insights and align the descriptions with use-case and bank policy requirements. This PR starts this effort by experimenting with setting global context using an environment variable in the notebook, which will be used to enhance the user prompt with external context related to the use case.

External Release Notes

Added the capability to include contextual information in LLM-based descriptions. The changes are primarily focused on improving the generation of descriptions for test results by incorporating additional context that can be specified through environment variables.

[johnwalz97]

Very nice! Looking forward to playing around with this!

[cachafla]

👏

[github-actions]

PR Summary

This pull request introduces enhancements to the ValidMind library by adding the capability to include contextual information in LLM-based descriptions. The changes are primarily focused on improving the generation of descriptions for test results by incorporating additional context that can be specified through environment variables.

Key Changes:

1. Notebook Enhancements: A new Jupyter notebook llm_descriptions_context.ipynb is added, demonstrating how to add context to LLM-based descriptions. It includes examples of setting up the environment, initializing datasets and models, and running tests with and without context.

2. Contextual Information Handling:

   • A new function _get_llm_global_context() is introduced in test_descriptions.py to retrieve context from environment variables. This function checks if the context is enabled and not empty before using it.
   • The generate_description function is updated to include the context in the input data for LLM description generation.

3. Template Update:

   • The user.jinja template is modified to conditionally include the context in the generated description if it is available.

These changes allow users to provide additional information about the test or use case, which can be useful for aligning LLM-generated insights with specific business requirements or policies.

Test Suggestions

• Test the notebook to ensure it runs without errors and produces expected outputs with and without context.
• Verify that the context is correctly retrieved from environment variables and included in the LLM descriptions.
• Check that the descriptions generated with context differ meaningfully from those without context.
• Test edge cases where the context is empty or the context-enabled flag is set to false.
• Ensure that the changes do not affect existing functionality when context is not used.