sc-11551: Improve Library dependencies

[johnwalz97]

Pull Request Description

What and why?

This PR represents a large effort to improve dependency handling and widen support for different environments. The following changes were made:

• Support for Python 3.12!
• Moving test-specific dependencies into optional "extras" groups and implementing graceful try-except blocks to let users know when they need to install said extras
• Removing unused dependencies and widening version constraints for existing dependencies
• Automated testing with tox to ensure different environments and version constraints are and remain working
• Code changes to fix some deperecations and other improvements to make all of the above work properly

How to test

Since this PR does represent quite a few changes, there was a big focus on adding automated testing as described above. Manual testing should include trying the library out with different Python versions and different existing Python environments and make sure that there are no regressions to existing features and functionality.

What needs special review?

Dependencies, breaking changes, and deployment notes

Release notes

Support for Python 3.12 has been added to the ValidMind library!

A large effort has been made to improve dependency management in an effort to make the core library lighter and ensure that its able to be easily installed in different existing Python environments

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[github-actions]

PR Summary

This PR introduces several important enhancements aimed at improving dependency management, testing, and error handling in the project. Key changes include:

1. Dependency Testing Workflows:

   • A new GitHub Actions workflow (.github/workflows/dependency-testing.yaml) has been added to test the compatibility of various dependency sets across multiple Python versions and dependency modes (default vs max). This workflow also includes a pip freeze testing job to validate client environments via the workflow_dispatch event and scheduled runs.
   • The existing Python workflow has been updated to install extras using a consolidated command (poetry install --all-extras) rather than separate ones.

2. Pyproject.toml and Packaging Updates:

   • The package metadata has been updated to use the new PEP 621 style in pyproject.toml. Dependency groups have been refined, and several optional dependencies have been reorganized under descriptive keys (e.g., nlp, stats, credit_risk, explainability).
   • The version check mechanism in the package has been updated: the code now uses importlib.metadata to retrieve the installed version, with a graceful fallback if the package metadata is missing.

3. Enhanced Error Handling for Missing Dependencies:

   • Numerous modules throughout the project (tests, dataset loaders, model validators) now wrap external dependency imports in try/except blocks. When an import fails, a custom MissingDependencyError is raised with a clear message and guidance on how to install the required extra dependencies (e.g., pip install validmind[nlp]).
   • This change improves the developer and user experience by providing actionable feedback for missing libraries in different contexts (stats, NLP, credit risk, etc.).

4. Minor Adjustments in Notebooks and Documentation:

   • Notebook samples have been updated to reflect correct import names (e.g., changing plotly_express to plotly.express) and minor metadata changes (e.g., nullifying execution count in code cells) to improve reproducibility and clarity.
   • The Makefile and other scripts have been slightly adjusted to ensure compatibility with these changes, particularly in API documentation generation.

Overall, these changes strengthen the dependency testing strategy, improve error transparency for missing dependencies, and modernize the project packaging structure.

Test Suggestions

• Run the new dependency-testing GitHub Actions workflow to verify compatibility across all specified Python versions and dependency configurations.
• Trigger the pip-freeze testing job manually via the workflow_dispatch interface to ensure that pip freeze files are correctly handled.
• Test the version check functionality by installing the package in different environments (with and without package metadata available) to confirm the logic works as expected.
• Simulate missing dependency scenarios (e.g., uninstall a dependency like 'scorecardpy' or 'datasets') to verify that the appropriate MissingDependencyError is raised with clear instructions.

[cachafla]

Very nice! Great to see 3.12 support!