Configuration to not use local LLM calls

[AnilSorathiya]

Pull Request Description

What and why?

This PR adds a configuration option to disable local LLM calls and route all LLM requests through the ValidMind server. This is useful for environments where OpenAI access is blocked locally or when organizations want to ensure all LLM calls go through ValidMind's infrastructure.

Before: The SDK would attempt to use local OpenAI API keys (OPENAI_API_KEY or AZURE_OPENAI_KEY) for LLM operations like generating test result descriptions and running prompt validation tests. If these keys weren't available, operations would fail.

After: Users can now enable server-only mode via:

• The use_server_llm_only parameter in vm.init(use_server_llm_only=True)
• The VALIDMIND_USE_SERVER_LLM_ONLY environment variable (set to "1" or "True")

When enabled:

• Local LLM calls are blocked with a clear error message
• Test result descriptions continue to work (they already use server-side calls)
• is_configured() checks ValidMind API credentials instead of local OpenAI configuration
• Prompt validation tests that require judge LLM will raise an informative error directing users to contact support for server-side judge LLM support

How to test

1. Test server-only mode via parameter:

import validmind as vm

# Enable server-only mode
vm.init(
    api_key="your_key",
    api_secret="your_secret",
    use_server_llm_only=True
)

# Verify local LLM calls are blocked
from validmind.ai.utils import get_client_and_model
try:
    get_client_and_model()  # Should raise ValueError
except ValueError as e:
    assert "Local LLM calls are disabled" in str(e)

2. Test server-only mode via environment variable:

export VALIDMIND_USE_SERVER_LLM_ONLY=1
python -c "from validmind.ai.utils import get_client_and_model; get_client_and_model()"
# Should raise ValueError about local LLM calls being disabled

3. Test that test result descriptions still work:

import validmind as vm

vm.init(use_server_llm_only=True)
# Test result descriptions should work normally as they use server-side API

4. Run the comprehensive test suite:

python -m pytest tests/test_server_llm_only.py -v

The test suite covers:

• Parameter-based configuration
• Environment variable configuration
• Case-insensitive environment variable values
• Error handling when server-only mode is enabled
• Normal operation when server-only mode is disabled
• Integration with is_configured() function
• Prompt validation test error handling

What needs special review?

1. Error messages: Please review the error messages for clarity and user-friendliness, especially:

   • The error in get_client_and_model() that explains server-side routing
   • The error in call_model() for prompt validation tests

2. Environment variable handling: The implementation treats "1", "True", "true", "TRUE" as enabled and "0", "False", "false", "FALSE" as disabled. Verify this matches expected behavior.

3. Backward compatibility: Ensure that existing code without this configuration continues to work as before (local LLM calls should work normally when the flag is not set).

4. Integration points: Review how is_configured() now behaves differently in server-only mode - it checks ValidMind API credentials instead of attempting a local OpenAI ping.

Dependencies, breaking changes, and deployment notes

Dependencies: None

Breaking changes: None - this is a new optional feature that doesn't change existing behavior when not enabled.

Deployment notes:

• No special deployment considerations
• The feature is opt-in via configuration
• Existing users are unaffected unless they explicitly enable server-only mode
• Environment variable VALIDMIND_USE_SERVER_LLM_ONLY should be documented in deployment/environment configuration guides

Release notes

New Feature: Server-Only LLM Mode

Added support for disabling local LLM calls and routing all LLM requests through the ValidMind server. This is useful for environments where OpenAI access is blocked locally or when organizations want centralized LLM usage.

Enable server-only mode by:

• Setting use_server_llm_only=True in vm.init()
• Setting the VALIDMIND_USE_SERVER_LLM_ONLY environment variable to "1" or "True"

When enabled, test result descriptions continue to work via server-side calls. Prompt validation tests that require judge LLM will provide guidance on contacting support for server-side judge LLM support.

Checklist

• What and why
• Screenshots or videos (Frontend) - N/A (Backend feature)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut (SC-13495)
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[github-actions]

Pull requests must include at least one of the required labels: internal (no release notes required), highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.

[github-actions]

PR Summary

This PR introduces functionality to enforce a server-only mode for LLM calls. When enabled via the VALIDMIND_USE_SERVER_LLM_ONLY environment variable (or via an explicit parameter in the API client initialization), local LLM calls (e.g., using OpenAI) are disabled in favor of routing requests through the ValidMind server.

Key functional changes include:

1. In the API client initialization, an optional use_server_llm_only parameter has been added. If provided, it sets the corresponding environment variable to control whether local LLM calls are allowed.

2. In the LLM utilities (validmind/ai/utils.py), the function get_client_and_model() now checks for the server-only mode and raises an error if a local client is attempted to be created when the mode is enabled. Similarly, the is_configured() function has been updated to check for ValidMind API credentials when in server-only mode.

3. The prompt validation test (call_model in validmind/tests/prompt_validation/ai_powered_test.py) now immediately raises a ValueError if server-only mode is active, to avoid unintended local LLM access.

4. A comprehensive suite of tests (tests/test_server_llm_only.py) has been added. These tests verify various behaviors including:

   • Correct setting and unsetting of the VALIDMIND_USE_SERVER_LLM_ONLY environment variable based on the API client initialization parameters.
   • Proper error handling in server-only mode when local LLM calls are attempted (via get_client_and_model() and call_model).
   • Case-insensitivity of the environment variable, ensuring that different case variants (like "True" or "true") are correctly interpreted.
   • Behavior of the is_configured() function in both server-only and local modes.

Overall, the changes ensure that when server-side LLM support is required or when local OpenAI access is restricted, the application behaves consistently by preventing any local LLM invocations and routing requests through the ValidMind server.

Test Suggestions

• Run all unit tests, paying special attention to tests in tests/test_server_llm_only.py to confirm the correct behavior of the server-only mode.
• Test the behavior when the environment variable VALIDMIND_USE_SERVER_LLM_ONLY is set using different case variations (e.g., 'TRUE', 'false', etc.).
• Verify that the API client initialization correctly overrides the environment variable when the use_server_llm_only parameter is explicitly passed.
• Simulate scenarios with missing API credentials and ensure that is_configured() returns the expected boolean value in server-only mode.
• Manually test the end-to-end flow to confirm that error messages indicate the proper disabling of local LLM calls when server-only mode is active.

[github-actions]

Pull requests must include at least one of the required labels: internal (no release notes required), highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.