feat: Integration tests sdk main#1187
Merged
edwinjosechittilappilly merged 17 commits intomainfrom Mar 19, 2026
Merged
Conversation
Move the Python SDK integration test to tests/integration/test_sdk.py and update Makefile targets (test-integration, test-ci, test-ci-local, test-sdk) to run that test path. Makefile now sets SDK_TESTS_ONLY=true when invoking pytest so SDK tests target an already-running external stack (removing uv sync/extra setup steps). Add sdks/python as an editable dev dependency in pyproject.toml for local SDK development. Update tests/conftest.py to skip in-process backend setup when SDK_TESTS_ONLY=true to avoid wiping external test state.
Replace the path-based editable dev dependency with the package name "openrag-sdk" in the dev dependency group, and add a [tool.uv.sources] entry mapping "openrag-sdk" to the local sdks/python path (editable = true). This keeps the dev dependencies referencing the package name while preserving a local editable source for development and packaging tools.
…i/openrag into integration-tests-sdk
Add explicit installation of the local Python SDK (uv pip install -e sdks/python) to Makefile test targets (test-ci, test-ci-local, test-sdk) so integration tests use the workspace sdks/python in editable mode. Remove openrag-sdk from pyproject dev dependencies, tool.uv sources, and uv.lock so the SDK is no longer declared as a managed dependency and is instead installed at test time.
Move integration tests into tests/integration/core and tests/integration/sdk (add __init__.py files) and update Makefile targets to run the new test layout. Replace single pytest run with separate Core and SDK test runs, add nicer console headings and GitHub Actions ::group:: markers, and adjust SDK test invocations (Python and TypeScript) and waiting/logging behavior in test-ci and test-ci-local. These changes separate core vs SDK test responsibilities and improve CI/console readability.
Add a comprehensive, modular pytest integration suite for the OpenRAG Python SDK. Introduces tests/integration/sdk/conftest.py (onboarding, API key creation/caching, client and test file fixtures) and separate test modules for auth, chat, documents, e2e, errors, filters, models, search, and settings. The old monolithic tests/integration/sdk/test_sdk.py is removed and replaced by these focused async tests. Tests respect SKIP_SDK_INTEGRATION_TESTS and use OPENRAG_URL (default http://localhost:3000).
Update integration tests to match recent SDK API changes: - chat.delete now returns False for nonexistent conversations instead of raising NotFoundError; test updated to assert False and removed the NotFoundError import. - Search result field renamed from `content` to `text`; tests updated to check `result.text` is present and a string.
Return a clear not-found result when deleting conversations. sdk/python/openrag_sdk/chat.py now catches NotFoundError and returns False for deletes that don't exist. src/services/chat_service.py returns a payload with not_found when local deletion fails and treats Langflow deletion as best-effort, including local_deleted and langflow_deleted flags. src/api/v1/chat.py maps the service not_found flag to a 404 JSON response so clients receive a proper HTTP status for missing conversations.
The key fix in agent.py: removed the except Exception: return False wrapper that was conflating "not found" with storage errors. Storage errors now propagate up to delete_session's except Exception block which correctly returns a 500, not a 404.
Three changes: 1. ensure_onboarding — changed to async def decorated with @pytest_asyncio.fixture, uses httpx.AsyncClient with await. No longer blocks the event loop. 2. get_api_key — replaced the sync helper with _fetch_api_key() as an async def that also uses httpx.AsyncClient. Cached in the same module-level variable, called with await from the client fixture. 3. client — changed to @pytest_asyncio.fixture / async def with yield so it properly await c.close() after each test, and can await _fetch_api_key() without blocking.
All 7 asyncio.sleep calls removed across 3 files: - test_search.py — 4 sleeps removed (lines 22, 35, 45, 71); import asyncio dropped - test_chat.py — 1 sleep removed (line 120); import asyncio dropped - test_e2e.py — 2 sleeps removed (lines 33, 66); import asyncio dropped ingest() defaults to wait=True, which already calls wait_for_task() internally and only returns once the task reaches a terminal state. No external sleep needed.
github-actions
bot
added
backend
Contributor
There was a problem hiding this comment.
Pull request overview
This PR formalizes a dedicated SDK integration test suite under tests/integration/sdk/, scopes “core” integration tests to tests/integration/core/, and aligns conversation deletion semantics across backend + Python SDK (404/not-found → False in SDK).
Changes:
- Split integration tests into
core/vssdk/, added extensive SDK integration test coverage + fixtures/docs. - Updated backend conversation deletion to distinguish “not found” from other failures and return 404 accordingly.
- Updated Python SDK
chat.delete()to returnFalseon 404 instead of raising.
Reviewed changes
Copilot reviewed 18 out of 22 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Separates core vs SDK integration test targets; improves CI log grouping and runs SDK tests from tests/integration/sdk/. |
tests/conftest.py |
Skips in-process backend onboarding when SDK_TESTS_ONLY=true to avoid mutating an external stack. |
tests/integration/sdk/* |
Adds SDK integration test suite, shared fixtures (conftest.py), and a QA checklist README. |
tests/integration/core/test_api_endpoints.py |
Adds app-level integration coverage for upload/search/router flows and readiness probing. |
tests/integration/core/test_startup_ingest.py |
Adds startup ingest integration coverage. |
src/services/chat_service.py |
Changes delete-session behavior and response shape (not-found vs success). |
src/api/v1/chat.py |
Maps delete-session “not found” to HTTP 404. |
src/agent.py |
Refines conversation deletion semantics and error propagation. |
sdks/python/openrag_sdk/chat.py |
Makes chat.delete() return False on 404 (catches NotFoundError). |
sdks/python/tests/test_integration.py |
Removes old single-file SDK integration tests (replaced by new suite). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+83
to
+90
| create_result = await client.knowledge_filters.create({ | ||
| "name": "Search Test Filter Python", | ||
| "description": "Filter for testing search with filter_id", | ||
| "queryData": {"query": "test", "limit": 5}, | ||
| }) | ||
| assert create_result.success is True | ||
| filter_id = create_result.id | ||
|
|
| "scoreThreshold": 0.0, | ||
| }, | ||
| }) | ||
| assert create_result.success is True |
Comment on lines
+31
to
+33
| with pytest.raises(Exception): | ||
| await dead_client.settings.get() | ||
| finally: |
Comment on lines
+39
to
+53
| try: | ||
| async with httpx.AsyncClient(timeout=30.0) as ac: | ||
| response = await ac.post( | ||
| f"{_base_url}/api/onboarding", | ||
| json=onboarding_payload, | ||
| ) | ||
| if response.status_code in (200, 204): | ||
| print("[SDK Tests] Onboarding completed successfully") | ||
| else: | ||
| print(f"[SDK Tests] Onboarding returned {response.status_code}: {response.text[:200]}") | ||
| except Exception as e: | ||
| print(f"[SDK Tests] Onboarding request failed: {e}") | ||
|
|
||
| _onboarding_done = True | ||
|
|
Comment on lines
+610
to
+618
| if not local_deleted: | ||
| return { | ||
| "success": False, | ||
| "not_found": True, | ||
| "error": "Conversation not found", | ||
| } | ||
|
|
||
| if not success: | ||
| error_msg = "Session not found in local storage or Langflow" | ||
| # Delete from Langflow using the monitor API (best-effort) | ||
| langflow_deleted = await self._delete_langflow_session(session_id) |
Comment on lines
+63
to
+75
| create_result = await client.knowledge_filters.create({ | ||
| "name": "Chat Test Filter Python", | ||
| "description": "Filter for testing chat with filter_id", | ||
| "queryData": {"query": "test", "limit": 5}, | ||
| }) | ||
| assert create_result.success is True | ||
| filter_id = create_result.id | ||
|
|
||
| try: | ||
| response = await client.chat.create( | ||
| message="Hello with filter", | ||
| filter_id=filter_id, | ||
| ) |
mpawlow
approved these changes
Mar 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base to main from Approved PR: #1164
This pull request introduces several improvements to integration testing, error handling, and developer documentation for the OpenRAG project. The changes enhance clarity and reliability of test runs, improve the robustness of conversation deletion logic, and add comprehensive documentation and fixtures for SDK integration tests. The most important changes are grouped below.
Integration Test Workflow Improvements:
Makefile, with improved output formatting and explicit test grouping for core and SDK tests. Only thecoresubdirectory is targeted for core integration tests, and SDK tests now run against a dedicatedsdkdirectory. [1] [2] [3] [4] [5] [6]SDK_TESTS_ONLYenvironment variable, skipping backend setup when running SDK-only tests to avoid interfering with an external stack.SDK Integration Test Support:
tests/integration/sdk/README.mdthat provides a detailed checklist of all SDK integration tests, their purposes, and expected outcomes, improving developer understanding and coverage tracking.tests/integration/sdk/conftest.pywith shared fixtures for onboarding, API key management, and test file creation, ensuring SDK tests are self-contained and reliable against a running OpenRAG instance.Error Handling and API Consistency:
deletemethod for conversations now returnsFalseif the conversation is not found, rather than raising an exception, aligning the API with user expectations.These changes collectively improve the reliability, clarity, and maintainability of both the integration testing process and the OpenRAG SDK's developer experience.
References:
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]