feat: parallelize e2e tests with configurable concurrency (pytest-xdist)

[johntmyers]

Problem Statement

The e2e test suite (e2e/python/) runs ~36 tests sequentially in a single process. Each test spins up its own sandbox (K8s pod), waits for it to become ready (up to 300s), runs assertions, then tears it down. This makes the full suite slow — total wall time scales linearly with test count. Parallelizing with a configurable n-factor (e.g., 5 at a time) would significantly reduce wall time.

Technical Context

Tests are invoked via uv run pytest -o python_files='test_*.py' e2e/python (mise task test:e2e:sandbox). The sandbox fixture is function-scoped and creates isolated sandboxes per test, making individual tests naturally independent. However, several session-scoped fixtures (gRPC client, mock inference routes) use hard-coded resource names that would race under parallel execution. The standard pytest parallelization tool is pytest-xdist, which distributes tests across worker processes — each worker gets its own session-scoped fixtures.

Affected Components

Component	Key Files	Role
E2E test config	e2e/python/conftest.py	Session-scoped fixtures with hard-coded route names that would race
pytest config	pyproject.toml (lines 87-92)	Test runner configuration, no xdist settings today
mise task	build/test.toml (line 24)	test:e2e:sandbox task — needs -n flag support
Python deps	pyproject.toml (lines 29-39)	Dev dependencies — needs pytest-xdist

Proposed Approach

Add pytest-xdist as a dev dependency and update the mise task to accept a concurrency factor. Fix session-scoped fixtures in conftest.py that use hard-coded names — either make names unique per xdist worker, or use xdist's session-scope sharing mechanism (file lock) to create shared resources once across all workers. The sandbox fixture itself is already safe — it's function-scoped and creates isolated pods.

Scope Assessment

• Complexity: Low
• Confidence: High — clear path, well-understood tooling
• Estimated files to change: 3 (conftest.py, pyproject.toml, build/test.toml)
• Issue type: feat

Risks & Open Questions

• Cluster resource limits: Running N sandboxes concurrently requires the k3s node to have enough CPU/memory. May need to document recommended resource sizing or set a sensible default for N.
• Image pull thundering herd: If the sandbox image isn't cached on the node, N concurrent tests will all trigger image pulls simultaneously, potentially causing timeouts.
• Default concurrency value: Should default to sequential (-n0 / no flag) to avoid breaking existing workflows, or should a default like -n auto be set?
• CI impact: The e2e CI workflow (.github/workflows/e2e.yml) runs in a privileged container — need to verify it has enough resources for parallel sandboxes.

Test Considerations

• This is a change to test infrastructure itself, so validation is running the e2e suite with -n 5 (or similar) and confirming all tests pass without flakes or races
• Verify that session-scoped fixtures (mock inference routes) are correctly isolated per worker or shared safely
• Verify no test ordering dependencies exist (tests should already be independent, but parallelism will expose any hidden coupling)

---

Created by spike investigation. Use build-from-issue to plan and implement.

[johntmyers]

🔬 spike-agent

Technical Investigation

Architecture Overview

The e2e test suite is pytest-based, invoked via mise run test:e2e:sandbox which runs uv run pytest -o python_files='test_*.py' e2e/python. Tests are fully synchronous despite asyncio_mode = "auto" being configured.

The test lifecycle per test is:

1. sandbox() fixture factory is called - returns a Sandbox context manager
2. Sandbox.__enter__ (python/navigator/sandbox.py:474-494): creates a gRPC SandboxClient, calls CreateSandbox, then polls wait_ready() with 300s timeout
3. Test body runs - typically sb.exec_python() or sb.exec() calls
4. Sandbox.__exit__ (python/navigator/sandbox.py:496-517): calls DeleteSandbox, polls wait_deleted() up to 60s

Session-scoped resources (created once per pytest session):

• sandbox_client - gRPC channel to the cluster
• ensure_sandbox_persistence_ready - blocks until server sqlite is initialized
• inference_client - gRPC client for inference route management
• mock_inference_route, mock_anthropic_route, mock_disallowed_route - named inference routes

Code References

Location	Description
e2e/python/conftest.py:19-22	cluster_name session fixture - reads NEMOCLAW_CLUSTER env var
e2e/python/conftest.py:24-27	sandbox_client session fixture - gRPC connection via SandboxClient.from_active_cluster()
e2e/python/conftest.py:30-52	ensure_sandbox_persistence_ready - polls sqlite readiness, 60 retries
e2e/python/conftest.py:55-67	sandbox function fixture - factory returning Sandbox context manager
e2e/python/conftest.py:76-100	mock_inference_route - creates route named "e2e-mock-local" (hard-coded)
e2e/python/conftest.py:103-127	mock_anthropic_route - creates route named "e2e-mock-anthropic" (hard-coded)
e2e/python/conftest.py:130-155	mock_disallowed_route - creates route named "e2e-mock-disallowed" (hard-coded)
build/test.toml:20-24	mise task test:e2e:sandbox - invokes pytest
pyproject.toml:29-39	Dev dependencies - no pytest-xdist present
pyproject.toml:87-92	pytest config - addopts = "-v --tb=short", no parallel settings
.github/workflows/e2e-test.yml	CI workflow - 30-min timeout, single-threaded invocation

Current Behavior

All ~36 tests run sequentially in a single pytest process. The sandbox fixture is function-scoped, so each test creates and destroys its own sandbox pod. Session fixtures are initialized once at the start of the run.

The mock inference route fixtures follow a create-or-replace pattern: delete any leftover route with the hard-coded name, then create it fresh. At session teardown, they delete the route. This pattern is safe for single-process execution but would race with multiple workers.

What Would Need to Change

1. Add pytest-xdist dependency (pyproject.toml):
Add "pytest-xdist>=3.0" to the [dependency-groups] dev list.

2. Fix session-scoped inference route fixtures (e2e/python/conftest.py):

With xdist, each worker (gw0, gw1, ...) gets its own session-scoped fixtures. Three options:

• Option A (Recommended): Worker-unique names. Append the xdist worker ID to route names. Use the built-in worker_id fixture from xdist (returns "gw0", "gw1", etc. for workers, or "master" for non-xdist runs). Routes become "e2e-mock-local-gw0", "e2e-mock-local-gw1", etc. Simple, no shared state, no locks.

• Option B: Shared fixture via file lock. Use pytest-xdist's tmp_path_factory + file lock pattern to create routes once across all workers. More complex, requires careful teardown coordination.

• Option C: Create routes outside pytest. Move route setup to a pre-test script or mise task dependency. Simplest operationally but separates test setup from the test framework.

Option A is recommended - it's the simplest, keeps everything in pytest, and each worker is fully independent.

3. Update mise task (build/test.toml):

Add a concurrency flag. Example approaches:

• Use an environment variable: E2E_PARALLEL=5 mise run test:e2e:sandbox -> task reads it and passes -n $E2E_PARALLEL
• Default to sequential (no -n flag) for backwards compatibility
• Or default to -n auto (xdist auto-detects CPU count) for faster default experience

4. Verify no test ordering dependencies:

Quick audit of the test files shows tests are independent - each creates its own sandbox with its own policy spec. No shared mutable state between tests beyond the session fixtures (which are read-only after setup).

Alternative Approaches Considered

Approach	Pros	Cons
pytest-xdist (recommended)	Industry standard, well-maintained, simple -n flag	Each worker gets own session fixtures (need to handle route names)
pytest-parallel	Thread-based option available	Less mature, threading can cause issues with gRPC
Custom subprocess harness	Full control	Reinventing the wheel, maintenance burden
pytest --forked	Process isolation per test	No concurrency control (n-factor), just isolation

Patterns to Follow

The existing conftest.py already follows good patterns:

• Factory fixtures for sandbox creation (function-scoped)
• Session-scoped fixtures for expensive shared resources
• Context manager lifecycle for setup/teardown

For xdist compatibility, the standard pattern is to use the built-in worker_id fixture that pytest-xdist provides. It returns "gw0", "gw1", etc. for workers, or "master" for non-xdist runs. Route fixtures just need to accept worker_id and incorporate it into the name.

Test Coverage Notes

There are no tests for the test infrastructure itself. Validation would be:

1. Run uv run pytest -n 5 -o python_files='test_*.py' e2e/python against a running cluster
2. Verify all tests pass
3. Verify no flakes from route name collisions or resource contention
4. Verify sandboxes are properly cleaned up (no leaked pods)

---

This investigation provides context for implementation. Next step: review the issue, refine if needed, then use build-from-issue to create a plan and implement.

[johntmyers]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: Low
Confidence: High — clear path, well-understood tooling

Summary

Add pytest-xdist to enable parallel e2e test execution with a configurable concurrency factor. Default to 5 concurrent workers. Fix session-scoped inference route fixtures to use worker-unique names/routing hints, and update the tests that consume them to use fixture values instead of hard-coded routing hints.

Scope

• pyproject.toml: Add pytest-xdist>=3.0 to dev dependencies
• e2e/python/conftest.py: Make mock route fixtures worker-safe by incorporating xdist worker_id into route names and routing hints; yield routing hints instead of names so tests can use them in policies
• e2e/python/test_inference_routing.py: Use routing hints from fixtures instead of hard-coded strings in sandbox policies
• build/test.toml: Default to -n 5, support E2E_PARALLEL env var override (number or "auto" for CPU count)

Implementation Steps

1. Add pytest-xdist>=3.0 to [dependency-groups] dev in pyproject.toml and run uv sync
2. Update e2e/python/conftest.py:
   • Add a session-scoped _worker_suffix fixture that returns "" for non-xdist runs ("master") or "-gwN" for xdist workers
   • Update mock_inference_route to use f"e2e-mock-local{suffix}" / f"e2e_mock_local{suffix}" and yield the routing hint
   • Update mock_anthropic_route to use f"e2e-mock-anthropic{suffix}" / f"e2e_mock_anthropic{suffix}" and yield the routing hint
   • Update mock_disallowed_route similarly, yielding the routing hint
3. Update e2e/python/test_inference_routing.py:
   • test_inference_call_routed_to_backend: pass mock_inference_route (now routing hint) to _inference_routing_policy()
   • test_non_inference_request_denied: same — pass fixture value to policy helper
   • test_inference_anthropic_messages_protocol: use mock_anthropic_route fixture value in allowed_routes list instead of hard-coded "e2e_mock_anthropic"
   • test_inference_route_filtering_by_allowed_routes: pass mock_inference_route to policy helper
4. Update build/test.toml: default test:e2e:sandbox to -n 5; support E2E_PARALLEL env var to override (accepts a number or the string "auto" for CPU-count matching)
5. Run mise run pre-commit to validate

Test Plan

• Unit tests: N/A — this is test infrastructure, not application code
• Integration tests: N/A
• E2E tests: Validate by running the existing e2e suite (requires a live cluster). Without E2E_PARALLEL override, runs with 5 workers by default. E2E_PARALLEL=1 restores fully sequential execution.

Risks & Open Questions

• Cluster resource capacity for concurrent sandboxes is a deployment concern, not a code concern — the configurable n-factor lets users tune to their cluster
• Concurrent image pulls on cold nodes could hit the 300s ready timeout — mitigated by pre-pulling images before running parallel tests
• Default of 5 is a reasonable middle ground; users can set E2E_PARALLEL=1 to restore sequential behavior or E2E_PARALLEL=auto to match CPU count

Documentation Impact

• None expected — this is internal test infrastructure

---

Revision 2 — default to -n 5, support E2E_PARALLEL env var with number or "auto"
Revision 1 — initial plan

[johntmyers]

🏗️ build-from-issue-agent

Implementation Complete

PR: #102

What was built

Added pytest-xdist for parallel e2e test execution. Defaults to 5 concurrent workers, configurable via E2E_PARALLEL env var (number or "auto"). Made session-scoped mock inference route fixtures worker-safe by incorporating xdist worker IDs into route names and routing hints.

Tests

• Unit: N/A (test infrastructure change)
• Integration: N/A
• E2E: Existing tests are the validation — fixture changes ensure no races under parallel execution

Docs updated

• None needed

The issue will auto-close when the PR is merged.