Skip to content

Issue 13: Expose /analyze/custom endpoint for BYO RAG systems #13

@SriMed

Description

@SriMed

Description

Add a second analyze endpoint that accepts question, generated answer, and pre-retrieved chunks directly — bypassing RAGBench and ChromaDB entirely. This is the integration point for anyone with an existing RAG system (OpenSearch, Pinecone, Weaviate, etc.) who wants to run the forensics layer on their own pipeline's output.

The forensics services themselves don't change — this is purely a new router entry point.

Implementation details

  • Add to models.py:
class CustomChunk(BaseModel):
    chunk_id: str
    text: str
    score: float

class CustomAnalyzeRequest(BaseModel):
    question: str
    answer: str
    chunks: list[CustomChunk]
  • Add POST /analyze/custom in routers/analyze.py
  • Convert CustomChunkRetrievedChunk, pass directly to forensics services
  • Return same AnalyzeResponse shape as /analyze
  • Add README_INTEGRATION.md with minimal Python example:
import requests
chunks = [
    {"chunk_id": "doc_42_chunk_3", "text": "...", "score": 0.87},
]
response = requests.post("https://your-deployment.railway.app/analyze/custom", json={
    "question": "What is the refund policy?",
    "answer": "Refunds processed in 5-7 days.",
    "chunks": chunks
})
print(response.json())

Acceptance criteria

  • POST /analyze/custom with valid inputs returns 200 with AnalyzeResponse
  • Empty chunks list returns 422
  • Missing question or answer returns 422
  • Chunk missing score field returns 400
  • Response shape identical to /analyze
  • Forensics services called with caller's chunks, not re-retrieved from ChromaDB
  • README_INTEGRATION.md exists with working minimal Python example
  • No RAGBench or ChromaDB code imported by the custom endpoint router

TDD

tests/test_analyze_custom.py (assert ChromaDB retriever mock NOT called):

  1. Valid request returns 200 with AnalyzeResponse shape
  2. Empty chunks returns 422
  3. Missing question returns 422
  4. Chunk missing score returns 400
  5. All six forensics dimensions present in response
  6. attribution_map length matches sentence count in provided answer
  7. ChromaDB retriever is NOT called

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendFastAPI / Python work

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions