feat: context-window guard for LLM document context

[Copilot]

Uploading too many files silently produced empty answers: oversized context caused the LLM API to raise BadRequestError, which the generic exception handler caught and returned "" — no user-visible indication anything went wrong.

Changes

• filehandling.py — two new utilities:

  • estimate_token_count(text) — tiktoken (cl100k_base) with len // 4 char-based fallback
  • truncate_context_to_token_limit(text, max_tokens) -> (str, bool) — proportional trim with a _TRUNCATION_SAFETY_MARGIN = 0.95 buffer; appends a visible [... context truncated ...] marker in the returned text

• __init__.py — two new config values replacing a blanket token limit:

  • context_window_headroom_tokens = 10_000 — tokens reserved for system prompts, per-question instructions, and the model's output
  • context_window_fallback_tokens = 128_000 — conservative fallback used when the active model has no context-window tag

• views.py:

  • process_llm_async — resolves the active model's context_window tag (set via AZURE_E{n}_TAGS_{TAG}=...,context-window:131072,...) and computes the PDF context budget as model_context_window - headroom; falls back to fallback - headroom with a log warning when the tag is absent; on truncation writes a non-error warning to assay.status_context (visible in the UI) advising the user to upload fewer/shorter files
  • generate_answer — adds a dedicated except BadRequestError block using logger.exception so any API-level rejection is logged with a full traceback rather than silently swallowed

• tests/test_context_window.py — unit tests for both utilities; integration tests asserting that process_llm_async populates status_context when over-limit (fallback path and per-model path) and leaves it clean when context fits

# views.py — budget derived from per-model context-window tag
_context_budget = model_entry.context_window - config.context_window_headroom_tokens
full_pdf_context, was_truncated = truncate_context_to_token_limit(
    full_pdf_context, _context_budget
)
if was_truncated:
    add_status_context(assay, "Uploaded documents exceeded the available context-window budget …", is_error=False)
    assay.save()

[johannehouweling]

we now have a context-window flag on the .env model TAGS, so we have access to per-model context-window information.
Hence instead of defining a blanket threshold context-window for context pdfs what we should do is define a headroom for overhead like prompt and instrcutions and then leave context-window minus headroom for the pdf context files..

[johannehouweling]

@copilot can you update the PR to take into account the newest changes to the repo in #138, we now have context-window information per model accessible in the .env model TAGS. so instead of hard-coding context-window size in init.py we have to define a head-room token size that reservers space in the context window (for prompt, output etc) and the difference we can give to the pdf context

[Copilot]

@copilot can you update the PR to take into account the newest changes to the repo in #138, we now have context-window information per model accessible in the .env model TAGS. so instead of hard-codin...

Done in commit 8e9d022. The blanket context_window_max_tokens has been replaced with two new config values:

• context_window_headroom_tokens = 10_000 — tokens reserved for system prompts, per-question instructions, and the model's output
• context_window_fallback_tokens = 128_000 — conservative fallback used when the active model has no context-window tag

In process_llm_async, the budget is now computed per-model:

budget = model_entry.context_window - config.context_window_headroom_tokens

If the model has no context-window tag (or the key can't be resolved), it falls back to fallback - headroom and logs a warning. The .env.dummy already shows context-window:131072 as an example, e.g.:

AZURE_E1_TAGS_GPT4O=...,context-window:131072,...