PSv2: Async backend docs and diagram

[carlosgjs]

Add async ML backend diagram

Summary by CodeRabbit

• Documentation
  • Added comprehensive docs for the asynchronous ML backend architecture: system components and responsibilities, end-to-end async job flow with diagrams, API behaviors for job/task reservation and result submission, error handling, visibility/retry semantics, cancellation and lifecycle coordination guidance.

[netlify]

👷 Deploy request for antenna-ssec pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	0830b78

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	0830b78
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6994b7b21ab01800083867d7

[coderabbitai]

Warning

Rate limit exceeded

@carlosgjs has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 13 minutes and 46 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📝 Walkthrough

Walkthrough

Adds a new documentation page describing the async ML backend architecture, components (Django, Postgres, Celery, Redis, NATS JetStream, ML Worker), end-to-end async job/task flow, API shapes for GET /jobs/{id}/tasks and POST /jobs/{id}/result, design decisions, and error/cancellation semantics.

Changes

Cohort / File(s)	Summary
Documentation docs/diagrams/async_ml_backend.md	New doc: describes async ML backend architecture and data flow, lists components (Django, Postgres, Celery, Redis, NATS JetStream, ML Worker), includes a Mermaid sequence diagram, API request/response shapes for task reservation and result posting, design decisions (per-job streams, Redis state, reply_subject), and error/cancellation semantics.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Django
    participant Postgres
    participant Redis
    participant NATS
    participant MLWorker
    participant Celery

    Client->>Django: POST /jobs/{id}/result (or GET /jobs/{id}/tasks)
    Django->>Postgres: read/write job/task metadata
    Django->>Redis: update task reservation / visibility timeout
    Django->>NATS: publish to per-job stream (reply_subject for responses)
    NATS->>MLWorker: deliver task message (subscribe)
    MLWorker->>MLWorker: process task
    MLWorker->>Django: POST /jobs/{id}/result (uses reply_subject / API)
    Django->>Redis: acknowledge / update task state
    Django->>Celery: enqueue downstream processing (if needed)
    Celery->>Postgres: finalize results / persist outputs
    Django->>Client: respond with task/result acknowledgment

Loading

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Suggested reviewers

• mihow

Poem

🐰 I hopped through docs with joyful cheer,
I drew the streams both far and near,
NATS hums softly, Redis keeps time,
Tasks leap, results land—so sublime,
A rabbit's note: the backend's clear! 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is incomplete. It lacks a proper summary, list of changes, related issues, detailed description, testing information, and deployment notes as specified in the template.	Expand the description to include all required sections: summary, list of changes, related issues, detailed description, how to test, and deployment notes.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'PSv2: Async backend docs and diagram' directly and clearly describes the main change: adding documentation and diagrams for the async ML backend.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (2)

docs/diagrams/async_ml_backend.md (2)

85-98: Optional: Consider explaining the TTR vs. lock duration relationship.

The visibility timeout is 300 seconds (line 88) while the lock duration is 360 seconds (line 96). The 60-second buffer is likely intentional to ensure the lock covers the entire visibility window, but making this relationship explicit would help readers understand the coordination between NATS and Redis timing parameters.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/diagrams/async_ml_backend.md` around lines 85 - 98, The docs list a
Visibility Timeout (TTR) of 300s and a Redis Lock Duration of 360s but don't
explain their relationship; update the "Asynchronous Task Queue (NATS
JetStream)" and "Redis-Based State Management" sections to explicitly state that
the lock duration (360s) intentionally exceeds the TTR (300s) by a 60s buffer to
ensure the Redis distributed lock outlives the NATS visibility window,
preventing races where a task becomes visible again while the lock still holds
or vice versa; mention that this buffer covers clock skew and worker
reprocessing delays and note that the values (TTR and Lock Duration) must be
adjusted together if either is changed.

---

66-78: Consider clarifying the dual update pattern for pending images and progress.

The diagram shows two separate updates to both pending images (lines 67, 76) and job.progress (lines 71, 77), labeled as "process stage" and "results stage." The distinction between these stages and the rationale for updating the same state twice isn't explained elsewhere in the document.

If this accurately reflects the implementation, consider adding a brief note explaining why both stages are needed. If it's unintentional duplication, the flow could potentially be simplified.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/diagrams/async_ml_backend.md` around lines 66 - 78, The diagram shows
two updates to "pending images" (Redis) and "job.progress" (DB) labeled "process
stage" and "results stage" which is ambiguous; verify the worker implementation
(Celery task) to confirm whether both Redis updates and both DB job.progress
writes are required. If intentional, add a brief Note in the diagram (e.g.,
after "Celery->>Redis: Update pending images (remove processed)") explaining the
two-stage lifecycle: first update during per-image processing to reflect
work-in-progress and progress percentage, and second update in results stage to
record final state and release the lock; if unintentional, consolidate the two
Redis updates into a single "Update pending images" step and merge the two DB
writes into one "Update job.progress" step and adjust the surrounding sequence
(acknowledge_task, save detections) accordingly so the diagram matches the
actual Celery worker logic.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@docs/diagrams/async_ml_backend.md`:
- Around line 85-98: The docs list a Visibility Timeout (TTR) of 300s and a
Redis Lock Duration of 360s but don't explain their relationship; update the
"Asynchronous Task Queue (NATS JetStream)" and "Redis-Based State Management"
sections to explicitly state that the lock duration (360s) intentionally exceeds
the TTR (300s) by a 60s buffer to ensure the Redis distributed lock outlives the
NATS visibility window, preventing races where a task becomes visible again
while the lock still holds or vice versa; mention that this buffer covers clock
skew and worker reprocessing delays and note that the values (TTR and Lock
Duration) must be adjusted together if either is changed.
- Around line 66-78: The diagram shows two updates to "pending images" (Redis)
and "job.progress" (DB) labeled "process stage" and "results stage" which is
ambiguous; verify the worker implementation (Celery task) to confirm whether
both Redis updates and both DB job.progress writes are required. If intentional,
add a brief Note in the diagram (e.g., after "Celery->>Redis: Update pending
images (remove processed)") explaining the two-stage lifecycle: first update
during per-image processing to reflect work-in-progress and progress percentage,
and second update in results stage to record final state and release the lock;
if unintentional, consolidate the two Redis updates into a single "Update
pending images" step and merge the two DB writes into one "Update job.progress"
step and adjust the surrounding sequence (acknowledge_task, save detections)
accordingly so the diagram matches the actual Celery worker logic.

[Copilot]

Pull request overview

Adds architecture documentation for Antenna’s async ML job execution flow, illustrating how Django, Celery, Redis, and NATS JetStream interact with external ML workers.

Changes:

• Added a new Mermaid sequence diagram describing the async ML job lifecycle (queueing → reserving → result posting → ACK/progress updates).
• Documented key design decisions (per-job streams, TTR/max-deliver retries, Redis locking/state, reply-subject ACK pattern).
• Documented worker-facing task/result endpoints and example payloads.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The batch query parameter is documented as “(1-100)”, but the API currently only enforces min_value=1 and does not cap the maximum. Either remove the upper bound from the docs or add a corresponding max validation in the endpoint so documentation and behavior stay aligned.

Suggested change

	- `batch`: Number of tasks to fetch (1-100)
	- `batch`: Number of tasks to fetch (minimum 1)

[coderabbitai]

🧹 Nitpick comments (2)

docs/diagrams/async_ml_backend.md (2)

119-119: Add comma for clarity.

The sentence should have a comma before the main clause.

📝 Suggested grammar fix

-- If worker crashes and never reports a result or error NATS will redeliver after visibility timeout
+- If worker crashes and never reports a result or error, NATS will redeliver after visibility timeout

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/diagrams/async_ml_backend.md` at line 119, Update the sentence "If
worker crashes and never reports a result or error NATS will redeliver after
visibility timeout" by inserting a comma before the main clause so it reads
like: "If worker crashes and never reports a result or error, NATS will
redeliver after visibility timeout" (adjust surrounding articles for clarity if
desired); target the exact sentence in the document.

---

160-160: Hyphenate compound adjective.

The phrase "post processing results" should be hyphenated as "post-processing results" when used as a compound adjective modifying "results." As per static analysis hints, use a hyphen to join these words.

📝 Suggested grammar fix

-Worker endpoint to post processing results.
+Worker endpoint to post-processing results.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/diagrams/async_ml_backend.md` at line 160, Update the documented phrase
"Worker endpoint to post processing results." by hyphenating the compound
adjective so it reads "Worker endpoint to post-processing results." Locate the
string in docs/diagrams/async_ml_backend.md (the line containing "post
processing results") and replace it with the hyphenated form to satisfy the
static analysis/grammar check.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@docs/diagrams/async_ml_backend.md`:
- Line 119: Update the sentence "If worker crashes and never reports a result or
error NATS will redeliver after visibility timeout" by inserting a comma before
the main clause so it reads like: "If worker crashes and never reports a result
or error, NATS will redeliver after visibility timeout" (adjust surrounding
articles for clarity if desired); target the exact sentence in the document.
- Line 160: Update the documented phrase "Worker endpoint to post processing
results." by hyphenating the compound adjective so it reads "Worker endpoint to
post-processing results." Locate the string in docs/diagrams/async_ml_backend.md
(the line containing "post processing results") and replace it with the
hyphenated form to satisfy the static analysis/grammar check.