Speed Up Vector Chat Creation Pipeline

[avishek0769]

Summary

Optimize vector ingestion path to reduce time from chat creation to READY.

Problem

Vector processing currently does sequential scraping + embedding with relatively small batching and can be slow for medium/large docs sets.

Expected Solution

Improve throughput using controlled concurrency, smarter batching, and optional caching for repeated pages/chunks.

Scope

• backend/chatWorker.js
• backend/utils/ragUtilities.js
• Qdrant write strategy and queue behavior
• Progress reporting logic

Acceptance Criteria

• Measured end-to-end ingestion time improves on representative docs sets.
• Throughput improvements do not degrade indexing correctness.
• Worker remains stable under concurrent jobs.
• Progress feedback remains accurate and monotonic.