feat(ai): add clarification framework for ambiguous requests

[SOUMITRO-SAHA]

Summary

Enhanced the agent's system prompt to proactively ask for clarification when user queries are ambiguous, rather than making incorrect assumptions. This improves query accuracy and user experience by ensuring the agent has the necessary information before executing tools.
Type: Feature / Enhancement

Related Issue: #252

Changes Made

Modified Files

1. apps/backend/src/components/ai/system-prompt.tsx (208 → 273 lines)

   • Added new "Clarification & Ambiguity" section
   • Enhanced SQL Query Rules section
   • Updated Persona section to balance efficiency with accuracy
   • Added Examples section with concrete scenarios

2. apps/backend/tests/system-prompt.test.ts (7 → 21 tests)

   • Fixed import errors (formatDate, resolveTimezone)
   • Added 14 new tests for clarification behavior
   • All tests passing ✅

Testing

Unit Tests

• ✅ All 21 tests passing
• ✅ Tests cover clarification behavior, SQL rules, and persona changes
• ✅ Linting passes with no new errors

Manual Testing Needed

Before merging, please test with these scenarios:

1. Multiple Databases

   Query: "Show me the revenue data"
   Expected: Agent asks which database to use
   

2. Missing Date Range

   Query: "What's the total sales?"
   Expected: Agent asks for time period
   

3. Ambiguous Metric

   Query: "Analyze customer orders"
   Expected: Agent asks what aspect to analyze
   

4. Sufficient Detail

   Query: "Show me revenue from jaffle_shop for last month"
   Expected: Agent executes without asking
   

This is a pure prompt engineering change with no modifications to:

• Agent service logic
• Tool implementations
• Frontend components
• Database schema

[cubic-dev-ai]

No issues found across 2 files

[github-actions]

🚀 Preview Deployment

URL	https://pr-466-c7a4a1a.preview.getnao.io
Commit	c7a4a1a

⚠️ No LLM API keys configured - you'll see the API key setup flow when trying to chat.

---

_{Preview will be automatically removed when this PR is closed.}

[Bl3f]

Hello @SOUMITRO-SAHA, how did you come up with this idea? Was it because you used nao and you thought it was needed?

Regarding implementation I think this is something we should have in nao, but i'm not yet a fond of the current implementation, I think it takes too many tokens in the system prompt and maybe clarification questions should be a tool that might be interactive so we offer a better UX to users.

[MatLBS]

Thanks for this PR @SOUMITRO-SAHA ,

I would optimize the system-prompt, apply the changes and we will be ready to merge 😁

[SOUMITRO-SAHA]

Hello @SOUMITRO-SAHA, how did you come up with this idea? Was it because you used nao and you thought it was needed?

Regarding implementation I think this is something we should have in nao, but i'm not yet a fond of the current implementation, I think it takes too many tokens in the system prompt and maybe clarification questions should be a tool that might be interactive so we offer a better UX to users.

Hi @Bl3f , thank you for the feedback!

I actually came up with this based on my experience using nao. I noticed that the agent often assumes things when the query is not clear, which can lead to incorrect results. That’s why I felt we should guide it to ask for clarification in such cases.

I understand your concern about token usage and your suggestion of using a tool-based, interactive approach for clarification. I agree that it would provide a better UX and be more scalable.

That said, implementing a separate clarification tool feels like a larger change. For a v1 release, I was thinking this prompt-based approach could still be useful to improve behaviour quickly. We can then optimise it in v2 using a proper clarification tool, which I agree is a very good idea.

Happy to align with whatever direction you prefer.