fix: improve pagination

[luke-lombardi]

Summary by cubic

Add real server-side pagination and free-text search for view rows, and return pagination metadata in responses. Also improves Copilot context selection, import speed, and mailbox coverage.

• New Features

  • ResolveData accepts offset, limit, and search (search applies before pagination) and returns Total with paged rows.
  • Store adds GetRowsPage and SearchRowsText; tool output now includes total, offset, and returned.
  • Copilot picks relevant rows via vector search keyed on the user’s message (falls back to recent) for smaller, targeted prompts.

• Refactors

  • Resolver fast-path serves import-sourced pages directly from MongoDB; in-memory search/pagination only when needed.
  • Imports use batch upserts (UpsertRowsNoEmbed), ensure vector index, and run EmbedRowsAsync; propagation uses parallel workers with a longer timeout.
  • Mailbox loads leaner data via GetRowsForMailbox, increases email fetch limit to 500, and also queries outputs by the view’s agents for completeness; search_text is rebuilt in the background on cell edits.

^{Written for commit eeeab06. Summary will update on new commits.}

[github-actions]

✅ Integration Tests PASSED

Gateway: https://api.airstore.ai
Query path: /sources/gmail/unread-emails
Timestamp: 2026-04-07T18:47:15Z

I/O Smoke Tests

Test	Status	Latency	Bytes
list_sources	PASS	190ms	-
list_gmail	PASS	213ms	-
list_query_path	PASS	526ms	-
read_file	PASS	330ms	1,724
stat_file	PASS	172ms	69,626

Compression A/B Tests

Metric	Value
Files tested	8
Total raw bytes	73,033
Total strip bytes	46,330
Total raw tokens	41,729
Total strip tokens	27,530
Tokens saved	14,199
Byte reduction	36%
Token reduction	34%
Min threshold	10%
Cache consistent	Yes

Per-file results

File	Raw B	Strip B	Raw Tok	Strip Tok	Tok Saved	Red%	Raw ms	Strip ms
2024-10-15_Harris-Walz_Organizing_Tea...	2,337	2,305	976	967	9	1%	325ms	202ms
2024-10-15_Stephen_King_via_KamalaHar...	2,904	2,866	1,116	1,103	13	1%	309ms	205ms
2025-07-17_KAYAK_How_to_have_everythi...	1,724	1,670	1,220	1,204	16	3%	178ms	173ms
2025-07-17_Alamo_Drafthouse_Cinema_Th...	19,047	18,224	12,228	11,704	524	4%	892ms	220ms
2025-07-17_Alamo_Drafthouse_Cinema_Th...	18,978	18,155	12,105	11,578	527	4%	441ms	243ms
2024-10-15_Knut_at_Sanity_Join_us_tom...	1,272	726	504	209	295	42%	356ms	196ms
2025-07-17_Lindsay_Stanley_TODAY_Conf...	1,876	1,049	657	268	389	44%	324ms	225ms
2025-07-17_NORDSTROM_RACK_FLASH_Travi...	24,895	1,335	12,923	497	12,426	94%	364ms	214ms

Charts

Overall Compression	Raw vs Compressed Bytes	Token Comparison
Reduction by File	Read Latency	I/O Latency

[cubic-dev-ai]

2 issues found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pkg/views/copilot.go">

<violation number="1" location="pkg/views/copilot.go:599">
P2: Use the raw user message for vector retrieval context; passing the expanded prompt (with attached file text) can skew semantic search results.</violation>
</file>

<file name="pkg/views/resolver.go">

<violation number="1" location="pkg/views/resolver.go:1857">
P1: Negative `offset` is not clamped before slicing, which can panic on `rows[offset:end]`.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}