feat: Live stories with data caching and refresh#476
Draft
Conversation
- Add isLive and cacheTtlMinutes columns to storyVersion (pg + sqlite) - Add storyDataCache table for caching live story query results - Add live-story service for re-executing SQL queries - Add updateLiveSettings and refreshData mutations to story routes - Update getLatest and storyShare.get to use cached data for live stories - Add refreshData mutation to shared story routes Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
… data display - Add live story settings dialog with toggle and cache TTL configuration - Add useStoryViewerLiveSettings hook for managing live state - Add refresh button and live indicator badge in story viewer header - Support live data rendering in side-panel with backend-fetched query data - Add refresh button and live badge to full-page preview page - Add refresh button and live badge to shared story page Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
- Add migration 0026_live_stories for both SQLite and PostgreSQL - Add is_live and cache_ttl_minutes columns to story_version - Add story_data_cache table with composite PK (chat_id, story_id) - Add migration snapshots and journal entries - Fix unused import and formatting
When a new version is created (by agent or user), inherit isLive and cacheTtlMinutes from the latest existing version. This prevents losing live settings when the agent updates a story.
Contributor
🚀 Preview Deployment
Preview will be automatically removed when this PR is closed. |
Tooltip components require a TooltipProvider ancestor. Wrap each Tooltip usage in story-viewer, preview page, and shared story page with TooltipProvider, matching the existing codebase pattern.
When a live story refreshes its data, optionally use AI to regenerate the text analysis while preserving the story structure (charts, tables, grids stay identical, only text sections are updated). Backend: - Add story-text-regeneration service that parses story into a template with numbered text slots, sends to LLM with new query data, and reassembles the story with updated text - Add refreshText column to storyVersion - Add regeneratedCode column to storyDataCache - Wire text regeneration into refreshStoryData flow - Return regeneratedCode from getLatest and shared story routes Frontend: - Add 'Refresh text analysis' toggle (with Sparkles icon) in live settings dialog - Display regeneratedCode instead of original code when available - Migration 0027 for both SQLite and PostgreSQL
Replace the simple cacheTtlMinutes integer with a cacheSchedule text field storing cron expressions. This enables flexible refresh schedules: - 5 minutes, hourly, daily, weekly, monthly presets - Custom cron expression input - Natural language to cron conversion via LLM (uses extractor model) Backend: - Replace cacheTtlMinutes with cacheSchedule in schema (pg + sqlite) - Use cron-parser to check cache expiration against cron schedule - Add parseCronFromText mutation using LLM for NL-to-cron - Add cron-nlp service with validation via cron-parser - Migration 0028 with data migration from old TTL values to cron Frontend: - Redesign schedule picker with preset options + custom cron - Add cron expression manual input field - Add natural language input with Wand2 convert button - Show error feedback when NL conversion fails
Add a 'No cache (always fresh)' schedule option that re-executes each SQL query independently every time a chart or table is viewed. This is the most expensive but freshest option. Backend: - Add executeLiveQuery() to live-story service for single-query execution - Add story.getLiveQueryData and storyShare.getLiveQueryData query endpoints - Extract findSqlQueryById() and shared findSqlQueriesByIds() helper - Export NO_CACHE_SCHEDULE sentinel constant Frontend: - Add 'No cache (always fresh)' as first schedule option after manual - Add NoCacheStoryPreview with NoCacheChartEmbed/NoCacheTableEmbed that each independently call getLiveQueryData with staleTime: 0 - Each chart/table shows a loading spinner while fetching - Add no-cache support to full-page preview and shared story pages - Detect no-cache mode via cacheSchedule === 'no-cache' sentinel
- Merge origin/main (0026_add_log_table, 0027_add_shared_chat) - Consolidate our three migrations (0026, 0027, 0028) into single 0028_live_stories - Resolve code conflicts: ShareStoryDialog rename, notifySharedItemRecipients rename, isReadonlyMode addition - Regenerate package-lock.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements live stories (#312) and a data refresh button (#440) in combination. Users can now mark a story as "live" in the story editor, which enables re-running the original SQL queries to refresh the story's chart and table data — either manually via a refresh button or automatically via a configurable cron-based schedule. Optionally, AI can also regenerate the text analysis to match the new data while preserving the story's structure.
Key design decisions
getLiveQueryDataendpoint on every view. This is the freshest but most expensive option.story_data_cachetable and served from cache until the cron schedule triggers or a manual refresh occurs.cron-parser), supporting presets (5min, 1h, 24h, weekly, monthly), custom cron input, and natural language to cron conversion via LLM.isLive,cacheSchedule, andrefreshTextare stored onstory_versionrows. New versions inherit live settings from the previous version so settings aren't lost when the agent updates a story.message_parttool inputs (matching byquery_id), then re-executes them against the same FastAPI SQL endpoint used by the agent.refreshTextis enabled, the story is parsed into a template with numbered text slots and structural blocks (charts/tables/grids). The LLM receives the template + new query data and regenerates only the text sections, preserving the exact structure, heading levels, and formatting style.message_part. Text regeneration failures are also handled gracefully (keeps original text).Changes
Database
is_live(boolean),cache_schedule(nullable text for cron expression), andrefresh_text(boolean) columns tostory_versionstory_data_cachetable with composite PK(chat_id, story_id)storing cached query results, regenerated story code, and timestamp0026_live_stories,0027_live_story_text_regeneration, and0028_cron_schedulefor both SQLite and PostgreSQLBackend
services/live-story.ts— Service withrefreshStoryData()(re-executes all SQL queries, optionally regenerates text),getStoryQueryData()(cache-aware data fetcher with cron-based expiration), andexecuteLiveQuery()(single-query execution for no-cache mode)services/story-text-regeneration.ts— Parses story into structural template + text sections, calls LLM with new data to regenerate text while preserving structureservices/cron-nlp.ts— Converts natural language schedule descriptions to cron expressions using a small LLM, validates output withcron-parserqueries/story.queries.ts— AddedupdateLiveSettings(),getStoryDataCache(),upsertStoryDataCache(),collectSqlQueries(),findSqlQueryById(),resolveLiveSettings()trpc/story.routes.ts— AddedupdateLiveSettings,refreshData,getLiveQueryData, andparseCronFromTextmutations/queriestrpc/shared-story.routes.ts— AddedrefreshDataandgetLiveQueryData; updatedgetto return live status, cached data, and regenerated codeFrontend
NoCacheChartEmbedandNoCacheTableEmbedcomponents that independently fetch live data per-query withstaleTime: 0, showing loading spinners per chart/tableCloses