-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Description:
Currently, Ferret is a "post-mortem" analysis tool. It records span data to BeaverDB, and users must run ferret analyze after the fact to see what happened. This model is insufficient for debugging real-time performance issues, such as stuck processes, deadlocks, or live memory leaks.
To support real-time observability, we need a system that can visualize the current state of the application, including active (unfinished) spans and system-level metrics (CPU, RAM, Threads).
Proposed Solution:
- Event Sourcing Model: Transition the storage model from logging just "Finished Spans" to logging "Lifecycle Events" (
SPAN_STARTandSPAN_END). This allows a watcher to reconstruct the "currently open" call stack. - System Telemetry: Introduce a background thread in the Profiler that logs system metrics (CPU, Memory, Active Tasks) at a fixed interval.
- TUI Dashboard: Create a new CLI command
ferret watchusing Textual that connects to theBeaverDBin read-only mode, tails the log stream, and visualizes the live state.
Tasks:
- Data Model Updates (
ferret/models.py):- Define a generic
LogEntrymodel that can represent different event types:SPAN_START,SPAN_END, andMETRIC. - Update
SpanModelto support being serialized as partial events.
- Define a generic
- Profiler Logic (
ferret/core.py):- Update
Profiler.start()(and__enter__) to immediately write aSPAN_STARTevent to the non-blocking queue. - Update
Span.end()to write aSPAN_ENDevent. - Implement
SystemSampler: A daemon thread that wakes up every 1s, capturespsutilstats andthreading.active_count(), and writes aMETRICevent.
- Update
- The Watcher App (
ferret/tui.py):- Create a Textual application.
- Live Tailer: Implement a utility that reads
BeaverDBfrom the last known offset, yielding new records as they appear. - State Reconstruction: Maintain an in-memory tree of "Open Spans" based on the Start/End events received.
- UI Layout:
- Live Stack: A Tree widget showing currently running functions and their elapsed time.
- Metrics: Sparklines or numeric indicators for CPU/RAM.
- Recent Activity: A scrolling table of the most recently finished spans.
Acceptance Criteria:
ferret watchlaunches a TUI dashboard.- When the profiled script sleeps or performs a long task, the function appears in the "Live Stack" view immediately.
- When the function finishes, it disappears from the "Live Stack" and appears in the "Recent Activity" table.
- System metrics (CPU/RAM) update in real-time (approx. every second).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels