Add connection resilience layer with heartbeat and network awareness

CHANGELOG.md

@@ -14,6 +14,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Visible agent selection in the quick dispatch card so tasks can be routed to different agents intentionally
 - Full transcript reader with paginated history, scrollback loading, and timestamped session output
 - Session loading panel that appears immediately after launch so users see progress before terminal output arrives
+- **Connection heartbeat:** server sends a `ping` every 15 seconds and closes the socket if no `pong` arrives — dead connections are now detected in under 20 seconds instead of waiting for TCP timeout
+- **Client-side ping watchdog:** client force-closes and reconnects if no server ping is received for 35 seconds, catching the case where the TCP socket is silently stale (common after phone sleep with an expired NAT entry)
+- **Network-aware reconnect:** listening on the browser `online` event immediately cancels any pending backoff timer and opens a fresh WebSocket when the device changes networks (Wi-Fi↔cellular switch, airplane mode off, etc.)
+- **Improved visibility reconnect:** page-visibility handler now detects sockets stuck in `CONNECTING` state — a wake-from-sleep artifact — and replaces them immediately rather than waiting for the connection attempt to time out
 
 ### Changed
 

README.md

@@ -19,6 +19,7 @@ Start an agent on your laptop, walk away, and check in from your phone or tablet
 - **Transcript logs:** Shows the full session output in a scrollable, timestamped transcript view.
 - **QR code pairing:** Scan a QR code from your terminal to authenticate your phone. No passwords or SSH keys.
 - **Persistent sessions:** Sessions run inside `tmux`. Your agent keeps working if your laptop sleeps or your connection drops. Reconnect and pick up where you left off.
+- **Resilient connection:** A server-side heartbeat and client-side watchdog detect dead connections in under 20 seconds. The browser's network-change event triggers an immediate reconnect when you switch between Wi-Fi and cellular — no waiting for backoff timers to drain.
 - **Flexible networking:** Works on local Wi-Fi, over Tailscale (private network), or via Cloudflare Tunnels (no port-forwarding needed).
 - **Git worktree isolation:** Run agents in isolated `git worktrees` to keep your working directory clean.
 
@@ -158,6 +159,9 @@ CloudCode uses a small Go-based sidecar to interface with UNIX pseudo-terminals
 **What happens if my laptop goes to sleep while an agent is running?**
 The agent keeps running. Sessions are managed by `tmux`, which is independent of CloudCode's web server. Your agent's process continues as long as the machine is powered on. When you reconnect, CloudCode picks the session back up.
 
+**What happens to my phone's connection when it sleeps or switches networks?**
+CloudCode's connection layer is designed for exactly this. A server-side heartbeat ping detects that your phone's WebSocket is gone within 15 seconds rather than waiting for TCP's multi-minute timeout. On the client side, the browser's `online` event fires the moment a new network interface is ready (e.g. after a Wi-Fi→cellular switch), triggering an immediate reconnect without cycling through an exponential backoff queue. In practice, the terminal is back live within a few seconds of your phone waking up or changing networks.
+
 **Can I run multiple agents at the same time?**
 Yes. Each session is an independent `tmux` window. You can run as many concurrent sessions as your machine can handle and manage them all from the dashboard.
 

backend/src/terminal/routes.ts

@@ -132,7 +132,7 @@ const terminalRoutes: FastifyPluginAsync = async (fastify) => {
   fastify.get('/ws/terminal', { websocket: true }, (connection: any, request) => {
     // In @fastify/websocket v11, connection might be the socket itself or contain a socket
     const ws = connection.socket || connection;
-    
+
     if (!ws || typeof ws.send !== 'function') {
       fastify.log.error({ connection: !!connection }, 'Invalid WebSocket connection object');
       return;
@@ -142,6 +142,26 @@ const terminalRoutes: FastifyPluginAsync = async (fastify) => {
     let attachPromise: Promise<void> | null = null;
     let lastSize = { cols: 80, rows: 24 };
 
+    // Heartbeat: detect silent/dead connections (e.g. phone sleep, network change).
+    // Server pings every 15s; if no pong arrives before the next ping, the connection
+    // is considered dead and closed. This mirrors mosh's approach of actively probing
+    // the client-to-server link rather than waiting for TCP to eventually time out.
+    const HEARTBEAT_INTERVAL_MS = 15_000;
+    let heartbeatAlive = true;
+    const heartbeatTimer = setInterval(() => {
+      if (!heartbeatAlive) {
+        cleanupHeartbeat();
+        ws.close(1001, 'Ping timeout');
+        return;
+      }
+      heartbeatAlive = false;
+      if (ws.readyState === 1) {
+        ws.send(JSON.stringify({ type: 'ping', timestamp: Date.now() }));
+      }
+    }, HEARTBEAT_INTERVAL_MS);
+
+    const cleanupHeartbeat = () => clearInterval(heartbeatTimer);
+
     const cookieToken = request.cookies?.['session'];
     const queryToken = (request.query as Record<string, string>)['token'];
     const token = cookieToken ?? queryToken;
@@ -159,6 +179,7 @@ const terminalRoutes: FastifyPluginAsync = async (fastify) => {
         type: 'session.error',
         message: 'Authentication required',
       }));
+      cleanupHeartbeat();
       ws.close(1008, 'Unauthorized');
       return;
     }
@@ -242,6 +263,11 @@ const terminalRoutes: FastifyPluginAsync = async (fastify) => {
               break;
             }
 
+            case 'pong': {
+              heartbeatAlive = true;
+              break;
+            }
+
             case 'terminal.input': {
               if (!ptySession && attachPromise) {
                 await attachPromise;
@@ -295,12 +321,14 @@ const terminalRoutes: FastifyPluginAsync = async (fastify) => {
     });
 
     ws.on('close', () => {
+      cleanupHeartbeat();
       void ptySession?.close().catch(() => {});
       ptySession = null;
       attachedSession = null;
     });
 
     ws.on('error', () => {
+      cleanupHeartbeat();
       void ptySession?.close().catch(() => {});
       ptySession = null;
       attachedSession = null;

docs/how-it-works.md

@@ -17,13 +17,33 @@ Standard mobile keyboards are missing critical developer keys (`Ctrl`, `Esc`, `T
 *   **Haptic Feedback**: Every keypress provides a subtle vibration, making the virtual terminal feel tactile and responsive.
 *   **Live PTY Stream**: CloudCode uses a dedicated PTY sidecar to attach to tmux and stream raw terminal bytes to the browser, preserving interactive terminal behavior with scrollback and fewer rendering artifacts.
 
-### 3. Secure Remote Access
+### 3. Connection Resilience
+
+tmux guarantees the *agent* survives any disruption — but the *connection* between your phone and the server is a separate problem. Standard WebSocket over TCP has the same fragility as SSH: a network change kills the socket silently, and neither side knows until TCP's own timeout fires (which can take minutes).
+
+CloudCode uses a layered approach to detect and recover from these failures as fast as possible:
+
+**Server-side heartbeat**
+The server sends a `ping` message every 15 seconds. If a `pong` does not arrive before the next ping interval, the connection is declared dead and closed immediately. This bounds the detection window to under 20 seconds instead of waiting for TCP's multi-minute timeout.
+
+**Client-side ping watchdog**
+The client tracks the timestamp of the last server ping. If no ping has been received for 35 seconds — a signal that the TCP socket is silently dead — the client force-closes the socket and starts a fresh reconnect. This catches the mirror case where the server is alive but the client's side of the connection has gone stale (common after a phone wake from sleep with NAT table entries already expired).
+
+**Network-aware reconnect**
+The browser's `online` event fires when a network interface becomes available — including transitions between Wi-Fi and cellular. CloudCode listens for this event and immediately cancels any pending backoff retry and opens a new WebSocket. On a typical Wi-Fi↔cellular switch, the terminal is back live in under two seconds.
+
+**Improved wake-from-sleep recovery**
+When the browser tab becomes visible again, CloudCode checks not only for closed sockets but also for sockets stuck in the `CONNECTING` state — a common artifact of waking a phone that had an in-flight connection attempt. Stuck sockets are terminated and replaced immediately rather than waiting for the connection attempt to time out.
+
+---
+
+### 4. Secure Remote Access
 CloudCode is designed to be used over [Tailscale](https://tailscale.com).
 *   **Private Networking**: Your workstation gets a private IP that is only accessible to your devices.
 *   **Identity Validation**: When integrated with Tailscale, CloudCode can verify exactly *who* is accessing the server before they even see a login page.
 *   **Zero-Trust**: No ports need to be opened to the public internet.
 
-### 4. Safety & Auditing
+### 5. Safety & Auditing
 Because agents are powerful, CloudCode prioritizes transparency:
 *   **Path Sandboxing**: Agents are restricted to specific "Repository Roots" to prevent accidental directory traversal.
 *   **Live Audit Logs**: Every session creation, stop command, and profile change is logged with a timestamp and user ID.
@@ -35,3 +55,4 @@ Because agents are powerful, CloudCode prioritizes transparency:
 1.  **Workstation**: Runs the CloudCode backend, SQLite DB, and tmux.
 2.  **Tailscale**: Securely tunnels your phone to your workstation.
 3.  **Phone**: Accesses the CloudCode PWA to launch, monitor, and interact with agents via a live PTY stream backed by tmux sessions.
+4.  **Resilience layer**: Server heartbeat + client watchdog + network-event listener ensure the WebSocket reconnects within seconds of any network disruption — phone sleep, Wi-Fi↔cellular switch, or brief signal loss.

docs/remote-control.md

@@ -51,3 +51,16 @@ CloudCode uses a "Zero-Password" pairing system:
 2. It embeds this token into a QR code.
 3. When you scan the QR code, the remote device is instantly authenticated and granted a 30-day session cookie.
 4. No need to type passwords or manage SSH keys on your mobile device.
+
+## Connection Resilience on Mobile
+
+Pairing gets you connected — but mobile networks are inherently unstable. CloudCode is designed to stay live through the disruptions that are normal on a phone:
+
+| Scenario | What happens |
+|---|---|
+| Phone screen locks / sleeps | Server detects the silent socket within 15 s via heartbeat; client detects it within 35 s via ping watchdog. Both sides clean up and the next wake triggers an instant reconnect. |
+| Wi-Fi → cellular (or back) | Browser fires the `online` event the moment a new interface is ready. CloudCode immediately opens a fresh WebSocket — no waiting for the backoff queue. |
+| Brief signal loss | Existing exponential backoff (up to 10 retries, capped at 30 s) handles transient drops. |
+| Page becomes visible after background | Visibility handler checks for closed *and* stuck-CONNECTING sockets, terminates them, and reconnects before you can tap anything. |
+
+The agent itself is never affected by any of these events — it continues running in its `tmux` session regardless. The resilience work is entirely about getting your phone's view back to the live session as fast as possible.

frontend/src/components/Terminal.tsx

@@ -63,7 +63,7 @@ export function Terminal({ sessionId, sessionTitle, agentName }: TerminalProps)
   const fitAddonRef = useRef<FitAddon | null>(null)
   const searchAddonRef = useRef<SearchAddon | null>(null)
   const [terminalInstance, setTerminalInstance] = useState<XTerm | null>(null)
-  const { isConnected, bootState, sendInput, resize } = useTerminal({ sessionId, terminal: terminalInstance })
+  const { isConnected, bootState, sessionEnded, sendInput, resize } = useTerminal({ sessionId, terminal: terminalInstance })
 
   const [ctrlMode, setCtrlMode] = useState(false)
   const [showSearch, setShowSearch] = useState(false)
@@ -693,8 +693,8 @@ export function Terminal({ sessionId, sessionTitle, agentName }: TerminalProps)
                   title="Scroll to bottom"
                 >END</button>
               </div>
-              <span className={`w-1.5 h-1.5 rounded-full ${isConnected ? 'bg-emerald-500 animate-pulse' : 'bg-amber-500'}`} />
-              <span className="text-[9px] font-black text-zinc-600 uppercase tracking-widest">{isConnected ? 'Live' : 'Syncing'}</span>
+              <span className={`w-1.5 h-1.5 rounded-full ${sessionEnded ? 'bg-zinc-500' : isConnected ? 'bg-emerald-500 animate-pulse' : 'bg-amber-500'}`} />
+              <span className="text-[9px] font-black text-zinc-600 uppercase tracking-widest">{sessionEnded ? 'Ended' : isConnected ? 'Live' : 'Syncing'}</span>
             </div>
           </div>
 
@@ -728,7 +728,7 @@ export function Terminal({ sessionId, sessionTitle, agentName }: TerminalProps)
                     <div className="h-1.5 w-2/5 animate-[pulse_1.6s_ease-in-out_infinite] rounded-full bg-gradient-to-r from-cyan-300 via-white to-cyan-300" />
                   </div>
                   <div className="mt-4 flex items-center justify-between text-[11px] text-zinc-500">
-                    <span>{isConnected ? 'Stream attached' : 'Connecting...'}</span>
+                    <span>{sessionEnded ? 'Session ended' : isConnected ? 'Stream attached' : 'Connecting...'}</span>
                     <span>Session {sessionId.slice(0, 8)}</span>
                   </div>
                 </div>