VoiceOps Latency: Blocking Conversational Quality

[GreyforgeLabs]

Summary

Development on VoiceOps has stalled due to end-to-end latency that prevents genuine conversational interaction. The current pipeline takes several seconds from when the user finishes speaking to when a response begins playing back — far outside the threshold required for natural, phone-quality conversation.

Problem

The round-trip latency between human speech and agent response is not conversational. A typical human conversation tolerates roughly 200-300ms of delay. The current pipeline exceeds this by a significant margin, making real-time voice interaction feel robotic and unnatural.

Observed behavior: Multi-second delay between end of user speech and start of agent response.
Expected behavior: Near-instantaneous response onset, consistent with phone-call quality dialogue.

Why This Matters

Solving this unlocks something significant. VoiceOps is, to our knowledge, the first full-duplex voice pipeline built for an open agent platform like OpenClaw. If latency can be brought into a conversational range, this becomes a genuinely novel capability — the ability to speak with autonomous agents the same way you speak with a person on the phone.

Commercial solutions (e.g. ChatGPT Advanced Voice) have solved this, but with proprietary infrastructure, custom hardware, and large engineering teams. The open-source equivalent does not yet exist at this quality level.

Known Bottlenecks (for investigation)

• ASR (Whisper): Transcription latency after voice activity detection ends
• LLM inference: Time-to-first-token from the agent backend
• TTS (kokoro-js): Synthesis time before audio playback begins
• Streaming: Whether each stage is streamed or batched end-to-end

Goal

Identify and reduce latency at each stage of the pipeline so that the perceived response delay feels conversational. Streaming optimizations, model quantization, VAD tuning, and parallel pipeline stages are all worth exploring.

If anyone has experience with low-latency voice pipelines or has solved similar bottlenecks in open-source projects, contributions and suggestions are welcome.

[m13v]

the latency problem is brutal and usually not one bottleneck but death by a thousand cuts across the pipeline. from building a real-time voice agent on macOS, here is where the biggest wins came from:

ASR: the single biggest improvement was switching from batch processing (record full utterance, then transcribe) to chunked streaming. we send 3-second audio chunks with 0.5s overlap to the STT model while the user is still speaking. by the time they stop talking, 80% of the transcription is already done. the boundary deduplication is annoying but worth it.

VAD: using Silero VAD to detect speech end instead of a fixed silence timeout saved us ~500ms consistently. energy-based thresholds fail in any room with AC or keyboard noise.

TTS: the real trick is streaming the LLM response token-by-token into the TTS engine. don't wait for the full response. most TTS engines can start speaking from a partial sentence. this hides the LLM latency almost entirely for the first few seconds of the response.

[m13v]

our transcription service that implements the chunked streaming approach with VAD gating and overlap deduplication: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

covers the 16kHz audio preprocessing, WhisperKit model loading, and the streaming chunk pipeline that gets most of the transcription done before the user stops speaking.

[GreyforgeLabs]

This is useful information to digest! Thank you. I'll conduct research later to see if we can adapt these strategies to address the issue.

Feel free to fork it and experiment with it yourself. 🙂

[m13v]

nice, let me know what you find. the Silero VAD tuning is where most of the latency wins hide, specifically the speech probability threshold and min silence duration. happy to compare notes if you run into issues with the chunked streaming approach.

[GreyforgeLabs]

Quick note to replace my earlier wording: thank you again for the great tips. We took a real pass at this in the code: we’ve started streaming the agent’s response so it can begin speaking sooner, and we added a guard to prevent very long voice clips from stretching the delay. We also made those streaming settings configurable so we can tune the behavior as we keep iterating. Thanks for the guidance — we did reconcile your ideas and apply them ourselves.

[GreyforgeLabs]

Audited the current voiceops implementation at HEAD against this issue, and there is still one clear latency bottleneck in the runtime:

• src/gateway-client.mjs only resolves sendVoiceTurn() on state === 'final'.
• src/pipeline.mjs waits for that full final agent text before starting TTS.

That means the pipeline still serializes ASR -> full LLM completion -> TTS -> playback, so even if the gateway emits deltas, VoiceOps does not start speaking from them yet.

There is also no runtime guard for very long utterances. vad.silenceDurationMs is configurable, but once a clip is captured the whole buffer still goes through ASR in one shot, which can stretch perceived latency before the agent stage even begins.

The bounded next patch I would recommend is:

1. Surface gateway delta/partial text into the pipeline and start first-chunk TTS on a sentence boundary instead of waiting for final.
2. Add a vad.maxUtteranceDurationMs guard so very long clips do not dominate ASR latency.
3. Log stage timings explicitly: VAD end, ASR done, first gateway delta, final gateway response, TTS done, playback start.

That would turn this from a broad “latency feels bad” report into concrete timings plus one real architecture change that removes the current wait-for-final bottleneck.