feat: support voice message STT (Speech-to-Text) for Discord

[chaodu-agent]

Description

Add Speech-to-Text support so that Discord voice messages (audio attachments) are automatically transcribed and forwarded to downstream ACP agents as text input.

This is the next step after image attachment support (PR #158, #210) tracked in #161.

Use Cases

1. Mobile users sending voice messages — Discord mobile users often prefer tapping the voice message button over typing, especially for longer questions or bug descriptions. Without STT, these messages are silently dropped.
2. Accessibility — Users with motor disabilities, RSI, or situational impairments (driving, cooking) rely on voice input. STT makes the agent accessible to a wider audience.
3. Multilingual teams — Voice messages capture tone and intent that may be lost in typed text, especially for non-native English speakers. Whisper supports 50+ languages out of the box.
4. Quick context dumps — A 30-second voice note is faster than typing 200 words. With STT, the agent receives it as actionable text.
5. Workflow continuity — In a team Discord server, a mix of text and voice messages is natural. Without STT, voice messages create "blind spots" where the agent misses context in threads.

Prior Art Investigation

We investigated how OpenClaw and Hermes Agent implement STT:

OpenClaw (openclaw/openclaw)

• Core STT lives in src/media-understanding/ — a framework-level module, not plugin-specific.
• audio-preflight.ts — transcribes audio attachments before mention checking, so voice notes work in group chats with requireMention: true.
• audio-transcription-runner.ts — unified runAudioTranscription() entry point using a capability-based pipeline (runCapability("audio")).
• openai-compatible-audio.ts — generic OpenAI-compatible /audio/transcriptions endpoint via FormData multipart POST.
• Built-in providers (by priority): OpenAI (gpt-4o-transcribe), Groq (whisper-large-v3-turbo), Deepgram (nova-3), Google (gemini-3-flash-preview), Mistral (voxtral-mini-latest).
• Includes a hallucination guard (transcript-policy.ts) to filter known Whisper subtitle-credit artifacts.

Hermes Agent (NousResearch/hermes-agent)

• Core STT in tools/transcription_tools.py — standalone Python module.
• Three providers with auto-fallback: local (faster-whisper, free, ~150MB model), Groq (free tier), OpenAI (paid).
• Supported formats: mp3, mp4, mpeg, mpga, m4a, wav, webm, ogg, aac, flac.
• Discord voice channel integration via VoiceReceiver class: RTP → NaCl decrypt → DAVE E2EE decrypt → Opus decode → PCM → WAV (ffmpeg 16kHz mono) → transcribe_audio().
• Includes is_whisper_hallucination() guard for known Whisper artifacts.

Key Differences

Aspect	OpenClaw	Hermes Agent
Language	TypeScript	Python
Local STT	No (API only)	Yes (faster-whisper)
Providers	5 (OpenAI, Groq, Deepgram, Google, Mistral)	4 (local, Groq, OpenAI, Mistral)
Voice Channel capture	No	Yes (full RTP + realtime STT)
Preflight transcription	Yes (before mention check)	No
Hallucination guard	Yes	Yes

Proposed Design for openab

                         openab — Proposed STT Design
 ═══════════════════════════════════════════════════════════════════

  Discord User
       │
       │  sends voice message (.ogg opus)
       ▼
 ┌───────────┐
 │  Discord   │
 │  Gateway   │
 └─────┬─────┘
       │  msg.attachments[0].content_type = "audio/ogg"
       ▼
 ┌───────────────────────────────────────────────┐
 │              discord.rs                        │
 │                                                │
 │  ┌──────────────┐     ┌─────────────────────┐ │
 │  │ text content  │     │ audio/* attachment?  │ │
 │  │ (existing)    │     │                     │ │
 │  └──────┬───────┘     └────────┬────────────┘ │
 │         │                      │               │
 │         │               stt.enabled?           │
 │         │              ┌───┴───┐               │
 │         │             yes      no              │
 │         │              │       │               │
 │         │              ▼       ▼               │
 │         │         download   skip + warn       │
 │         │          .ogg                        │
 │         │              │                       │
 └─────────┼──────────────┼───────────────────────┘
           │              │
           │              ▼
           │   ┌─────────────────────┐
           │   │      stt.rs         │  (new module, ~60 lines)
           │   │                     │
           │   │  POST /audio/       │
           │   │   transcriptions    │──────► Groq API (free tier)
           │   │                     │        whisper-large-v3-turbo
           │   │  FormData:          │
           │   │   file: audio.ogg   │◄────── { "text": "..." }
           │   │   model: whisper-*  │
           │   └────────┬────────────┘
           │            │
           │            │ transcript string
           │            ▼
           │   "[Voice transcript]: hello, I have a bug..."
           │            │
           ▼            ▼
 ┌──────────────────────────────────────┐
 │         content_blocks: Vec          │
 │                                      │
 │  [0] ContentBlock::Text {            │
 │        "<sender_context>..."         │
 │        "[Voice transcript]: ..."     │  ◄── injected here
 │        + original text (if any)      │
 │      }                               │
 │  [1] ContentBlock::Image { ... }     │  ◄── if image attached
 │                                      │
 └──────────────────┬───────────────────┘
                    │
                    ▼
 ┌──────────────────────────────────────┐
 │          ACP Connection              │
 │     session_prompt(content_blocks)   │
 │                                      │
 │  JSON-RPC → "prompt": [             │
 │    {"type":"text", "text":"..."},    │
 │    {"type":"image", ...}             │
 │  ]                                   │
 └──────────────────┬───────────────────┘
                    │
                    ▼
 ┌──────────────────────────────────────┐
 │        Downstream ACP Agent          │
 │     (Kiro CLI / Claude Code / etc)   │
 │                                      │
 │  Receives voice as plain text ✅     │
 └──────────────────────────────────────┘

Configuration

stt:                                # optional — omit to disable
  enabled: true
  api_key: ${GROQ_API_KEY}
  model: whisper-large-v3-turbo
  # base_url: https://api.groq.com/openai/v1   # default for groq

Deployment Options

Since openab uses a generic OpenAI-compatible /audio/transcriptions endpoint, the same code supports cloud APIs, local whisper servers, and self-hosted solutions — all via the base_url config field.

 ┌─────────────────────────────────────────────────────────────────┐
 │                    openab  (stt.rs)                              │
 │              POST {base_url}/audio/transcriptions                │
 └────────┬──────────────┬──────────────────┬──────────────────────┘
          │              │                  │
          ▼              ▼                  ▼
 ┌──────────────┐ ┌─────────────┐ ┌─────────────────────────┐
 │  Cloud API   │ │ Local Server│ │  LAN / Sidecar Server   │
 └──────────────┘ └─────────────┘ └─────────────────────────┘

Option	base_url	Cost	Latency	Privacy	Setup
Groq Cloud (recommended)	https://api.groq.com/openai/v1	Free tier (rate limited)	~1-2s	Audio sent to Groq	Just set api_key
OpenAI Cloud	https://api.openai.com/v1	~$0.006/min	~1-2s	Audio sent to OpenAI	Set api_key
Local whisper server (Mac Mini, home lab)	http://localhost:8080/v1	Free	~2-5s (CPU) / ~1s (Apple Silicon)	Audio stays local ✅	Run a local server (see below)
LAN / sidecar server	http://192.168.x.x:8080/v1	Free	~1-3s	Audio stays in network ✅	Run server on another machine
Ollama	❌ Not supported	—	—	—	Ollama does not expose /audio/transcriptions

Local Whisper Server Options (for Mac Mini / home lab users)

Users who already run or want to run a local whisper server can point openab at it directly:

Server	Install	Apple Silicon	GPU	OpenAI-compatible
faster-whisper-server	pip install faster-whisper-server	✅ CoreML	✅ CUDA	✅
whisper.cpp server	brew install whisper-cpp	✅ Metal	✅ CUDA	✅
LocalAI	Docker or binary	✅	✅ CUDA	✅

Example config for a local whisper server on the same Mac Mini:

stt:
  enabled: true
  base_url: http://localhost:8080/v1
  api_key: "not-needed"
  model: large-v3-turbo

Files Changed

File	Change	Description
src/stt.rs	NEW	~60 lines — HTTP POST to /audio/transcriptions
src/discord.rs	MOD	Detect audio attachment → call STT → inject transcript
src/config.rs	MOD	Add SttConfig struct
src/main.rs	MOD	Wire SttConfig

Implementation Steps

1. Discord handler (src/discord.rs): Detect audio/* attachments (Discord voice messages are .ogg Opus), download the file.
2. STT module (src/stt.rs): Call an OpenAI-compatible /audio/transcriptions endpoint (Groq free tier with whisper-large-v3-turbo is the most practical default).
3. ACP prompt injection: Prepend the transcript to the ACP prompt text content, e.g. [Voice message transcript]: <text>.
4. Configuration: Add optional stt section to config. No config = feature disabled, zero impact.
5. Hallucination guard (optional): Filter known Whisper artifacts like subtitle credits.

References

• OpenClaw core STT: src/media-understanding/audio-transcription-runner.ts, audio-preflight.ts, openai-compatible-audio.ts
• Hermes Agent STT: tools/transcription_tools.py, gateway/platforms/discord.py
• Related: feat: support multimodal input (images, files, voice) from Discord attachments #161 (multimodal input tracking issue)
• Image support already merged: feat: support Discord image attachments via ACP ImageContent blocks #158, feat: resize and compress images before base64 encoding #210

[chaodu-agent]

Architecture Diagrams

OpenClaw STT Pipeline

Discord/Telegram/WhatsApp
         │
         ▼
┌─────────────────────┐
│   audio-preflight   │  ← transcribe BEFORE mention check
│  (attachments.ts)   │
└────────┬────────────┘
         │ audio/* attachment found
         ▼
┌──────────────────────────────┐
│  audio-transcription-runner  │
│   runCapability("audio")     │
└────────┬─────────────────────┘
         │ picks provider by priority
         ▼
┌──────────────────────────────────────────────┐
│        Provider Registry (auto-select)       │
│                                              │
│  ┌──────────┐  priority                      │
│  │  OpenAI  │  10  gpt-4o-transcribe         │
│  ├──────────┤                                │
│  │   Groq   │  20  whisper-large-v3-turbo    │
│  ├──────────┤                                │
│  │ Deepgram │  30  nova-3                    │
│  ├──────────┤                                │
│  │  Google  │  40  gemini-3-flash-preview    │
│  ├──────────┤                                │
│  │ Mistral  │  50  voxtral-mini-latest       │
│  └──────────┘                                │
└────────┬─────────────────────────────────────┘
         │ all use OpenAI-compatible API
         ▼
┌──────────────────────────────┐
│  openai-compatible-audio.ts  │
│  POST /audio/transcriptions  │
│  (FormData: file + model)    │
└────────┬─────────────────────┘
         │
         ▼
┌──────────────────────────┐
│   transcript-policy.ts   │  ← hallucination guard
│  filter Whisper artifacts │
└────────┬─────────────────┘
         │
         ▼
    { transcript }
    injected into agent prompt

Hermes Agent STT Pipeline

                    ┌─────────────────────────┐
                    │    Two Input Paths       │
                    └─────┬───────────┬────────┘
                          │           │
            ┌─────────────┘           └──────────────┐
            ▼                                        ▼
┌───────────────────────┐              ┌──────────────────────────┐
│  Voice Message (file) │              │  Voice Channel (realtime)│
│  Discord/Telegram/etc │              │  Discord VC only         │
│  .ogg/.mp3/.wav       │              └────────┬─────────────────┘
└────────┬──────────────┘                       │
         │                                      ▼
         │                          ┌───────────────────────┐
         │                          │    VoiceReceiver      │
         │                          │                       │
         │                          │  RTP packet           │
         │                          │    ▼                  │
         │                          │  NaCl decrypt         │
         │                          │    ▼                  │
         │                          │  DAVE E2EE decrypt    │
         │                          │    ▼                  │
         │                          │  Opus decode → PCM    │
         │                          │    ▼                  │
         │                          │  silence detect (1.5s)│
         │                          │    ▼                  │
         │                          │  PCM → WAV (ffmpeg)   │
         │                          │  16kHz mono           │
         │                          └────────┬──────────────┘
         │                                   │
         └──────────┬────────────────────────┘
                    │  .wav / .ogg / .mp3
                    ▼
         ┌─────────────────────────────┐
         │    transcribe_audio()       │
         │  transcription_tools.py     │
         └────────┬────────────────────┘
                  │ auto-fallback
                  ▼
         ┌────────────────────────────────────┐
         │     Provider (auto-detect)         │
         │                                    │
         │  1. local   → faster-whisper (free)│
         │               ~150MB model         │
         │                                    │
         │  2. groq    → whisper-large-v3     │
         │               (free tier)          │
         │                                    │
         │  3. openai  → whisper-1 (paid)     │
         │                                    │
         │  4. mistral → voxtral-mini-latest  │
         └────────┬───────────────────────────┘
                  │
                  ▼
         ┌────────────────────────────┐
         │ is_whisper_hallucination() │ ← filter artifacts
         └────────┬───────────────────┘
                  │
                  ▼
            { transcript }
            → callback / agent prompt

Key difference: OpenClaw is API-only with a clean pipeline and preflight design; Hermes adds local STT and realtime voice channel capture — more capable but more complex.

[chaodu-agent]

Revised Analysis: Option A vs Option B

After reviewing openab's actual codebase, the original proposal needs refinement.

The Question

openab is a thin bridge — it downloads Discord attachments and forwards them to ACP agents. Image support (PR #158) follows this pattern:

Discord image → download → resize/compress → ContentBlock::Image → ACP agent

Should audio follow the same pattern?

Option A: Forward audio as ContentBlock::Audio (like images)

Discord voice msg          ACP Agent
     │                        │
     ▼                        │
  download .ogg               │
     │                        │
     ▼                        │
  base64 encode               │
     │                        │
     ▼                        │
  ContentBlock::Audio ───────►│ agent does STT
     │                        │
     ▼                        ▼
  ❌ BLOCKED — downstream agents (Kiro CLI / Claude)
     do NOT support audio content blocks

Verdict: Not viable. The ACP protocol only defines text and image content blocks. Claude (which powers Kiro) does not accept audio input natively.

Option B: openab performs STT, forwards transcript as text ✅

Discord voice msg
     │
     ▼
  download .ogg
     │
     ▼
  POST /audio/transcriptions  ──► Groq API (free tier)
     │                             whisper-large-v3-turbo
     ▼
  receive transcript
     │
     ▼
  ContentBlock::Text { "[Voice transcript]: ..." }
     │
     ▼
  ACP agent receives plain text ✅

Verdict: This is the only viable path today.

Why Option B is consistent with openab's architecture

openab already does content transformation — image attachments go through resize_and_compress() (resize to 1200px max, JPEG quality 75). STT is the same concept: format adaptation — converting a format the agent cannot understand (audio) into one it can (text).

Image pipeline:    raw PNG/WebP  ──► resize + JPEG compress ──► ContentBlock::Image
Audio pipeline:    raw OGG/Opus  ──► STT transcription       ──► ContentBlock::Text
                   ▲                 ▲
                   └─────────────────┘
                   Both are "format adaptation"
                   to fit downstream agent capabilities

Updated Implementation Plan

1. Discord handler (discord.rs): Detect audio/* attachments, download the .ogg file
2. STT module (new src/stt.rs): Call OpenAI-compatible /audio/transcriptions endpoint
   • Default provider: Groq (free tier, whisper-large-v3-turbo)
   • Fallback: OpenAI (whisper-1), or any OpenAI-compatible endpoint
3. Config (config.rs): Add optional stt section:

   stt:
     enabled: true
     provider: groq          # groq | openai | custom
     api_key: ${GROQ_API_KEY}
     model: whisper-large-v3-turbo
     # base_url: https://custom-endpoint/v1  # optional

4. Prompt injection: Wrap transcript in ContentBlock::Text with a clear prefix so the agent knows it came from voice
5. Graceful degradation: If STT is not configured, log a warning and skip audio attachments (don't crash)