Add browser-side Speech-to-Text with Web Worker architecture#3
Add browser-side Speech-to-Text with Web Worker architecture#3
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- Create whisper-worker.js for isolated model loading and inference - Add useSpeechToText hook for audio capture and processing - Create SpeechToTextButton component with multiple UI states - Integrate STT button into PromptInputBox - Update vite.config.ts for Web Worker support - Add @xenova/transformers dependency for Whisper model Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
- Fix progress value conversion (handle both 0-1 and 0-100 formats) - Remove hardcoded sample rate constraint for better device compatibility - Fix race condition by setting onstop handler before starting MediaRecorder Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
可以参考以下样式来优化。按键放右边, 时间和声音条放在左边。You are given a task to integrate an existing React component in the codebase The codebase should support:
If it doesn't, provide instructions on how to setup project via shadcn CLI, install Tailwind or Typescript. Determine the default path for components and styles. ai-voice-input.tsx
"use client";
import { Mic } from "lucide-react";
import { useState, useEffect } from "react";
import { cn } from "@/lib/utils";
interface AIVoiceInputProps {
onStart?: () => void;
onStop?: (duration: number) => void;
visualizerBars?: number;
demoMode?: boolean;
demoInterval?: number;
className?: string;
}
export function AIVoiceInput({
onStart,
onStop,
visualizerBars = 48,
demoMode = false,
demoInterval = 3000,
className
}: AIVoiceInputProps) {
const [submitted, setSubmitted] = useState(false);
const [time, setTime] = useState(0);
const [isClient, setIsClient] = useState(false);
const [isDemo, setIsDemo] = useState(demoMode);
useEffect(() => {
setIsClient(true);
}, []);
useEffect(() => {
let intervalId: NodeJS.Timeout;
if (submitted) {
onStart?.();
intervalId = setInterval(() => {
setTime((t) => t + 1);
}, 1000);
} else {
onStop?.(time);
setTime(0);
}
return () => clearInterval(intervalId);
}, [submitted, time, onStart, onStop]);
useEffect(() => {
if (!isDemo) return;
let timeoutId: NodeJS.Timeout;
const runAnimation = () => {
setSubmitted(true);
timeoutId = setTimeout(() => {
setSubmitted(false);
timeoutId = setTimeout(runAnimation, 1000);
}, demoInterval);
};
const initialTimeout = setTimeout(runAnimation, 100);
return () => {
clearTimeout(timeoutId);
clearTimeout(initialTimeout);
};
}, [isDemo, demoInterval]);
const formatTime = (seconds: number) => {
const mins = Math.floor(seconds / 60);
const secs = seconds % 60;
return `${mins.toString().padStart(2, "0")}:${secs.toString().padStart(2, "0")}`;
};
const handleClick = () => {
if (isDemo) {
setIsDemo(false);
setSubmitted(false);
} else {
setSubmitted((prev) => !prev);
}
};
return (
<div className={cn("w-full py-4", className)}>
<div className="relative max-w-xl w-full mx-auto flex items-center flex-col gap-2">
<button
className={cn(
"group w-16 h-16 rounded-xl flex items-center justify-center transition-colors",
submitted
? "bg-none"
: "bg-none hover:bg-black/10 dark:hover:bg-white/10"
)}
type="button"
onClick={handleClick}
>
{submitted ? (
<div
className="w-6 h-6 rounded-sm animate-spin bg-black dark:bg-white cursor-pointer pointer-events-auto"
style={{ animationDuration: "3s" }}
/>
) : (
<Mic className="w-6 h-6 text-black/70 dark:text-white/70" />
)}
</button>
<span
className={cn(
"font-mono text-sm transition-opacity duration-300",
submitted
? "text-black/70 dark:text-white/70"
: "text-black/30 dark:text-white/30"
)}
>
{formatTime(time)}
</span>
<div className="h-4 w-64 flex items-center justify-center gap-0.5">
{[...Array(visualizerBars)].map((_, i) => (
<div
key={i}
className={cn(
"w-0.5 rounded-full transition-all duration-300",
submitted
? "bg-black/50 dark:bg-white/50 animate-pulse"
: "bg-black/10 dark:bg-white/10 h-1"
)}
style={
submitted && isClient
? {
height: `${20 + Math.random() * 80}%`,
animationDelay: `${i * 0.05}s`,
}
: undefined
}
/>
))}
</div>
<p className="h-4 text-xs text-black/70 dark:text-white/70">
{submitted ? "Listening..." : "Click to speak"}
</p>
</div>
</div>
);
}
demo.tsx
import { AIVoiceInput } from "@/components/ui/ai-voice-input";
import { useState } from "react";
export function AIVoiceInputDemo() {
const [recordings, setRecordings] = useState<{ duration: number; timestamp: Date }[]>([]);
const handleStop = (duration: number) => {
setRecordings(prev => [...prev.slice(-4), { duration, timestamp: new Date() }]);
};
return (
<div className="space-y-8">
<div className="space-y-4">
<AIVoiceInput
onStart={() => console.log('Recording started')}
onStop={handleStop}
/>
</div>
</div>
);
}Install NPM dependencies: lucide-reactImplementation Guidelines
Steps to integrate
|
- Add inline voice recording bar with visualizer bars on left - Show recording timer in center - Place stop/mic button on right side - Add animated visualizer bars during recording - Extract magic numbers to named constants - Remove unused SpeechToTextButton and AIVoiceInput components Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
|
按停止录音的时候, 有的时候不转录 |
… display - Fix race condition in stopRecording by clearing mediaRecorderRef in onstop handler - Add validation for empty audio chunks and recordings that are too short - Add better error handling for audio decode failures and worker readiness - Update UI to properly show error messages and reset state on errors - Simplify worker transcription result logic Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
Fixed in commit f06f24b. The issues were:
The changes include better error handling for audio decode failures, worker readiness checks, and user-friendly error messages. |
|
@copilot |
The issue was that after transcription, the status changes to 'ready', which triggered the auto-start effect to restart recording before the handleSpeechTranscript callback could set voiceInputActive to false. Added justFinishedProcessingRef to track when processing just finished, preventing the auto-start effect from re-triggering recording after transcription or error states. Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
Fixed in commit 98854f2. The issue was that the auto-start recording effect was triggering after transcription completed because the voiceStatus changed to 'ready' before the callback could set voiceInputActive to false. Added a ref to track when processing just finished and prevent auto-restart in that case. |
|
@copilot |
The issue was that justFinishedProcessingRef was being cleared unconditionally at the start of the effect, even when the auto-start conditions weren't met. This caused the flag to be cleared before it could prevent auto-restart on subsequent recordings. Fix: - Only clear the flag when auto-start conditions are actually met - Clear the flag when user manually clicks to start new recording Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
Fixed in commit 24c3a5b. The issue was that Changes:
|
|
@copilot Same problem! When I press the stop recording button, a "Transcribing..." message appears briefly, but after a few seconds it automatically switches back to recording mode without transcribing any text. |
Previous approach used justFinishedProcessingRef flag which was too complex and prone to race conditions. New approach: - Use waitingForModelRef that is ONLY set when user clicks mic while model not loaded - Auto-start ONLY triggers when this flag is set AND model becomes ready - After transcription, the flag is already false so no restart happens This removes the problematic voiceInputActive dependency for auto-start. Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
Fixed in commit fb9c75d. Completely rewrote the auto-start logic with a simpler approach:
The previous approach with |
|
@copilot When I press the stop recording button, a "Transcribing..." message appears briefly, but after a few seconds it didn't transcribing any text! |
The onTranscript callback was captured in the initWorker useCallback closure. When the component re-rendered, the worker's onmessage handler still had the old callback reference that didn't update the UI. Fix: Use a ref (onTranscriptRef) that is updated via useEffect whenever onTranscript changes. The worker's onmessage handler now uses this ref to always call the latest callback. Co-authored-by: Ryan-Guo123 <169242962+Ryan-Guo123@users.noreply.github.com>
Fixed in commit ca502c5. The issue was a stale callback closure - the Fix: Use a ref ( |
worker.js) for Whisper model loading and inference@xenova/transformersdependency for Whisper modeluseSpeechToText.ts) with audio processing and worker communicationPromptInputBoxcomponent with UI states (initial, loading, recording, processing)Summary
This PR implements browser-side Speech-to-Text using the Whisper model (
Xenova/whisper-tiny.en) running entirely on the client. Key features:Bug Fixes (Latest)
New Voice Input UI Design
The voice recording UI now features:
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.