You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds a new TTS backend (vibevoice) to the ones supported
Notes for Reviewers
Tested with:
name: vibevoice-realtimebackend: vibevoicedescription: | VibeVoice-Realtime is a real-time text-to-speech model that generates natural-sounding speech. This model supports voice cloning through voice preset files (.pt files).parameters:
model: microsoft/VibeVoice-Realtime-0.5B# TTS configurationtts:
# Voice selection - can be:# 1. Voice preset name (e.g., "Frank", "en-Frank_man", "Grace") - looks for .pt files in voices/streaming_model/# 2. Path to a voice preset .pt file (relative to model directory or absolute)# Available English voices: Carter, Davis, Emma, Frank, Grace, Mikevoice: "Frank"# Alternative: use audio_path to specify a voice file directly# audio_path: "voices/streaming_model/en-Frank_man.pt"known_usecases:
- tts# Backend-specific options# These are passed as "key:value" strings to the backendoptions:
# CFG (Classifier-Free Guidance) scale for generation (default: 1.5)# Higher values can improve quality but may slow generation
- "cfg_scale:1.5"# Number of inference steps for the diffusion process (default: 5)# More steps = better quality but slower. Typical range: 3-10
- "inference_steps:5"# Enable sampling (default: false)# When true, uses temperature and top_p for sampling
- "do_sample:false"# Temperature for sampling (only used if do_sample=true, default: 0.9)
- "temperature:0.9"# Top-p (nucleus) sampling (only used if do_sample=true, default: 0.9)
- "top_p:0.9"# Voices directory path# This explicitly sets where to look for voice preset files (.pt files)# Since we're downloading voices to voices/streaming_model/, we set it here## Examples:# - Relative path (relative to models directory): "voices/streaming_model"# - Absolute path: "/custom/path/to/voices/streaming_model"# - Custom relative path: "my_custom_voices/streaming_model"## If not specified, the backend will auto-detect from common locations:# 1. {ModelFile directory}/voices/streaming_model/# 2. {models_dir}/voices/streaming_model/# 3. Backend directory
- "voices_dir:voices/streaming_model"# Download voice preset files# Voice presets are downloaded to: {models_dir}/voices/streaming_model/# The voices_dir option above tells the backend to look in this locationdownload_files:
# English voices
- filename: voices/streaming_model/en-Frank_man.pturi: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/en-Frank_man.pt
- filename: voices/streaming_model/en-Grace_woman.pturi: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/en-Grace_woman.pt
- filename: voices/streaming_model/en-Mike_man.pturi: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/en-Mike_man.pt
- filename: voices/streaming_model/en-Emma_woman.pturi: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/en-Emma_woman.pt
- filename: voices/streaming_model/en-Carter_man.pturi: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/en-Carter_man.pt
- filename: voices/streaming_model/en-Davis_man.pturi: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/en-Davis_man.pt# Uncomment to add more languages:# - filename: voices/streaming_model/fr-Spk0_man.pt# uri: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/fr-Spk0_man.pt# - filename: voices/streaming_model/de-Spk0_man.pt# uri: https://raw.githubusercontent.com/microsoft/VibeVoice/main/demo/voices/streaming_model/de-Spk0_man.pt
::group::gallery/index.yaml
::warning file=gallery/index.yaml,line=30,col=5::30:5 [comments-indentation] comment not indented like content
::warning file=gallery/index.yaml,line=3332,col=6::3332:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3350,col=6::3350:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3368,col=6::3368:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=6986,col=32::6986:32 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=8981,col=59::8981:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=10874,col=70::10874:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13397,col=61::13397:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13462,col=64::13462:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14073,col=59::14073:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14494,col=64::14494:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15613,col=70::15613:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15965,col=59::15965:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16010,col=63::16010:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16027,col=62::16027:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18309,col=59::18309:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18520,col=61::18520:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18540,col=62::18540:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18595,col=28::18595:28 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19470,col=59::19470:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19776,col=62::19776:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19831,col=63::19831:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19953,col=63::19953:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20168,col=73::20168:73 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20504,col=62::20504:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20635,col=61::20635:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20764,col=21::20764:21 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21141,col=65::21141:65 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21367,col=56::21367:56 [comments] too few spaces before comment: expected 2
::endgroup::
::group::gallery/vibevoice.yaml
::error file=gallery/vibevoice.yaml,line=33,col=1::33:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=37,col=1::37:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=41,col=1::41:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=44,col=1::44:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=47,col=1::47:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=51,col=8::51:8 [trailing-spaces] trailing spaces
::endgroup::
::group::gallery/index.yaml
::warning file=gallery/index.yaml,line=3326,col=6::3326:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3344,col=6::3344:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3362,col=6::3362:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=6980,col=32::6980:32 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=8975,col=59::8975:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=10868,col=70::10868:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13391,col=61::13391:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13456,col=64::13456:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14067,col=59::14067:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14488,col=64::14488:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15607,col=70::15607:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15959,col=59::15959:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16004,col=63::16004:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16021,col=62::16021:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18303,col=59::18303:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18514,col=61::18514:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18534,col=62::18534:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18589,col=28::18589:28 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19464,col=59::19464:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19770,col=62::19770:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19825,col=63::19825:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19947,col=63::19947:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20162,col=73::20162:73 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20498,col=62::20498:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20629,col=61::20629:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20758,col=21::20758:21 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21135,col=65::21135:65 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21361,col=56::21361:56 [comments] too few spaces before comment: expected 2
::endgroup::
::group::gallery/vibevoice.yaml
::error file=gallery/vibevoice.yaml,line=33,col=1::33:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=37,col=1::37:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=41,col=1::41:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=44,col=1::44:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=47,col=1::47:1 [trailing-spaces] trailing spaces
::error file=gallery/vibevoice.yaml,line=51,col=8::51:8 [trailing-spaces] trailing spaces
::endgroup::
::group::gallery/index.yaml
::warning file=gallery/index.yaml,line=3326,col=6::3326:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3344,col=6::3344:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3362,col=6::3362:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=6980,col=32::6980:32 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=8975,col=59::8975:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=10868,col=70::10868:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13391,col=61::13391:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13456,col=64::13456:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14067,col=59::14067:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14488,col=64::14488:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15607,col=70::15607:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15959,col=59::15959:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16004,col=63::16004:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16021,col=62::16021:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18303,col=59::18303:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18514,col=61::18514:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18534,col=62::18534:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18589,col=28::18589:28 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19464,col=59::19464:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19770,col=62::19770:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19825,col=63::19825:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19947,col=63::19947:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20162,col=73::20162:73 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20498,col=62::20498:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20629,col=61::20629:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20758,col=21::20758:21 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21135,col=65::21135:65 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21361,col=56::21361:56 [comments] too few spaces before comment: expected 2
::endgroup::
::group::gallery/vibevoice.yaml
::error file=gallery/vibevoice.yaml,line=46,col=8::46:8 [trailing-spaces] trailing spaces
::endgroup::
::group::gallery/index.yaml
::warning file=gallery/index.yaml,line=3326,col=6::3326:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3344,col=6::3344:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=3362,col=6::3362:6 [comments] missing starting space in comment
::warning file=gallery/index.yaml,line=6980,col=32::6980:32 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=8975,col=59::8975:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=10868,col=70::10868:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13391,col=61::13391:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=13456,col=64::13456:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14067,col=59::14067:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=14488,col=64::14488:64 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15607,col=70::15607:70 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=15959,col=59::15959:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16004,col=63::16004:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=16021,col=62::16021:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18303,col=59::18303:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18514,col=61::18514:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18534,col=62::18534:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=18589,col=28::18589:28 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19464,col=59::19464:59 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19770,col=62::19770:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19825,col=63::19825:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=19947,col=63::19947:63 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20162,col=73::20162:73 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20498,col=62::20498:62 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20629,col=61::20629:61 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=20758,col=21::20758:21 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21135,col=65::21135:65 [comments] too few spaces before comment: expected 2
::warning file=gallery/index.yaml,line=21361,col=56::21361:56 [comments] too few spaces before comment: expected 2
::endgroup::
::group::gallery/vibevoice.yaml
::error file=gallery/vibevoice.yaml,line=46,col=8::46:8 [trailing-spaces] trailing spaces
::endgroup::
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new TTS backend (
vibevoice) to the ones supportedNotes for Reviewers
Tested with:
Signed commits