Twitch Toolkit

Turn your streams into content. Automate the boring stuff.

Scorer · Bot · Packages · Setup

---

The Problem

You stream for hours. Great moments happen. Chat goes wild. Then... nothing. Those highlights are buried in a 4-hour VOD that nobody will watch.

Meanwhile, you're supposed to:

• Watch your own stream to find clips
• Create content for TikTok, Shorts, Reels
• Keep chat entertained when you're focused on the game
• Add chapters and SEO to your YouTube uploads
• Remember inside jokes and community lore

That's a second job.

The Solution

A toolkit that does the tedious work:

Tool	What it does
Scorer	Watches your VOD, finds highlights, detects songs, uploads chapters
Bot	Your AI sidekick in chat - with YOUR personality
Social	Formats clips for every platform (coming soon)

All powered by LLMs that actually understand context, not just audio peaks.

---

Stack

Layer	Tech
Runtime	Bun
Language	TypeScript
LLMs	DeepSeek, Ollama, Groq, OpenAI, OpenRouter
Transcription	Whisper.cpp (whisper-cli)
Music Detection	ACRCloud
Audio Separation	Demucs
Video	yt-dlp + ffmpeg
APIs	Twitch Helix, YouTube Data v3, Brave/Tavily Search

---

🎯 Scorer

Find highlights without watching the whole stream.

The scorer transcribes your VOD, detects songs, finds talking segments, and rates each one with an LLM. Then uploads chapters and SEO metadata to YouTube automatically.

Pipeline Overview

VOD → Audio → Song Detection → Speech Segments → Transcription → LLM Scoring → Clips + Chapters

Commands

# 1. Detect songs in a VOD (uses ACRCloud)
bun run detect-songs-acrcloud VIDEO_ID

# 2. Transcribe speech segments (skips music)
bun run transcribe VIDEO_ID

# 3. Score highlights
bun run score VIDEO_ID

# 4. Extract top clips
bun run extract-clips VIDEO_ID --top=5

# 5. Upload chapters + SEO to YouTube
bun run upload-chapters --video=VIDEO_ID --songs=data/song-detection/VIDEO_ID.json

Song Detection (ACRCloud)

Automatically identifies songs playing in your stream:

bun run detect-songs-acrcloud VIDEO_ID --skip-minutes=10 --volume-threshold=-20

• --skip-minutes: Skip intro music (default: 10)
• --volume-threshold: Skip quiet background music (default: -20 dB)
• Outputs: song list with timestamps, gaps for transcription

YouTube SEO Upload

Upload chapters and optimized metadata to your YouTube videos:

# Preview without uploading
bun run upload-chapters --video=VIDEO_ID --songs=data/song-detection/VIDEO_ID.json --dry-run

# Actually upload
bun run upload-chapters --video=VIDEO_ID --songs=data/song-detection/VIDEO_ID.json

Generates:

• Title: SEO optimized, 100 chars max, popular artists first
• Description: Intro, clickable chapters, instruments, artists, keywords, social links
• Tags: Artist + song combinations, 480 chars max

How Scoring Works

1. Downloads audio from YouTube (or uses local file)
2. Detects songs with ACRCloud (skips background music)
3. Identifies speech gaps for transcription
4. Transcribes with Whisper.cpp
5. Splits into 2-minute chunks with context overlap
6. LLM rates each chunk (1-10) with reasoning
7. Merges adjacent high-scoring segments
8. Extracts clips with ffmpeg

---

🤖 Bot

A chat bot that sounds like YOU, not a robot.

Most bots feel generic. This one learns your channel's vibe:

• Your speech patterns and catchphrases
• Inside jokes the community knows
• How to treat regulars vs new viewers
• What topics to avoid

Personality System

Two files define your bot:

config/personality.json - Who is the bot?

{
  "name": "StreamBot",
  "backstory": "The streamer's witty AI sidekick",
  "tone": "friendly with a bit of sass",
  "catchphrases": ["Let's go!", "Big plays only"],
  "specialUsers": {
    "top_supporter": "VIP treatment, always hype them up"
  }
}

context/CHANNEL.md - What's this channel about?

# The Streamer
Variety streamer focusing on indie games

# The Vibe
Chill gaming with active chat interaction

# Inside Jokes
- "The run" = that one failed speedrun attempt
- Fridge check = when streamer forgets to eat

Commands

Command	What it does
@Bot <question>	Ask anything, get a response in character
@Bot busca <query>	Web search + LLM synthesis
!mismensajes	Check your remaining message quota
!clip	Create a clip (mods/VIPs only)
!resumen	Generate stream summary (broadcaster only)

Tier System

Rate limits based on subscriber tier - your regulars get more access:

Tier	Messages	Searches
T3 / VIP / Mod	Unlimited	150
T2	60	10
T1	30	0
Non-sub	0	0

---

📦 Packages

Reusable modules for building your own tools.

@toolkit/llm

LLM abstraction with cache and retry built-in.

import { createProvider } from '@toolkit/llm';

const llm = createProvider('deepseek', {
  apiKey: process.env.DEEPSEEK_API_KEY
});

const response = await llm.complete(
  'Summarize this chat log',
  'You are a helpful assistant'
);

Providers: ollama · deepseek · groq · openai · openrouter

@toolkit/api

Twitch, YouTube, and search integrations.

import {
  TwitchApi,
  downloadYouTubeAudio,
  getVideoDetails,
  updateVideo,
  createSearchProvider
} from '@toolkit/api';

// Download VOD audio
await downloadYouTubeAudio('VIDEO_ID', './audio');

// Update YouTube video metadata
await updateVideo(auth, 'VIDEO_ID', {
  title: 'New Title',
  description: 'New description with chapters',
  tags: ['tag1', 'tag2']
});

// Search with multiple providers
const search = createSearchProvider('brave', { apiKey });
const results = await search.search('query');

@toolkit/shared

Common utilities.

import { formatTime, parseDuration, withRetry } from '@toolkit/shared';

formatTime(3661);       // "01:01:01"
parseDuration('PT1H5M'); // "1:05:00"

await withRetry(() => flakeyApiCall(), {
  maxRetries: 3,
  retryableStatuses: [429, 503]
});

---

🚀 Quick Start

Prerequisites

System Dependencies

Tool	Purpose	Required
Bun	JavaScript runtime	✅
FFmpeg	Audio/video processing	✅
yt-dlp	YouTube downloads	✅
whisper-cli	Speech-to-text transcription	✅
Python 3.11+	Required for Demucs	✅

macOS Installation (Homebrew)

# Install Homebrew if not installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install system dependencies
brew install bun ffmpeg yt-dlp python@3.11

# Install whisper.cpp (whisper-cli)
brew install whisper-cpp

# Or build from source for better performance:
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make
cp main /usr/local/bin/whisper-cli

# Download a Whisper model (required)
# Models: tiny, base, small, medium, large
whisper-cli --download-model base

Python Dependencies (Demucs)

Demucs is used for vocal/music separation. Set it up in a virtual environment:

# Create virtual environment in project
python3.11 -m venv data/demucs-venv

# Install Demucs
data/demucs-venv/bin/pip install demucs

# Verify installation
data/demucs-venv/bin/demucs --help

External Services & API Keys

Service	Purpose	Required
ACRCloud	Music recognition	✅ For song detection
DeepSeek / OpenAI / Ollama	LLM for analysis	✅ One provider needed
Twitch Developer	Bot integration	✅ For bot app
Google Cloud Console	YouTube API	✅ For chapter upload
Brave Search	Web search in bot	Optional
Discord Webhook	Notifications	Optional

Install

git clone https://github.com/teseo/twitch-toolkit
cd twitch-toolkit
bun install

Configure

cp .env.example .env

Edit .env:

# Required for scorer
LLM_PROVIDER=deepseek
DEEPSEEK_API_KEY=sk-xxx

# ACRCloud (for song detection)
ACRCLOUD_ACCESS_KEY=xxx
ACRCLOUD_SECRET_KEY=xxx
ACRCLOUD_HOST=identify-eu-west-1.acrcloud.com

# Required for bot
TWITCH_CLIENT_ID=xxx
TWITCH_CLIENT_SECRET=xxx
TWITCH_CHANNEL=your_channel
TWITCH_BOT_USERNAME=your_bot

# Optional
BRAVE_API_KEY=xxx  # For web search
DISCORD_WEBHOOK_URL=xxx  # Clip notifications

Authenticate

# YouTube (for scorer - needed for chapters upload)
cd apps/scorer && bun run setup-auth

# Twitch (for bot)
cd apps/bot && bun run auth

Run

# Full pipeline: detect songs → transcribe → score → extract
cd apps/scorer
bun run detect-songs-acrcloud VIDEO_ID
bun run transcribe VIDEO_ID
bun run score VIDEO_ID
bun run extract-clips VIDEO_ID --top=5
bun run upload-chapters --video=VIDEO_ID --songs=data/song-detection/VIDEO_ID.json

# Or start the bot
cd apps/bot && bun run start

---

🏗️ Project Structure

twitch-toolkit/
├── apps/
│   ├── scorer/           # VOD analysis & clip extraction
│   │   ├── src/
│   │   │   ├── scripts/  # CLI commands
│   │   │   └── services/ # ACRCloud, Whisper, YouTube SEO
│   │   └── tests/        # Unit tests
│   ├── bot/              # Twitch chat bot
│   └── socialmedia/      # Coming soon
├── packages/
│   ├── llm/              # LLM providers
│   ├── api/              # Platform APIs (Twitch, YouTube, Search)
│   └── shared/           # Common utilities
└── .env                  # Your secrets

---

Development

# Type check everything
bun run typecheck

# Run all tests
bun test

# Dev mode (auto-reload)
cd apps/bot && bun run dev

---

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feat/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feat/amazing-feature)
5. Open a Pull Request

License

MIT - Use it, modify it, share it. Attribution required.

---

Built for streamers who'd rather stream than edit.

Report Bug · Request Feature