3Speak Embed Video Upload Service

A modern, scalable video upload and encoding service with TUS resumable uploads, IPFS storage, multi-encoder support, and MongoDB job tracking. Built for the 3Speak decentralized video platform.

Features

TUS Resumable Uploads: Robust video uploads with pause/resume support
Instant Embed URLs: Get playable embed URLs immediately upon upload start
IPFS Storage: Automatic pinning to IPFS (local daemon + supernode fallback)
Multi-Encoder Support: Round-robin load balancing across multiple encoder nodes
Encoder Management: Add, update, enable/disable encoders via admin panel or API
Job Dispatcher: Automatic job queuing and distribution to available encoders
Webhook Callbacks: Secure encoder-to-service communication for status updates
MongoDB Integration: Tracks videos, jobs, and API keys
Upload Tokens: Short-lived, signed tokens for secure client-side uploads without exposing API keys
Encoding Progress: Real-time encoding progress tracking via polling
API Key Management: Secure admin panel for managing application access
RESTful API: Simple endpoints for video metadata and management

Prerequisites

Node.js v20 or higher
MongoDB 6.0+
IPFS daemon (for local pinning) or access to IPFS gateway
npm or yarn
One or more encoder nodes (see 3speak-encoder)

Installation

Clone the repository:

git clone <repository-url>
cd 3speakembed

Install dependencies:

npm install

Create a .env file based on .env.example:

cp .env.example .env

Configure your environment variables in .env (see .env.example for full details):

PORT=3001
MONGODB_URI=mongodb://user:pass@host:27017/threespeak
ENCODERS=[{"name":"encoder1","url":"https://encoder.example.com","apiKey":"key","enabled":true}]
WEBHOOK_API_KEY=your-secure-webhook-key
WEBHOOK_URL=https://embed.3speak.tv/webhook
# Optional: enable upload tokens for secure client-side uploads
UPLOAD_TOKEN_SECRET=your-random-secret-here

See .env.example for complete configuration options.

Usage

Development Mode

npm run dev

Production Mode

Build the project:

npm run build

Start the server:

npm start

Authentication

The service supports two authentication methods for uploads. Both are fully supported — choose the one that fits your architecture.

Option 1: Direct API Key (Server-to-Server / Mobile Apps)

Pass the API key directly in the X-API-Key header. This is the simplest integration and is appropriate when:

Your uploads happen server-side (your backend uploads on behalf of users)
You're building a mobile app where the key is bundled in the binary (not exposed in browser DevTools)

Option 2: Upload Tokens (Web Apps / Client-Side Uploads)

For web applications where JavaScript runs in the browser, exposing the API key in network requests is a security risk. Upload tokens solve this:

Your backend (which holds the API key) requests a short-lived token from the embed service
Your frontend uses that token to upload directly — the API key never reaches the browser

Tokens are HMAC-SHA256 signed, single-use, and time-limited (default 10 min, max 30 min).

Requesting a token (server-side):

curl -X POST https://embed.3speak.tv/uploads/token \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "owner": "username",
    "frontend_app": "your-app-name",
    "short": false,
    "allowed_origins": ["https://your-app.com"],
    "max_file_size": 1073741824,
    "ttl": 600
  }'

Response:

{
  "token": "eyJ...",
  "upload_url": "https://embed.3speak.tv/uploads",
  "expires_at": "2026-03-17T12:10:00.000Z"
}

Token parameters:

Field	Type	Description
`owner`	string	Required. Username of the video owner
`frontend_app`	string	Required. Your application identifier
`short`	boolean	Required. `true` for short-form videos (≤60s, 480p)
`allowed_origins`	string[]	CORS origins allowed to use this token (recommended for web)
`max_file_size`	number	Max upload size in bytes (capped by server config, default 1GB)
`ttl`	number	Token lifetime in seconds (capped by server config, default 600s)

API Endpoints

Health Check

GET /health

Returns the service status.

Get Video Metadata

GET /video/:permlink

Retrieves metadata for a specific video, including encoding progress. This endpoint is public (no auth required) and can be polled to track upload and encoding status. See Tracking Progress below.

Check User Premium Status

GET /users/:username/premium

Returns whether a user has premium status (multi-resolution encoding). Requires X-API-Key header. Returns 404 if the user does not exist.

Response:

{
  "username": "coolmole",
  "premium": false
}

Frontends use this to adjust upload size limits in the UI before requesting an upload token.

TUS Upload Endpoint

POST /uploads

TUS protocol endpoint for video uploads. Requires either X-API-Key header or Authorization: Bearer <upload-token>.

Required Metadata:

owner or username: The username for the video owner (ignored when using upload tokens — the token carries this)
frontend_app: Frontend application identifier (ignored when using upload tokens)
short: String "true" or "false" - whether this is a short-form video (ignored when using upload tokens)
filename: (optional) Original filename

Response Headers:

X-Embed-URL: The embed URL for the video (format: https://play.3speak.tv/embed?v={owner}/{permlink})

Request Upload Token

POST /uploads/token

Generates a short-lived, single-use upload token. Requires X-API-Key header. See Authentication for details.

Upload Examples

Direct API Key (Server-to-Server / Mobile)

import * as tus from 'tus-js-client';

const file = document.getElementById('file-input').files[0];

const upload = new tus.Upload(file, {
  endpoint: 'https://embed.3speak.tv/uploads',
  headers: {
    'X-API-Key': 'your-api-key'
  },
  metadata: {
    filename: file.name,
    owner: 'chessfighter',
    frontend_app: 'my-video-app',
    short: 'false',
    filetype: file.type
  },
  onError: (error) => {
    console.error('Upload failed:', error);
  },
  onProgress: (bytesUploaded, bytesTotal) => {
    const percentage = (bytesUploaded / bytesTotal * 100).toFixed(2);
    console.log(`Uploaded ${percentage}%`);
  },
  onSuccess: () => {
    console.log('Upload completed!');
  },
  onAfterResponse: (req, res) => {
    const embedUrl = res.getHeader('X-Embed-URL');
    console.log('Embed URL:', embedUrl);
    // Example: https://play.3speak.tv/embed?v=chessfighter/yn77aj9g
  }
});

upload.start();

Upload Token (Web App — Recommended for Browsers)

Step 1 — Your backend requests a token:

// Server-side (Node.js, Python, etc.)
const response = await fetch('https://embed.3speak.tv/uploads/token', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.EMBED_API_KEY,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    owner: 'chessfighter',
    frontend_app: 'my-video-app',
    short: false,
    allowed_origins: ['https://my-video-app.com'],
    ttl: 600
  })
});

const { token, upload_url, expires_at } = await response.json();
// Send `token` and `upload_url` to your frontend

Step 2 — Your frontend uploads with the token:

// Client-side (browser)
const upload = new tus.Upload(file, {
  endpoint: uploadUrl,  // from your backend
  headers: {
    'Authorization': `Bearer ${token}`  // from your backend
  },
  metadata: {
    filename: file.name,
    filetype: file.type,
    // owner, frontend_app, short are already in the token — no need to set them
  },
  onError: (error) => {
    console.error('Upload failed:', error);
  },
  onProgress: (bytesUploaded, bytesTotal) => {
    const percentage = (bytesUploaded / bytesTotal * 100).toFixed(2);
    console.log(`Uploaded ${percentage}%`);
  },
  onSuccess: () => {
    console.log('Upload completed!');
  },
  onAfterResponse: (req, res) => {
    const embedUrl = res.getHeader('X-Embed-URL');
    console.log('Embed URL:', embedUrl);
  }
});

upload.start();

Tracking Progress

After starting an upload, you can track the video's status and encoding progress by polling the public GET /video/:permlink endpoint. Extract the permlink from the embed URL returned in the X-Embed-URL header.

Key fields to monitor:

Field	Description
`status`	`uploading` → `processing` → `published` (or `failed`)
`encodingProgress`	Encoding percentage (0–100). Updates in real-time as the encoder works.

Example — polling for progress:

const permlink = embedUrl.split('/').pop(); // e.g. "yn77aj9g"

async function pollProgress() {
  const res = await fetch(`https://embed.3speak.tv/video/${permlink}`);
  const video = await res.json();

  console.log(`Status: ${video.status}, Progress: ${video.encodingProgress}%`);

  if (video.status === 'published') {
    console.log('Video is ready!');
    console.log('Manifest CID:', video.manifest_cid);
    return;
  }

  if (video.status === 'failed') {
    console.error('Encoding failed');
    return;
  }

  // Poll every 5 seconds while uploading/processing
  setTimeout(pollProgress, 5000);
}

pollProgress();

Status lifecycle:

uploading → processing → published
                ↘ failed

The embed player handles all states automatically (showing upload/processing animations), so the embed URL is usable immediately — but polling lets your app show custom progress UI.

Project Structure

3speakembed/
├── src/
│   ├── config/
│   │   └── config.ts          # Configuration loader with multi-encoder support
│   ├── database/
│   │   └── mongodb.ts         # MongoDB connection and operations
│   ├── dispatcher/
│   │   └── jobDispatcher.ts   # Job queue manager with round-robin load balancing
│   ├── middleware/
│   │   ├── auth.ts            # API key validation middleware
│   │   └── adminAuth.ts       # Admin password middleware
│   ├── utils/
│   │   ├── videoId.ts         # Video ID generator
│   │   ├── keyGenerator.ts    # API key generator
│   │   ├── ipfs.ts            # IPFS pinning utilities
│   │   └── uploadToken.ts     # Upload token signing and verification
│   └── index.ts               # Main server file
├── public/
│   ├── index.html             # Landing page with integration docs
│   ├── demo.html              # Upload demo interface
│   └── admin.html             # Admin panel (API keys & encoder management)
├── scripts/
│   ├── dropOldIndex.ts        # Database maintenance utilities
│   └── testEncoder.ts         # Encoder testing script
├── uploads/                   # TUS upload storage directory
├── .env.example               # Environment variables template
├── .gitignore
├── package.json
├── tsconfig.json
├── ENCODERS.md                # Multi-encoder configuration guide
└── README.md

Data Schemas

VideoMetadata

interface VideoMetadata {
  owner: string;                // Username
  permlink: string;             // Random 8-character ID
  frontend_app: string;         // Frontend application identifier
  status: 'uploading' | 'processing' | 'published' | 'failed' | 'deleted';
  input_cid: string | null;     // IPFS CID of uploaded file
  ipfs_pin_endpoint: string | null; // IPFS endpoint used for pinning
  manifest_cid: string | null;  // IPFS CID of HLS manifest
  thumbnail_url: string | null; // Video thumbnail URL
  short: boolean;               // Is short-form video (≤60s, 480p max)
  duration: number | null;      // Video duration in seconds
  size: number | null;          // File size in bytes
  encodingProgress: number;     // Encoding progress (0-100)
  originalFilename: string | null; // Original filename
  hive_author: string | null;   // Linked Hive post author
  hive_permlink: string | null; // Linked Hive post permlink
  hive_title: string | null;    // Hive post title
  hive_body: string | null;     // Hive post body
  hive_tags: string[] | null;   // Hive post tags
  embed_url: string | null;     // Embed URL path (e.g., @user/permlink)
  embed_title: string | null;   // Display title for embed (set by frontend)
  listed_on_3speak: boolean;    // Whether listed on 3speak.tv
  processed: boolean;           // Whether fully processed
  processedAt: Date | null;     // When processing completed (set by frontend)
  views: number;                // View count (set by player/analytics)
  createdAt: Date;              // Upload start timestamp
  updatedAt: Date;              // Last modification timestamp
}

EncodingJob

interface EncodingJob {
  owner: string;                // Video owner username
  permlink: string;             // Video ID
  status: 'pending' | 'encoding' | 'completed' | 'failed';
  input_cid: string;            // IPFS CID of source video
  encoder?: string;             // Assigned encoder name
  attempts: number;             // Retry counter (max 3)
  error?: string;               // Error message if failed
  manifest_cid?: string;        // IPFS CID of output manifest
  thumbnail_url?: string;       // Generated thumbnail URL
  duration?: number;            // Video duration
  createdAt: Date;              // Job creation time
  updatedAt: Date;              // Last update time
}

ApiKey

interface ApiKey {
  key: string;                  // Hashed API key
  name: string;                 // Application name
  createdAt: Date;              // Creation timestamp
}

Encoder

interface Encoder {
  name: string;                 // Unique encoder name
  url: string;                  // Encoder API endpoint URL
  apiKey: string;               // Encoder authentication key
  enabled: boolean;             // Whether encoder is active
  createdAt: Date;              // Creation timestamp
  updatedAt: Date;              // Last modification timestamp
}

Status Flow & Architecture

Video Processing Pipeline

Upload Start: TUS creates video record with status uploading, returns embed URL immediately
Upload Complete: File pinned to IPFS, input_cid stored, encoding job created with status pending
Job Dispatch: Dispatcher picks up pending job, assigns to available encoder via round-robin, status becomes encoding
Encoding: Encoder processes video and calls webhook with results
Webhook Update: Video status set to published, manifest_cid and thumbnail_url stored, job marked completed
Player Ready: Embed URL now serves encoded HLS video

Video Status States

uploading - TUS upload in progress
processing - Upload complete, encoder is working on it
published - Video is ready to watch
failed - Processing failed
deleted - Video marked for deletion

The video player handles all these states automatically, showing appropriate animations for uploading/processing/failed states, making the embed URL usable immediately.

Job Dispatcher

The dispatcher runs every 30 seconds:

Queries MongoDB for jobs with status pending
Selects next available encoder using round-robin algorithm
Sends job to encoder with IPFS gateway URL and webhook credentials
Updates job status to encoding with assigned encoder name
Retries failed jobs up to 3 times before marking as failed

See ENCODERS.md for multi-encoder configuration details.

Automatic File Cleanup

The service includes an automatic cleanup system to prevent disk space exhaustion:

Scheduled Cleanup: Runs automatically at configurable intervals (default: every 24 hours)
Retention Period: Deletes temporary upload files older than configured days (default: 7 days)
Manual Trigger: Admin endpoint to run cleanup on-demand
Preview Mode: Check what files would be deleted without actually deleting them

Configuration:

CLEANUP_ENABLED=true                # Enable/disable cleanup (default: true)
CLEANUP_INTERVAL_HOURS=24          # How often to run cleanup (default: 24)
CLEANUP_RETENTION_DAYS=7           # Delete files older than this (default: 7)

Admin Endpoints:

GET /admin/cleanup/preview - Preview files that would be deleted
POST /admin/cleanup/run - Manually trigger cleanup

Encoder Management

Encoders can be configured via environment variables (for initial seeding) or managed dynamically through the admin panel at /admin.html.

Initial Setup (Environment Variable):

ENCODERS=[{"name":"encoder1","url":"https://encoder.example.com","apiKey":"key","enabled":true}]

On first startup, encoders from the env var are seeded into MongoDB. After that, encoders are managed via the database.

Admin API Endpoints:

GET /admin/encoders - List all encoders (API keys excluded)
POST /admin/encoders - Add a new encoder
PATCH /admin/encoders/:name - Update encoder (URL, API key, enable/disable)
DELETE /admin/encoders/:name - Remove an encoder

Add Encoder Example:

curl -X POST http://localhost:3001/admin/encoders \
  -H "X-Admin-Password: your-admin-password" \
  -H "Content-Type: application/json" \
  -d '{"name": "encoder2", "url": "https://encoder2.example.com", "apiKey": "secret-key"}'

What Gets Cleaned:

Abandoned TUS uploads (files never completed)
Failed IPFS pinning uploads (pinning failed, file left behind)
Orphaned .json metadata files
Any temporary files older than retention period

Files are only deleted after successful IPFS pinning or after exceeding the retention period, ensuring no data loss for active uploads.

User Management

Users are created automatically on first upload. Admins can manage users via the admin panel at /admin.html or the API.

Admin API Endpoints:

GET /admin/users - List all users (supports ?search=, ?limit=, ?skip=)
PATCH /admin/users/:username/ban - Ban/unban a user (body: { "banned": true })
PATCH /admin/users/:username/premium - Grant/revoke premium (body: { "premium": true })

Premium Users:

Premium users get multi-resolution encoding (1080p, 720p, 480p) instead of the default 480p-only. The premium flag is sent to encoders at dispatch time.

Frontends check premium status via GET /users/:username/premium (API key required)
Admins toggle it via PATCH /admin/users/:username/premium or the admin panel

Grant Premium Example:

curl -X PATCH http://localhost:3001/admin/users/coolmole/premium \
  -H "X-Admin-Password: your-admin-password" \
  -H "Content-Type: application/json" \
  -d '{"premium": true}'

Embed URL Format

Videos are accessible via embed URLs in the following format:

https://play.3speak.tv/embed?v={username}/{videoId}

Example:

https://play.3speak.tv/embed?v=chessfighter/yn77aj9g

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 57 Commits
.github		.github
public		public
scripts		scripts
src		src
.env.example		.env.example
.gitignore		.gitignore
ENCODERS.md		ENCODERS.md
README.md		README.md
embed-upload.service		embed-upload.service
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

3Speak Embed Video Upload Service

Features

Prerequisites

Installation

Usage

Development Mode

Production Mode

Authentication

Option 1: Direct API Key (Server-to-Server / Mobile Apps)

Option 2: Upload Tokens (Web Apps / Client-Side Uploads)

API Endpoints

Health Check

Get Video Metadata

Check User Premium Status

TUS Upload Endpoint

Request Upload Token

Upload Examples

Direct API Key (Server-to-Server / Mobile)

Upload Token (Web App — Recommended for Browsers)

Tracking Progress

Project Structure

Data Schemas

VideoMetadata

EncodingJob

ApiKey

Encoder

Status Flow & Architecture

Video Processing Pipeline

Video Status States

Job Dispatcher

Automatic File Cleanup

Encoder Management

User Management

Embed URL Format

License

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages