AI-Vtuber

This code is designed to read chat messages from YouTube and then utilize OpenAI's GPT-3 language model to generate responses. The output from GPT-3 is then read out loud using a TTS (Text-to-Speech) engine provided by ElevenLabs.

create a config.json

{
    "keys": [
        {
            "EL_key":  "",
            "OAI_key": ""
        }
    ],
    "EL_data": [
        {
          "voice": "MF3mGyEYCl7XYWbV9V6O"
        }

    ],
    "OAI_data": [
        {
          "model": "gpt-3.5-turbo-instruct",
          "prompt": "This is how a toxic entitled evil female streamer responded in a conversation. She would respond in a toxic and rude manner. She would talk about the message and would elaborate on it as well as share some of her experiences if possible. She would also go on a tangent if possible.",
          "temperature": 0.9,
            "max_tokens": 128,
            "top_p": 1,
            "frequency_penalty": 1,
            "presence_penalty": 1
        }

    ]
}

Setup

Install dependencies

git clone https://github.com/UGing265/VtuberAI_Neko.git
cd VtuberAI_Neko
pip install -r requirements.txt

It also requires ffmpeg to be installed

if you use window download this link and chose "essentials_build"

you have to set up "edit environments" and link to file bin contains .exe

win + R -> cmd -> typing: ffmpeg (if it work u can run project now)

Usage

Edit the variables EL_key and OAI_key in config.json

EL_key is the API key for ElevenLabs. Found in Profile Settings

OAI_key is the API key for OpenAI. Found here

Then run run.py

Default TTS

python run.py -id STREAMID 

Elevenlabs TTS

python run.py -id STREAMID -tts EL 

then you're set

Notes

Replace STREAMID with the stream's ID that you can find on the Youtube Stream link

You can change the voice by changing voice in config.json. You can find the ID's here in Get Voices

EXTRA (IMPORTANT)

FFMPEG IS REQUIRED: https://www.gyan.dev/ffmpeg/builds

If you met local bot errors (TTS only works once) --> Switch to EL (run.py ->> parser.add_argument("-tts", "--tts_type", default="EL", choices=tts_list, type=str))

Live Demo

Livestream 1

Livestream 2

Other

I used This VTS plugin and VB Audio cable to make her mouth move and be able to play music at the same time

Please note that this project was created solely for fun and as part of a YouTube video, so the quality and reliability of the code may be questionable. Also, after the completion of the project checklist, there won't be much activity in updating or improving this repository. Nonetheless, we hope that this project can serve as a source of inspiration for anyone interested in building their own AI Vtuber.

• Clean up
• GUI
• Executables (exe, bat or sh)
• Extra features (maybe) (Prompt injection protection, questions only mode, virtual audio)

License

This program is under the MIT license