github_harvester #285
Replies: 2 comments 1 reply
-
🚜 Universal GitHub Harvester v1.0Generated by robumf with Gemini 3 Sealed Warehouse Manager for UltraSinger & AI Models A robust automation tool designed to harvest, validate, and maintain high-performance AI models (Whisper, MDX-Net, Demucs) in a local "Warehouse." This tool is optimized for Python 3.10 environments and NTFS-shared storage, ensuring your AI workflow is completely offline-ready. 🌟 Key Features
📂 Warehouse ArchitectureAfter running the harvester, your
🚀 How to Use1. SetupEnsure your Seagate Cabinet (Source) and your NTFSshare (Warehouse) are both mounted. 2. Run the HarvesterExecute the script from your terminal: bash github_harvester.sh
# Force Offline Mode
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
# Execute with Local Alignment
python UltraSinger.py \
--input "your_song.mp3" \
--language en \
--whisper_align_model "/media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/whisper"
## 🚀Admendments
Due to Conflicting Info Between What I have and A.I Gemini
How to find dirrect links of the biggest files for off location downloads
WHEELS
cuDNN 663 M https://pypi.org/project/nvidia-cudnn-cu12/9.1.0.70/#files
TF-CPU 260 M https://pypi.org/project/tensorflow-cpu/2.18.0/#files
Torch 210 M https://pypi.org/project/torch/2.5.1/#files
ONNX-OV 70 M https://pypi.org/project/onnxruntime-openvino/1.16.0/#files
SciPy 36M https://pypi.org/project/scipy/1.15.1/#files
tensorflow tensorflow-2.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 644 M https://pypi.org/project/tensorflow/2.19.0/#files
torch torch-2.9.1+cpu-cp310-cp310-manylinux_2_28_x86_64.whl 899.8 MB https://pypi.org/project/torch/#files
torch torch-2.7.1+cpu-cp310-cp310-manylinux_2_28_x86_64.whl 821.2 M https://pypi.org/project/torch/2.7.1/#files
llvmlite llvmlite-0.46.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl 56.3 M https://pypi.org/project/llvmlite/#files
llvmlite llvmlite-0.44.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 30.3 M https://pypi.org/project/llvmlite/0.44.0/#files
MODELS Whisper python3 UltraSinger.py --input "song.mp3" --whisper_model
When you launch UltraSinger, use the --language flag. This prevents the auto-detection script from trying to download language-specific alignment models:
python UltraSinger.py --input "song.mp3" --language en --whisper_model large-v3
Version / Type File Name Approx. Size Official Direct Download (Plain Text Only)
Large-v3 large-v3.pt 2.9 GB https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt
Large-v2 large-v2.pt 2.9 GB https://openaipublic.azureedge.net/main/whisper/models/81f7c96c852ee8fc832187b0132e569d6c3065a3252ed18e56effd0b6a73e524/large-v2.pt
Turbo? turbo.pt 1.6 GB https://openaipublic.azureedge.net/main/whisper/models/aff26ae408abcba5fbf8813c21e62b0941638c5f6eebfb145be0c9839262a19a/large-v3-turbo.pt
Medium medium.pt 1.4 GB https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt
Medium.en medium.en.pt 1.4 GB https://openaipublic.azureedge.net/main/whisper/models/d7440d1dc186f76616474e0ff0b3b6b879abc9d1a4926b7adfa41db2d497ab4f/medium.en.pt
Small small.pt 466 MB https://openaipublic.azureedge.net/main/whisper/models/9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794/small.pt
Small.en small.en.pt 466 MB https://openaipublic.azureedge.net/main/whisper/models/f953ad0fd29cacd07d5a9eda5624af0f6bcf2258be67c92b79389873d91e0872/small.en.pt
Base base.pt 142 MB https://openaipublic.azureedge.net/main/whisper/models/ed3a0b6b1c0edf879ad9b11b1af5a0e6ab5db9205f891f668f8b0e6c6326e34e/base.pt
Base.en base.en.pt 142 MB https://openaipublic.azureedge.net/main/whisper/models/25a8566e1d0c1e2231d1c762132cd20e0f96a85d16145c3a00adf5d1ac670ead/base.en.pt
Tiny tiny.pt 75 MB https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt
Tiny.en tiny.en.pt 75 MB https://openaipublic.azureedge.net/main/whisper/models/d3dd57d32accea0b295c963554e173b78ee57fe7ca2ad289e47a2a4c2206430c/tiny.en.pt
MDX python UltraSinger.py --input "path/to/song.mp3" --demucs mdx_extra
.onnx and .json
models/mdx_networks
--demucs mdx_extra
Open your terminal.
Run this command to create the file directly in your warehouse:
cat <<EOF > /media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/mdx/Kim_Vocal_2.json
{
"model_path": "Kim_Vocal_2.onnx",
"model_name": "Kim_Vocal_2",
"is_kim_model": true,
"mdx_dim_f_set": 3072,
"mdx_dim_t_set": 8,
"mdx_n_fft_scale_set": 6144,
"mdx_hop_set": 1024,
"mdx_sr_set": 44100
}
EOF
demucs
https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/f7e0c4bc-ba3fe64a.th
https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/d12395a8-e57c48e6.th
https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/92cfc3b6-ef3bcb9c.th
https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/04573f0d-f3cf25b2.th
https://raw.githubusercontent.com/facebookresearch/demucs/main/demucs/remote/htdemucs_ft.yaml
hugginface?
# 1. Create the standardized folder
mkdir -p /media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/mdx_networks
# 2. Move your existing ONNX and math files
mv /media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/mdx/* /media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/mdx_networks/
# 3. Remove the old directory
rmdir /media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/mdx
# Navigate to your warehouse
cd /media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/mdx_networks/
# Download the ONNX weights
wget https://huggingface.co/seanghay/uvr_models/resolve/main/MDX23C-InstVoc.onnx
# Download the required JSON config
wget https://raw.githubusercontent.com/Anjok07/ultimatevocalremovergui/master/gui_data/model_data/mdx_model_data/model_data/MDX23C-InstVoc.json
https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/resolve/main/pytorch_model.bin
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python UltraSinger.py --input "your_song.mp3" --language en --whisper_align_model "/media/robumf/NTFSshare/Warehouse/UltraSinger-main/Models/whisper"
https://github.com/ShadowLoveElysia/Whisper-vits-svc-LargeV3
https://github.com/Recordtini/MusicSepGUI/blob/main/models.json
https://github.com/sindresorhus/awesome |
Beta Was this translation helpful? Give feedback.
-
|
#!/bin/bash --- UNIVERSAL GITHUB HARVESTER v1.0 (Public Beta) ---Linux CPU plus A.I ModelsOptimized for Python 3.10 "Sweet Spot" Environments1. IDENTITY & DYNAMIC DETECTIONecho "--------------------------------------------------------" CRITICAL FIX: Define CURRENT_DIR before the IF statementCURRENT_DIR=$(pwd) if [[ "$CURRENT_DIR" == "install/CPU" ]]; then The -i flag provides the detected name as a default,but -e (readline) allows the user to backspace and change it.read -p "🏷️ Project ID: " -i "$DETECTED_NAME" -e PROJECT_NAME Scan for Requirements Filesmapfile -t FILES < <(find $SEARCH_PATH -maxdepth 2 -name "*.txt" 2>/dev/null | grep -i "requirement" | sort -u) 2. UNIVERSAL DRIVE MAPPINGDefaults to standard Linux media mount pointUSER_HOME="/media/$USER" A. FIND CABINET: Smart Detection for External Source(Change 'Seagate Basic' to your specific drive label if needed, or leave dynamic)DETECTED_SEAGATE=$(ls -d "$USER_HOME"/Seagate* 2>/dev/null | head -n 1) B. FIND WAREHOUSE: Defaults to a "Warehouse" folder in Home if NTFSshare isn't foundif [ -d "$USER_HOME/NTFSshare" ]; then C. FIND BUILD DIR: NVMe/Root DetectionNVME_PATH=$(df "$PWD" | tail -1 | awk '{print $6}') echo "--------------------------------------------------------" 3. SAFETY CHECKS (The "Stop the User" Logic)echo "--------------------------------------------------------" Verify Build Dirif [ ! -w "$NVME_PATH" ]; then Verify Warehouse RootWAREHOUSE_ROOT=$(dirname "$WAREHOUSE") If parent doesn't exist, check if we can create itif [ ! -d "$WAREHOUSE_ROOT" ]; then WHEEL_DEST="$WAREHOUSE/Wheelhouse" 4. PYTHON ENGINE & SWEET SPOT LOCKecho "--------------------------------------------------------" if [ -n "$PY_BIN" ]; then 5. SANITIZATION CHAMBER (v6.9 - Build-Proof)OFFLINE_REQ="$WAREHOUSE/requirements_offline.txt" Fixes: AV build error, +cpu 404 error, and Windows bloatgrep -vE "nvidia-|file://|winshell|pywin32" "$REQ_FILE" | 6. THE HARVESTecho "--------------------------------------------------------" A. CABINET SYNC (Local Import)if [ -d "$SOURCE_ROOT/Wheelhouse" ]; then B. WEB DOWNLOAD LOOP (Dual-Index + Mount Guard)while read -r line; do done < "$OFFLINE_REQ" 7. VALIDATION & SPECULATIONecho "--------------------------------------------------------" echo "📊 SUMMARY: $WHEEL_COUNT wheels collected for $REQ_COUNT core requirements." 8. RECURSIVE RESCUE (Consolidated Model Harvest)echo "--------------------------------------------------------" if [[ "$DO_MODELS" == "y" || "$DO_MODELS" == "Y" ]]; then { 9. FINAL OUTPUT & INVENTORYEND_TIME=$(date +%s) echo "--------------------------------------------------------" Report from the specific Warehouse pathsecho "📂 MODEL REPOSITORY SUMMARY:" echo " 🎙️ Whisper/Align Assets: ${W_MODELS:-0}" |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Ready for review.
I've been learning how to effectively use AI, specifically Gemini, to achieve my desired outcomes. I learn best through hands-on experience, so I've set myself three tasks that I can currently perform, albeit imperfectly.
I'm confident that the harvester is as complete as it can be given my current hardware limitations.
I wasn't satisfied with the default behavior of virtual environments, where downloads were deleted upon creation. This meant redownloading everything if I needed to rebuild.
I've strived to make it as universally compatible as possible and have included extensive comments throughout. I'm running this on a small HP Elitedesk 800 G3 Mini (no GPU) with Linux Mint.
The GitHub Harvester will:
You now have the option to copy the wheelhouse, then delete it. The models, don't delete, to a USB drive for later use or to install them on a similar system. If run again, it will locate needed components based on the requirement file. Don't forget the .json .yaml files
Commits are welcome. Thanks for participating.
The links didn't work. I'll come up with something
github_harvester.sh
github_harvester.mk.txt
Beta Was this translation helpful? Give feedback.
All reactions