Can this be made to be fully local + test multiple <8B models?

[TomLucidor]

This looks interesting! Wondering if the feature set can slowly expand to multiple tasks

• Long Context (long conversation and/or character card handling)
• Compliance to worldbuilding and not break common sense logic
• Diversity of writing + anti-slop mechanism (see EQ-Bench and Heretic)

[inflatebot]

It is fully local! It uses the KoboldAI API, and KoboldCPP is designed to be run locally.
In hindsight, I should have used something more flexible, but it shouldn't take much work to adapt if you want to use a different backend. I just haven't cared to touch it, since it served its purpose (testing a model merge I was working on.)
It's a really simplistic, procedural thing, I wrote it by hand with the w3schools reference open in a Firefox window.

As a result, it should be easy to extend. Really, this is a framework for automating evaluations you're already doing, rather than an evaluation in and of itself.

Any new task which can be defined without external data is a matter of adding it to run_config.yaml and, if it's a new process that's outside the existing "ic" and "oc" procedures, adding an entry in setup_run (which in hindsight should really be done more elegantly, wow) and adding build_procedure and execute_procedure calls in main() (which is just copy-pasting some lines.)

For things involving pre-provided contexts from other files (long-context, card handling, lore consistency) we'd have to load them in somewhere, but that doesn't seem a terribly difficult thing to do.
Truthfully, though, I've been wanting to redo this, so maybe that's the direction I go in.
Health permitting, I'll look into it, and if I forget, feel free to steal it and plug it into whichever agentic coding app you're most privy to.

[TomLucidor]

It is fully local! It uses the KoboldAI API, and KoboldCPP is designed to be run locally.

Maybe I just used vLLM-MLX or MLX-OpenAI-Server for too long, along side vLLM or llama.cpp too often. Something that is more generic and not strictly trapped into an RP engine (since SillyTavern is getting less RP-focused while other RP engines are getting better)

this is a framework for automating evaluations you're already doing, rather than an evaluation in and of itself

I guess we need some kinda uncensored benchmark to see if any of the SLMs out there are really "functional"

ruthfully, though, I've been wanting to redo this, so maybe that's the direction I go in.
Health permitting, I'll look into it, and if I forget, feel free to steal it and plug it into whichever agentic coding app you're most privy to.

Please do, and I don't helping along the way as well