LLM judge: add details on novelty score liabilities

[infotexture]

What problem does this solve?

Initial tests on my skills score well apart from low novelty scores, but the validator doesn't seem to provide many clues on what's dragging this value down.

I'd like to fix that, but I'm not sure what the problem is. I've tried asking Claude if the validator provides details on what it considers not-novel, but the answer seems inconclusive:

The validator doesn’t itemize what’s not novel — the feedback is always a holistic qualitative comment.

What would you like to see?

If the validator is aware of what an LLM already knows from training data, it seems like it should be possible to provide a few pointers to specific things to fix rather than just a score.

[dacharyc]

Hmm, I've got a corollary for this when the novel info is 3 or higher, asking the LLM judge to identify which details specifically are novel so an SME can try to assess whether they're accurate or hallucinations. I have some ideas about how I might try to get the judge to point to specific non-novel things and make recommendations, but will need to do some testing to see how effective I can make this.

As a general rule of thumb, though, skills that are just restating what's in the training data without any proprietary workflow information, project-specific details, or other non-obvious additions tend to rank low in novelty. I've seen this in practice in skills derived from public documentation, or skills that state basic debugging workflows that an LLM thinks it could reasonably produce on its own. One specific example is: I added a reference file to a skill instructing an agent how to walk through debugging a user's missing AWS Bedrock environment setup details. It consistently scored the reference file low on novelty because the agents already knew all the env variables, and thought the file was unnecessary. AWS Bedrock environment setup is well-covered in AWS documentation and probably well represented in other docs sets, so there's a reasonable possibility I could have dropped it and the agent would still be able to help debug a user's AWS Bedrock env setup.

I'll have a think about this and play around and see if anything I can do for this is actually helpful in practice.