TDD use cases for rspec-llama

[Sergyenko]

Let's gather most obvious or easy to reproduce use cases for rspec-llama, and then I will convert it into an overview topic.

[vidas]

rspec-llama use cases

Disclaimer	RSpec is not about tests, it’s about specification.

**Some general ideas, no specific cases, though. Could work as an inspiration? **

Due to its heavy compute use and/or price (when using external APIs) rspec-llama should not be used when regular matchers suffice.

It’s definitelly not a good use case for unit tests.

The best fit - when the spec is difficult to specify in code due to ambiguity or subjectiveness, ie. you would like to use a person (eg amazont turk).

Test-Driven-Development-Design

There is this TDD (or BDD) practice, where the third D stands for Design, not Development.

This is the practice where you design specific functionality from a user perspective.

You write an example use cases of user code interacting with API (eg calling methods etc) first pretending it’s already implemented (not unlike doctests in python). After you are happy with the API, you start implementing it. Implementation often uncovers another use cases or gotchas, which make you adapt the API and so on, until the solution converges to something acceptable.

The intermediate specs are often handwavy, schemas are undefined etc. It would be great to write something like "response includes document identifiers", but you have no idea how the field is names, what is identifier (UUID? URL? slug?). With rspec-llama you can write specs like this.

They should be replaced with precise specs after the design phase is finished. (This also allows you to control LLM usage, as you don’t have to ship these specs to CI).

Acceptance specs

In some cases rspec-llama could substitute cucumber scenarios, mostly acceptance specs, where requirements are given in a human language, eg as part of a ticket.

Specs can describe criteria in vague words like "convenient", "beautiful", "describes".

The llamagator

When your codepath includes LLMs, even with fixed inputs and fixed seeds it’s difficult to get deterministic outputs. Sometimes you change the model and the output changes significantly to break some criteria.

The requirement can state "be polite". The spec can check for politeness and will catch cases, where the change of the model or the prompt, or adding some unverified input makes it rude.

Such specs would provide specification for the LLM (agent) pipelines and you can even do "AI BDD" - write the spec and then tweak the prompt until it passes.

(wrote this as an asciidoc for fun, and had to convert to html before pasting it here, sorry)