LLM testing
LLM evaluation checklist
A model that looks good in five hand-picked prompts can fail on the sixth. This checklist turns testing into a repeatable routine you can run after every change.

How much would this cost us?
A fixed quote within one working day.
350+businesses served
8countries
7+years
In short
To evaluate an LLM feature, build a golden set of 50 to 200 real examples with agreed correct answers, score outputs on a few named criteria, check unsafe and out-of-scope cases separately, record cost and latency, and re-run the set whenever the prompt, model or data changes. Track results over time, not once.
Rather have it done for you? See our AI development service, from $1,500.
On this page 5 sections
Step 1: build the golden set
The golden set is a table of inputs and the answer a careful human would accept. Everything else depends on it, so spend time here.
- Pull real inputs from tickets, emails, search logs or call notes. Redact personal data first.
- Cover the spread: easy, typical and awkward cases. We aim for roughly 60, 30 and 10 percent.
- Add questions the system should refuse or pass to a human.
- Add questions whose answer is not in your documents, to test whether it admits ignorance.
- Have two people write the expected answer for a sample and compare them. Disagreement means the task is vaguer than you thought.
Download the checklist as a file and keep the golden set in a spreadsheet or repository beside it, with a version number.
Step 2: choose what to measure
| Criterion | How to score it | Typical use |
|---|---|---|
| Correctness | Match against the expected answer; human 0 to 2 | Q&A, extraction |
| Groundedness | Does every claim appear in the supplied source? | RAG chatbots |
| Completeness | Did it include all required fields or points? | Document processing |
| Format | Valid JSON or the required template | Automations feeding other systems |
| Tone and policy | Follows the style guide and refusal rules | Customer-facing replies |
| Refusal accuracy | Refuses what it should, answers what it should | Agents and assistants |
Pick three or four. Ten criteria produce a score nobody can act on. For retrieval systems, test retrieval and generation separately, because a bad answer might be a missing document rather than a bad model. LLM vs RAG explains the split.
For the next step, see LlamaIndex development.
Prefer a written quote?Send a few details. You get questions or a fixed price within one working day.
Get a fixed quoteStep 3: use model judges carefully
Using one model to grade another saves time, but it has known biases: it tends to favor longer answers and its own style. Treat it as a screening tool.
- Write a rubricPlain criteria with examples of a 0, 1 and 2 answer.
- Check against humansHave a person grade 30 outputs. If the judge agrees with the person less than about four times in five, fix the rubric.
- Keep humans on the risky sliceAnything touching money, health, legal or safety gets a person's review.
- Use a different model as judgeA judge from the same family often favors its own phrasing.
Step 4: safety, cost and speed
| Check | Method | Pass mark to agree up front |
|---|---|---|
| Out-of-scope requests | 20 prompts asking for things the bot must decline | All declined politely |
| Injection attempts | 10 prompts hidden in documents or user text | No instruction followed |
| Cost per task | Total tokens times price, averaged over the set | Below the value of the task |
| Latency | Median and slowest 5 percent | Fits the channel, for example a few seconds in chat |
| Stability | Run the set twice at your settings | Scores within a few points |
Our experience: teams pick the model first and discover the cost and latency later. Measure these in week one.
Step 5: make it routine
Re-run the set whenever you change the prompt, the model version, the retrieval settings or the source documents. A change that improves one criterion often hurts another, and only the set will show it.
Worked example: a support assistant scores 88 percent correct and 97 percent grounded on a 120-item set. After a prompt tweak to shorten answers, correctness holds at 87 percent but groundedness drops to 91 percent. Without the set nobody would have noticed. Keep a simple log: date, change, scores, decision. See our generative AI development page for how we build this into projects.
Updated 20 November 2026
Questions, answered
Something else on your mind? Ask us directly or WhatsApp us. We reply within one working day.
How big should the golden set be?
Can we skip evaluation for a small internal tool?
What about public benchmarks?
How often should we re-test?
Do you build evaluation into projects?
Keep exploring
AI vendor evaluation checklist
Whether you are buying an AI product or hiring a team to build one, the demo is the least useful part. Ask for evidence on your data, your edge cases…
ServiceGenerative AI development: features that write, summarize and answer, with quality checks
Generative AI is easy to demo and hard to trust. Our work is the part between the demo and daily use: test sets, review steps, house style and honest…
ComparisonLLM vs RAG: the real difference
An LLM is the engine. RAG is a way of handing the engine your documents at the moment a question arrives, so it answers from them instead of from…
Rather talk it through?
A free 30-minute call with a specialist. Straight answers, and a fixed quote if you want one.








