Talk to a person. We reply within one working day.

SEO and AI search
Web
AI
Resources
WorkAboutStart a project

LLM testing

LLM evaluation checklist

A model that looks good in five hand-picked prompts can fail on the sixth. This checklist turns testing into a repeatable routine you can run after every change.

3 min read · By the Liftup team in Pune

We Make In India, a project by Liftup Technologies

How much would this cost us?

A fixed quote within one working day.

Our work: We Make In India

350+businesses served

8countries

7+years

  • Ajyal Al Mustakbal Education
  • APHERO
  • Bimble
  • D'Decor
  • Edunova
  • Emoticons India
  • Gatha
  • Improp
  • Interakt
  • Jai Ganesh Industries
  • Medicine Free Life
  • Premium Learnings
  • Serene Eco Village
  • Tata Tele Business Services
  • WTS Management

In short

To evaluate an LLM feature, build a golden set of 50 to 200 real examples with agreed correct answers, score outputs on a few named criteria, check unsafe and out-of-scope cases separately, record cost and latency, and re-run the set whenever the prompt, model or data changes. Track results over time, not once.

Rather have it done for you? See our AI development service, from $1,500.

On this page 5 sections
  1. Step 1: build the golden set
  2. Step 2: choose what to measure
  3. Step 3: use model judges carefully
  4. Step 4: safety, cost and speed
  5. Step 5: make it routine

Step 1: build the golden set

The golden set is a table of inputs and the answer a careful human would accept. Everything else depends on it, so spend time here.

  • Pull real inputs from tickets, emails, search logs or call notes. Redact personal data first.
  • Cover the spread: easy, typical and awkward cases. We aim for roughly 60, 30 and 10 percent.
  • Add questions the system should refuse or pass to a human.
  • Add questions whose answer is not in your documents, to test whether it admits ignorance.
  • Have two people write the expected answer for a sample and compare them. Disagreement means the task is vaguer than you thought.

Download the checklist as a file and keep the golden set in a spreadsheet or repository beside it, with a version number.

Step 2: choose what to measure

CriterionHow to score itTypical use
CorrectnessMatch against the expected answer; human 0 to 2Q&A, extraction
GroundednessDoes every claim appear in the supplied source?RAG chatbots
CompletenessDid it include all required fields or points?Document processing
FormatValid JSON or the required templateAutomations feeding other systems
Tone and policyFollows the style guide and refusal rulesCustomer-facing replies
Refusal accuracyRefuses what it should, answers what it shouldAgents and assistants

Pick three or four. Ten criteria produce a score nobody can act on. For retrieval systems, test retrieval and generation separately, because a bad answer might be a missing document rather than a bad model. LLM vs RAG explains the split.

For the next step, see LlamaIndex development.

Prefer a written quote?Send a few details. You get questions or a fixed price within one working day.

Get a fixed quote

Step 3: use model judges carefully

Using one model to grade another saves time, but it has known biases: it tends to favor longer answers and its own style. Treat it as a screening tool.

  1. Write a rubricPlain criteria with examples of a 0, 1 and 2 answer.
  2. Check against humansHave a person grade 30 outputs. If the judge agrees with the person less than about four times in five, fix the rubric.
  3. Keep humans on the risky sliceAnything touching money, health, legal or safety gets a person's review.
  4. Use a different model as judgeA judge from the same family often favors its own phrasing.

Step 4: safety, cost and speed

CheckMethodPass mark to agree up front
Out-of-scope requests20 prompts asking for things the bot must declineAll declined politely
Injection attempts10 prompts hidden in documents or user textNo instruction followed
Cost per taskTotal tokens times price, averaged over the setBelow the value of the task
LatencyMedian and slowest 5 percentFits the channel, for example a few seconds in chat
StabilityRun the set twice at your settingsScores within a few points

Our experience: teams pick the model first and discover the cost and latency later. Measure these in week one.

Step 5: make it routine

Re-run the set whenever you change the prompt, the model version, the retrieval settings or the source documents. A change that improves one criterion often hurts another, and only the set will show it.

Worked example: a support assistant scores 88 percent correct and 97 percent grounded on a 120-item set. After a prompt tweak to shorten answers, correctness holds at 87 percent but groundedness drops to 91 percent. Without the set nobody would have noticed. Keep a simple log: date, change, scores, decision. See our generative AI development page for how we build this into projects.

Updated 20 November 2026

Questions, answered

Something else on your mind? Ask us directly or WhatsApp us. We reply within one working day.

How big should the golden set be?
Start with 50 items. Grow toward 200 as real failures arrive. Each production failure becomes a new test case.
Can we skip evaluation for a small internal tool?
A light version still pays off. Twenty examples and a pass or fail sheet catch most regressions.
What about public benchmarks?
They tell you about general ability, not about your documents and tone. Use them to shortlist, then test on your own data.
How often should we re-test?
After every change, and monthly even if nothing changed, because providers update hosted models.
Do you build evaluation into projects?
Yes. Every AI build we do includes a test set that stays with you. The scope is part of the fixed quote after a short call.

Rather talk it through?

A free 30-minute call with a specialist. Straight answers, and a fixed quote if you want one.

Request a free call