AI vendor evaluation checklist

A weighted scorecard

Score each vendor 0 to 5 on every row, multiply by the weight, and add up. Agree the weights before you see any demos, or the loudest presenter will set them for you.

Download the checklist as a file and give each person on your panel a copy. Compare scores before you talk, so nobody anchors on the first opinion.

Questions to ask, grouped

Use these as a written questionnaire. Vendors who answer in writing are easier to hold to their answers.

• Which model or models do you use, and who decides when to change them?
• Is our data used to train any model, yours or your provider's? Where is that written?
• Where is data stored and processed, and which sub-processors touch it?
• How long do you keep prompts, outputs and logs, and can we set the period?
• Can you share a recent security assessment or a summary of your controls? Which standard does it map to?
• What happens to our data and configuration if we end the contract?
• How do you measure accuracy, and can we see the test set and the failures?
• What does a human review step look like in your product, and can we change it?
• How do you handle a wrong or harmful answer that reaches a customer?
• What do you need from us in the first month, and from whom?

Evidence beats a demo

1. Give them a sampleSend 30 to 50 real, redacted items. Include difficult ones, such as scans, odd formats and angry customers.
2. Define a pass mark firstFor example, 90 percent of answers correct and none that are confidently wrong on pricing or policy.
3. Ask for the miss listEvery system fails somewhere. A vendor who shows you the failures is more trustworthy than one who shows none.
4. Repeat after a weekRun the same sample again. Large differences mean the output is less stable than the demo suggested.

Our rule: if a vendor will not run a trial on your data, the price is not the real cost. The cost is finding out later.

Contract checks

If you are comparing building with buying, read build vs buy AI and custom AI vs off the shelf first.

Red flags

• Only canned demos, never your data
• Promises of fixed accuracy before seeing the task
• No clear answer on where data goes
• No named technical contact
• A contract that renews automatically with a long notice period
• Claims that the product needs no human oversight

If you are hiring an agency, apply the same scorecard to us. We are happy to be evaluated this way.