# AI vendor evaluation checklist

> Whether you are buying an AI product or hiring a team to build one, the demo is the least useful part. Ask for evidence on your data, your edge cases and your exit.

- Page: https://liftup.tech/checklists/ai-vendor-evaluation-checklist/
- Type: Checklist
- Updated: 2026-11-18

## In short

Evaluate an AI vendor by scoring five areas with weights: fit to your use case, evidence on your own data, security and data handling, commercial terms including exit, and delivery and support. Ask for a trial on your real documents or tickets, written data terms, and a clear answer on what happens when the model changes.

## A weighted scorecard

Score each vendor 0 to 5 on every row, multiply by the weight, and add up. Agree the weights before you see any demos, or the loudest presenter will set them for you.

| Area | Weight | What a 5 looks like |
|---|---|---|
| Fit to your use case | 25% | They have done a close match and can describe what went wrong the first time |
| Evidence on your data | 25% | They ran a trial on a sample you chose and showed errors, not only successes |
| Security and data handling | 20% | Written answers on storage, retention, sub-processors and training on your data |
| Commercial terms and exit | 15% | You can export your data, prompts and configuration and leave without penalty |
| Delivery and support | 15% | Named people, response times in writing, a plan for model updates |

Download the checklist as a file and give each person on your panel a copy. Compare scores before you talk, so nobody anchors on the first opinion.

## Questions to ask, grouped

Use these as a written questionnaire. Vendors who answer in writing are easier to hold to their answers.

- Which model or models do you use, and who decides when to change them?
- Is our data used to train any model, yours or your provider's? Where is that written?
- Where is data stored and processed, and which sub-processors touch it?
- How long do you keep prompts, outputs and logs, and can we set the period?
- Can you share a recent security assessment or a summary of your controls? Which standard does it map to?
- What happens to our data and configuration if we end the contract?
- How do you measure accuracy, and can we see the test set and the failures?
- What does a human review step look like in your product, and can we change it?
- How do you handle a wrong or harmful answer that reaches a customer?
- What do you need from us in the first month, and from whom?

## Evidence beats a demo

1. **Give them a sample**: Send 30 to 50 real, redacted items. Include difficult ones, such as scans, odd formats and angry customers.
2. **Define a pass mark first**: For example, 90 percent of answers correct and none that are confidently wrong on pricing or policy.
3. **Ask for the miss list**: Every system fails somewhere. A vendor who shows you the failures is more trustworthy than one who shows none.
4. **Repeat after a week**: Run the same sample again. Large differences mean the output is less stable than the demo suggested.

> Our rule: if a vendor will not run a trial on your data, the price is not the real cost. The cost is finding out later.

## Contract checks

| Clause | Why it matters | Ask for |
|---|---|---|
| Data use | Your content could train someone else's model | No training on customer data, in writing |
| Sub-processors | Your data may pass through several companies | A current list and notice of changes |
| Model change | A new model version can change behavior | Notice period and a chance to re-test |
| Liability | Wrong output can cause real loss | Clear limits and what is excluded |
| Exit | Lock-in grows quietly | Export of data, prompts, workflows and logs |
| Service levels | Downtime stops your process | Uptime figure and remedy |

If you are comparing building with buying, read [build vs buy AI](https://liftup.tech/compare/build-vs-buy-ai/) and [custom AI vs off the shelf](https://liftup.tech/compare/custom-ai-vs-off-the-shelf/) first.

## Red flags

- Only canned demos, never your data
- Promises of fixed accuracy before seeing the task
- No clear answer on where data goes
- No named technical contact
- A contract that renews automatically with a long notice period
- Claims that the product needs no human oversight

If you are hiring an agency, apply the same scorecard to us. We are happy to be evaluated this way.

## Questions, answered

### How many vendors should we compare?

Three is enough. More than that and the evaluation becomes the project.

### Should we ask for references?

Yes, and ask what went wrong as well as what went well. A reference from a similar size company in your industry carries the most weight.

### Is a security attestation enough?

It is a good sign but not the whole picture. Check the scope of the report, whether it covers the AI features, and what the vendor says about model providers.

### What should a pilot cost?

Agree a fixed scope and price before it starts, and ask what is credited to the full build. For agency pilots our own approach is a fixed quote after a short call.

### Do we need a lawyer?

For data terms and liability, yes. Bring the questionnaire answers with you so the review is faster.

## Related

- [Build vs buy AI: the real difference](https://liftup.tech/compare/build-vs-buy-ai/)
- [LLM evaluation checklist](https://liftup.tech/checklists/llm-evaluation-checklist/)
- [AI consulting services: decide what to build, what to buy and what to skip](https://liftup.tech/services/ai-consulting/)

---

Get a fixed quote: https://liftup.tech/contact/ · email rohit@liftup.tech · WhatsApp +91 77689 10234 (https://wa.me/917768910234) · reply within one working day.
Source page: https://liftup.tech/checklists/ai-vendor-evaluation-checklist/ (Liftup Technologies)
