How to choose an AI development partner: an enterprise checklist

Vendor decks have converged. Everyone has the same logos of the same model providers, the same "agentic workflow" diagram, and the same claim about shipping fast. None of it discriminates. The questions below do, because they are hard to answer well without having actually run something in production.

Part 1 — Evidence of production, not demos

"What does your last shipped system do when the model gets it wrong?"

This is the single highest-signal question you can ask. A team that has only built demos will talk about prompt tuning and model quality. A team that has shipped will describe specifics: the confidence threshold, where low-confidence output goes, who reviews it, how the reviewer's correction gets back into the system, and what gets logged for audit.

There is no way to fake a good answer here. Ask it first.

"How do you know a change made things better?"

Prompts, models, and retrieval strategies get changed constantly. Without an evaluation set, every change is a guess and regressions are invisible until a user complains. Ask to see the shape of an eval harness from a real project: how many cases, who labelled them, what the rubric is, whether it runs in CI.

"What would you refuse to build with AI?"

Teams with judgement have a ready list — usually things where the error cost is high and the error rate is irreducible, or where a database index and some rules would do the job for a fraction of the cost. Teams selling hours do not have a list. This question is a cheap integrity check.

Part 2 — Who you actually get

• Ask who will be on the weekly call, by name, and whether that person writes code. Sales engineers who disappear after signature are the oldest problem in this business.
• Ask the team size on your project and how many other projects each person is on simultaneously.
• Ask about key-person risk directly. Small teams carry it; good ones have an honest answer about documentation, handover, and what happens if someone leaves mid-build.
• Ask who owns the code and the model artefacts at the end. Get it in writing before work starts.

Part 3 — Data, security, and where things run

For most enterprise buyers this is where the deal actually lives or dies, and it is worth resolving before you get attached to a prototype.

1. Where does our data go? Name every third party in the path, including the model provider and any observability tooling.
2. Is our data used to train anyone's model? Get the specific contractual term, not a reassurance.
3. Can this run in our own cloud tenancy, or on-premises, if compliance requires it? What breaks if it has to?
4. What is retained, where, and for how long — prompts, outputs, embeddings, logs?
5. Which regulatory regimes have you actually delivered under, and can we speak to that client?

If your sector puts hard constraints on where inference can happen, resolve it in week one — it changes the architecture, not just the paperwork. We wrote about the practical shape of that in private and on-premises LLM deployment.

Part 4 — Commercial shape

The commercial structure tells you what the vendor is optimising for. Open-ended time and materials rewards duration. A fixed scope per milestone rewards shipping. Neither is inherently dishonest, but they pull in different directions and you should know which one you have signed.

• Is the scope defined in writing before work starts, and what is explicitly out of it?
• What is the smallest useful engagement — can you buy one real slice before committing to a programme?
• What are the ongoing costs after launch: inference, hosting, monitoring, and whose budget they land in?
• What happens commercially if the prototype proves the use case does not work? A partner confident in their judgement will have a graceful answer.

The best signal in a vendor process is a partner willing to tell you, early and in writing, which parts of your idea will not work.

Part 5 — Buy a small thing first

No amount of diligence substitutes for watching a team work. Scope one narrow, real slice of the problem — one document type, one workflow, one queue — with a defined deliverable and a short timeline. You will learn more about communication, judgement, and code quality in three weeks of real work than in three months of procurement.

Pick a slice that is genuinely representative rather than the easiest one. The point is to surface the hard parts early, while switching costs are still low.

The short version

1. What does your last shipped system do when the model is wrong?
2. How do you know a change made things better?
3. What would you refuse to build with AI?
4. Who is on the weekly call, and do they write code?
5. Where does our data go, and is it used for training?
6. Can we buy one real slice before the programme?

If you want to run this checklist against us, the fastest route is to tell us the problem in a couple of sentences — we will tell you honestly whether it is a good fit before anyone writes a proposal.

• buying AI
• vendor selection
• due diligence
• enterprise