Last month I sat across a conference table from a CEO who’d just signed a $340,000 AI contract. Two months in, the platform was producing dashboards nobody trusted and predictions nobody acted on. He looked at me and asked the question I hear constantly: “How did I miss this?”
He didn’t miss anything. The vendor sold him a product he was never equipped to evaluate. And honestly? Most buyers aren’t.
I’ve spent 20+ years in enterprise transformation, first at IBM, then at Kyndryl, leading global teams of 200+ engineers across some of the most complex AI and infrastructure programs you’ll find. I now run an AI consulting practice with multiple active clients, and a meaningful chunk of my work is cleaning up after vendors who oversold and underdelivered.
So I want to give you the questions I’d ask if I were sitting in your seat. Not generic procurement questions. The specific ones that, when a vendor can’t answer them clearly, tell you to walk away.
Why Most AI Vendor Conversations Go Sideways
Here’s the uncomfortable truth: most AI vendors are excellent at demos and average at delivery. The demo runs on curated data, in a sandbox environment, with a presenter who knows exactly which buttons to press. Your business is none of those things.
The gap between “the demo worked” and “the platform produces value in your environment” is where projects die. Quietly, Expensively, and almost always before AI is even the problem. This is because the foundation underneath was never ready.
That’s why I tell every client the same thing: AI projects fail before AI is ever introduced. The questions below are designed to expose that gap before you sign, not after.
Question 1: “Walk me through a deployment that failed. What went wrong?”
This is my favorite opening question, and it tells you more than any case study deck.
A confident vendor will give you a real answer. They’ll talk about a client whose data wasn’t where they thought it was, or a security review that surfaced gaps mid-implementation, or an integration that took three times longer than scoped. They’ll tell you what they learned and what they do differently now.
A vendor who claims they’ve never had a failed deployment is either lying or hasn’t done enough deployments to matter. Both should disqualify them.
What you’re really testing: intellectual honesty and pattern recognition. The best vendors have scars. The dangerous ones have only PowerPoints.
Question 2: “What does my infrastructure need to look like before this works?”
If the answer is “don’t worry, our platform handles all of that” … slow down.
Every AI system has assumptions baked into it. Assumptions about data quality, network throughput, identity management, monitoring, and integration patterns. When those assumptions don’t match your reality, the platform doesn’t fail loudly, it just quietly produces worse and worse outputs until someone notices six months in.
A serious vendor will ask you about your data sources, your security posture, your existing observability, and your integration architecture before they’ll even quote you. If they don’t ask, they don’t care and you’ll pay for that lack of curiosity.
Question 3: “Who owns the model after we deploy it and what happens if I leave?”
This is where vendor lock-in lives, and most buyers don’t read closely enough to catch it.
Ask specifically: Who owns the trained model weights? Who owns the embeddings? Who owns the prompts and the orchestration logic? If you exit the contract, what do you walk away with and in what format?
I’ve seen contracts where a client paid millions to fine-tune a model on their own proprietary data, and the vendor’s exit clause meant they walked away with nothing but a CSV of inputs. The intelligence they paid to create stayed with the vendor.
Get this in writing. Before signing.
Question 4: “What’s your data isolation architecture, and can you draw it on a whiteboard?”
I love this one because it separates vendors who actually understand their own platform from the ones reading off a sales sheet.
Ask them to diagram, in real time, how your data flows through their system. Where does it sit at rest? Where does it move in transit? Who has access to it, and how is that logged? If their model is fine-tuned on your data, is your data ever used to influence outputs for other customers? Ever?
If they can’t draw it, they don’t know it. And if they don’t know it, your CISO is going to have a very bad day six weeks into the project.
Question 5: “What’s the total cost of ownership over three years including everything you’re not putting in this proposal?”
The proposal in front of you is an iceberg, and what’s above the waterline is rarely the expensive part.
Ask for the realistic three-year total: licensing, implementation, infrastructure costs (which are almost never included), integration work, ongoing model retraining, monitoring tools, the headcount you’ll need internally to support it, and the inevitable scope expansion in years two and three.
In one engagement at Kyndryl, I documented over $2M in savings for a client largely by surfacing the hidden ongoing costs of a tool stack the prior vendor had quietly normalized. Hidden costs are the rule, not the exception. Make the vendor put them on paper.
Question 6: “How do you measure success and what specifically happens if we don’t hit those metrics?”
Vague success criteria are how vendors stay employed indefinitely while delivering nothing.
Push for concrete, time-bound, business-outcome metrics. Not “model accuracy of 92%.” Something like: “incident detection improves by X% within Y months, measured against the baseline we’ll establish in week one.” On one Kyndryl engagement, we delivered a 38% improvement in incident detection and we knew it because we’d defined the baseline before we started.
Then ask the harder question: what happens if we miss? Is there a true-up clause? A reduction in fees? Or does the vendor just shrug and propose a phase two?
Question 7: “Show me a customer my size, in my industry, who’s been live for at least 18 months.”
Pilots prove nothing. Production at scale, over time, proves everything.
Most AI vendor case studies showcase month-three results which is the honeymoon period, when everyone’s still excited and the model hasn’t drifted yet. The real story shows up at month 12 and month 18, when data has shifted, edge cases have accumulated, and the people who originally championed the project have moved on.
If a vendor can’t put you in touch with a similar-sized customer who’s been in production for 18 months, you’re paying to be their case study. Charge them for that privilege, or pass.
Question 8: “Who’s actually going to do the work and how long have they been at your company?”
The pre-sales team is paid to win the deal. The delivery team is paid to deliver. They are almost never the same people, and the gap between them is where projects derail.
Ask for the names and titles of the people who’ll be on your account. Ask their tenure. Ask how many similar projects they’ve personally delivered. Ask if they’re contractors or full-time. Ask what the staffing plan looks like in month six, when half the original team has rotated to the next deal.
I’ve led 200+ engineer global teams. I can tell you with certainty: continuity of senior people on an account is the single biggest predictor of whether the engagement lands. Get the names. Lock them in if you can.
Question 9: “What would you tell me to do if I weren’t paying you?”
This is the one that breaks the mold, and it’s the one that tells you everything.
A vendor who’s there for a transaction will deflect or repackage their pitch. A vendor who’s there to build a real partnership will tell you the truth even when the truth is “you’re not ready for what we sell yet, and here’s what to fix first.” Those are the vendors worth signing with.
In my own practice, the most important conversation I have with a prospective client is the one where I tell them they don’t need our full implementation engagement yet. They need an Infrastructure Assessment first, because everything else is going to fail until the foundation is solid. That conversation costs me revenue in the short term. It earns trust that compounds for years.
The Real Test Behind All Nine Questions
Read these questions again and you’ll notice something: none of them are technical. Not really. They’re tests of how a vendor thinks, how they communicate, and whether they treat your business as a partnership or a transaction.
The technical evaluation matters too. But by the time you’re deep in technical due diligence, you’ve usually already decided emotionally. These nine questions are designed to surface the truth before you get there and when you can still walk away cleanly.
If a vendor handles all nine of these well, you’ve found someone worth working with. If they bristle, deflect, or oversell on three or more then believe what they’re showing you.
Where to Go From Here
If you’re evaluating an AI vendor right now, print this list and bring it to your next meeting. Watch what happens.
If you’re earlier than that and still trying to figure out whether your business is even ready for AI then start with the foundation. Most of the failures I see aren’t vendor problems. They’re infrastructure, data, and readiness problems that no vendor can fix for you.
That’s the work I do with every new client before we touch a single AI deployment. It’s also the conversation I’d rather have with you now than after a $340,000 contract has gone sideways.
Ready to evaluate where you actually stand?
→ Start with an AI Infrastructure Assessment — A clear, vendor-neutral readiness scorecard built on 20+ years of enterprise transformation experience.
→ See our full services — From assessment through implementation, infrastructure-first, no shortcuts.
→ Read more on the blog — Including “You Don’t Have an AI Problem. You Have a Data Problem.” and other practical reads for leaders evaluating AI seriously.
Russell Love is the Founder & CEO of Summit AI Business Solutions, based in Browns Summit, NC. With 20+ years of enterprise transformation experience at IBM and Kyndryl, Russell helps businesses build the foundations that make AI actually work.