Enterprises poured $30 to $40 billion into generative AI, and MIT's 2025 NANDA report — "The GenAI Divide" — found that 95% of those pilots delivered no measurable impact on the P&L. The standard reading of that number is that we are in a bubble, the hype outran the tech, and generative AI was oversold. That reading is wrong. The technology is fine. The frontier models are extraordinary and getting cheaper by the quarter. What got bought, in 95 cases out of 100, was never a solution. It was a demo with a budget line.
The demo is the problem, not the model
Here is the conventional recipe for a generative AI solution, the one repeated in every pitch deck: pick a model, write a clever prompt, wrap a chat box around it, ship. That produces a demo. A demo proves the model can do the thing once, on a clean input, in front of an audience that already wants to believe. A solution does the thing ten thousand times a day, on inputs nobody anticipated, when the model is having a statistically bad afternoon, with no human standing by to quietly fix the bad ones before anyone notices. The 95% failure rate is the entire distance between those two sentences. MIT was explicit that the barrier is organizational and integration-shaped, not a model limitation — they call it the learning gap. The model was rarely the thing that failed. The thing around the model was never built.
The model is the cheap part
The lazy consensus treats model choice as the decision. The endless GPT-versus-Claude-versus-Gemini bake-offs, the benchmark screenshots, the breathless threads about who topped a leaderboard this week — all of it frames the central question as which model. That is the least important decision you will make. Frontier models are a commodity you rent by the token, and swapping one for another is an afternoon of work, not a re-architecture. The work that actually constitutes a solution is the unglamorous 95% nobody puts on the demo reel: getting your real, messy data into a form the model can retrieve against, building an evaluation harness that tells you when output quietly got worse, guardrails for when it goes off the rails, fallbacks for when the API is down, and the workflow the whole thing has to plug into without breaking what already works.
I run a boutique studio with a Python and AI-ML practice that has been shipping production generative AI pipelines for the past couple of years, and the pattern is consistent: the LLM integration is almost never where the weeks go. The weeks go into wiring retrieval over a client's data that was never organized for retrieval, and standing up evaluation and observability — the Langfuse and LangSmith layer — so we can prove a change made the system better instead of hoping it did. We treat the AI coding tools the same way, as reviewed accelerators, not crutches. The model is an ingredient. The solution is the kitchen.
Buy-versus-build is a red herring
The same MIT report found that purchased and vendor-built tools succeeded roughly twice as often as internal builds — about 67% versus a third of that. Every vendor on earth now quotes that line as "so buy from us." That is a misread of their own favorite statistic. Vendors do not win because they hold a secret model; they rent the same models the internal team does. They win when they ship the boring engineering the internal team skipped — the integration, the evals, the iteration loop, the second senior pair of eyes on every change. The real axis is not buy versus build. It is engineered versus improvised. An internal team that builds the system like a product, with code review and an eval suite, beats a vendor selling a thin wrapper every time. We require every pull request to be reviewed by at least one other senior engineer before it merges, and a generative AI feature is not exempt from that — it is production software that happens to call a probabilistic API, and the probabilistic part is exactly why it needs more discipline, not less.
Where the money goes is not where the value is
MIT also found the budgets concentrated in sales and marketing, where the ROI was lowest, while back-office functions like customer service and operations quietly delivered the better returns. That is the whole pathology in one sentence: organizations fund the demo that is visible, not the workflow that moves a number. The flashy copilot that wows a board meeting gets the budget. The narrow, unsexy automation that actually saves a team six hours a week does not. The solutions that survive are narrow on purpose. On a rescue engagement we picked up — a stalled build called Meal4U — we spent a week deconstructing scope and re-estimating it close to ten times, cutting thirty to fifty percent on each pass until we found the smallest thing that could actually ship. That discipline matters double for generative AI. A narrow agent that does one workflow reliably is worth more than an everything-copilot that demos beautifully and ships never.
What a generative AI solution actually is
So define the thing honestly. A generative AI solution is a software system that happens to call a model. It has retrieval over your own data, usually a vector store like pgvector or Qdrant, because the model does not know your business and pretending otherwise is how you get confident nonsense. It has an evaluation harness, so a prompt tweak does not silently degrade output you only notice three weeks later from an angry customer. It has guardrails and a human in the loop wherever a wrong answer costs real money. It has monitoring, because a probabilistic system fails quietly, not loudly. It has document processing, or agents that take real actions in real systems, depending on the job. None of that is on the demo reel, and all of it is the difference between the 5% and the 95%.
Yes, the number is contested — it doesn't matter
Critics have picked at the MIT figure, and fairly: it leaned on a modest set of interviews and a narrow definition of success — direct P&L impact inside six months, which ignores softer gains like efficiency and churn. Grant all of it. Say the real failure rate is 80% instead of 95%. The position does not move an inch, because the failure mode is identical at either number. Whether four out of five pilots stall or nineteen out of twenty, they stall for the same reason: someone shipped a demo and called it a solution, and the gap between the two does not care which study you cite.
The strategy-versus-shipping split runs through all our AI writing; what AI consulting really is states it plainly. Our AI engineering services describe the shipping half.
Build it like software, or don't build it
Here is the position, with no hedge. The companies in the 5% are not the ones with privileged model access — everyone has the same models, that is the entire point of a commodity. They are the ones who treated their generative AI solution as an engineering problem from day one: scoped it narrow, wrapped it in evals, reviewed the code, and integrated it into a workflow people actually use instead of a chat box they admire and abandon. The model was never the moat. The engineering discipline around it was. So the next time someone pitches you a generative AI solution and ninety percent of the conversation is about which model and ten percent is about your data, your evaluations, and your workflow, you are being sold a demo. Walk.