Gartner expects roughly 30% of generative-AI projects to be abandoned after proof of concept by the end of 2025, and the reasons it lists are not exotic: poor data quality, escalating costs, and unclear business value. None of those are model problems. Yet walk into almost any RAG consulting pitch and you'll hear about models, embeddings, and which framework — LangChain or LlamaIndex — they'll standardize on. That's the tell. The pitch is selling you the half of the problem that was already solved, and staying very quiet about the half that actually kills projects.
The conventional pitch, and why it's backwards
The standard RAG consulting narrative goes like this: point a large language model at your documents, add a vector database, wire up retrieval, and now your company has a chatbot that knows everything. The deliverable in the deck is a slick demo where someone asks a question and gets a paragraph back with citations. Everyone nods. The engagement gets approved on the strength of that demo. Then it goes to production and starts confidently making things up, and nobody can explain why.
The reason it's backwards is structural. RAG stands for retrieval-augmented generation, and the consulting industry has fallen in love with the G. The G is the easy part. Generation is a commodity you rent by the token from OpenAI, Anthropic, Google, or an open-weight model you host yourself. Swapping one for another is an afternoon. Embeddings are nearly free — current embedding models run around two cents per million tokens at published API prices, which is to say the part everyone obsesses over costs less than the coffee in the kickoff meeting.
It's a retrieval problem wearing a generation costume
Here is the thing the demo hides: a language model can only reason over what you hand it. If retrieval surfaces the wrong three chunks, the model doesn't shrug and say it doesn't know. It writes a fluent, well-formatted, completely wrong answer over the garbage you gave it. The generation step launders bad retrieval into something that looks authoritative. That's why RAG fails silently — the output never looks broken, it just is.
So the real engineering question was never 'which model.' It's whether, for any given question, the system reliably pulls the right passages out of your corpus before the model ever sees them. That is a retrieval problem, and retrieval quality is governed by deeply unglamorous things: how your documents are chunked, whether your PDFs parsed into clean text or into ligature soup, how stale your data is, whether you have metadata to filter on, whether you're doing hybrid keyword-plus-vector search or just cosine similarity and hope. None of that fits on a slide.
Where engagements actually die
In practice, RAG projects die in three predictable places, and none of them is the model. First, the data: a corpus full of duplicate policy documents, three contradictory versions of the same spec, and scanned images with no text layer. Second, chunking: split a contract every 500 tokens and you guillotine the clause that answered the question. Third — and this is the one that separates real work from theater — the absence of any way to measure whether retrieval is getting better or worse as you change things.
That third one is the whole ballgame. If a team is tuning a RAG system by asking it a few questions and eyeballing whether the answers 'feel right,' they are not engineering, they are doing improv. You cannot improve what you don't measure, and 'it seemed good in the demo' is not a measurement. The projects that survive have a ground-truth question set, retrieval metrics like recall and precision at k, and a harness that runs every time someone touches the pipeline. The ones that get abandoned after the PoC don't.
The eval harness is the deliverable that matters
If you are buying RAG consulting, the single most useful question you can ask is: 'How will you measure retrieval quality, and how will I see that number change over the engagement?' A serious answer involves a curated set of representative questions with known-correct source passages, an evaluation suite that scores retrieval and answer faithfulness, and observability so you can trace a bad answer back to the chunk that caused it. A weak answer involves the words 'you'll see, the results are really impressive.'
At EltexSoft our generative-AI practice has spent the last couple of years building production AI pipelines, and the pattern that holds across all of them is boring and non-negotiable: the eval harness gets built early, not bolted on at the end. We lean on tooling like Langfuse, Phoenix, and LangSmith for tracing and scoring, and on vector stores like pgvector, Qdrant, and Weaviate chosen to fit the data rather than the other way round. The model choice is close to the last decision we make, because it's the cheapest one to change and the easiest one to reverse. That ordering is the actual expertise. The model is interchangeable; the discipline isn't.
The best RAG consulting sometimes tells you not to do RAG
There's a version of honesty the industry is structurally bad at, because the incentive is to sell the engagement: a good RAG advisor will sometimes tell you your data isn't ready, or that you don't need RAG at all. Plenty of 'we need a RAG system' problems are actually 'our search is bad' problems, or 'our documentation contradicts itself' problems, or 'a well-structured FAQ would have answered 80% of this' problems. Adding a language model on top of broken retrieval doesn't fix any of those. It just makes the broken answers more persuasive.
We run engagements deliberately to surface this early — a free discovery week and a paid pilot with no lock-in, specifically so the question 'is RAG even the right tool here, and is your corpus in a state where it can work' gets answered before anyone signs a long contract. The same scope discipline applies once a build is underway: every change goes through review by a second senior engineer, which is mostly how the quiet retrieval regressions get caught before they ship. A consultant whose process can't tell you 'don't build this yet' is a vendor, not an advisor.
What you're actually paying for
Strip away the model worship and RAG consulting is worth real money for a short list of concrete things: getting your corpus into a clean, deduplicated, parseable state; designing chunking and metadata that match how your documents are actually structured; building hybrid retrieval that doesn't fall over on acronyms and exact-match queries; and standing up the evaluation and observability that let you trust the system in front of customers. For context on cost, a small dedicated team in our model runs in the range of $25,000–$55,000 a month, and fractional CTO-level oversight in the $4,000–16,000 range — and on a sane engagement most of that spend lands on data and retrieval, not on prompt-wording.
For the category-level version of this argument, what AI consulting really is is the place to start. Our AI engineering services cover the engineering side we actually sell.
The position, stated plainly
Don't hire someone to 'add AI to your documents.' Hire someone to fix your retrieval and prove it with numbers. The model is a rounding error in your budget and a one-line change in your config; the data pipeline and the eval harness are where the months go and where the value is. If a RAG consultant opens with the model, the framework, or the demo, walk. If they open with your data quality and how they're going to measure whether retrieval is working, you've found the rare one who actually knows where the project lives or dies. The G was never the hard part. It was always the R, and pretending otherwise is exactly how a third of these projects end up abandoned in a slide deck nobody reopens.