Retrieve or Compute: The Read That Decides Whether AI Helps or Lies

A while ago a business owner came to see me, genuinely excited. He had been at a presentation where someone demonstrated an AI tool, and it had stayed with him. The presenter typed in an ordinary question, the tool answered in two seconds, and it even pointed back to the document the answer came from. He wanted the same thing for his own company, and he wanted it now. "I just want to be able to ask my business anything," he said.

I could have taken the brief and built him what he described. Instead I did the thing that actually earns the fee. Before we agreed on any solution, we ran a quick diagnostic on the questions he wanted answered - not the tool he was asking for, but the job he was hiring it to do. So I asked him to list them, the real ones, the questions that would change a decision.

Whether one client had quietly grown into too large a share of his revenue. How many months of payroll he could cover if his three biggest customers paid late. Whether last year's margin gain was real, or one good project flattering the average. Good questions, every one. And as he spoke I felt the familiar drop, because almost none of them were the kind the tool from his demo could actually answer.

That gap is the whole reason a client needs a consultant and not a vendor. He had described his need in the language of the solution he had just seen - "an AI that answers anything." Our job is to hear the need underneath the request, and tell him the truth about whether that solution delivers it. Here it would not, and I did not have to guess. I knew at a glance, because we had already built the exact thing he was describing, for our own product, and watched it fail in a way that cost us months. More on that scar shortly. First, what the diagnostic actually found.

A client describes the solution they saw in a demo. Our job is to diagnose the need underneath it - and say so, even when it is not what they asked for.

What the diagnostic found: two kinds of question

You can sort a client's questions a hundred ways - by department, by who is asking, by how urgent they feel. Forget all of them. Only one cut decides whether AI can answer honestly, and it is this: where does the answer live, and what has to happen to produce it? Run every question through that single lens and it lands in one of two piles.

The first pile is find-it questions. The answer already exists, written down somewhere, word for word. Nobody has to work anything out; it just has to be located and read back. In his business they sounded like:

• What did we commit to in the service-level agreement?
• Which risks were flagged in the last board pack?
• What did this client actually ask for in that long email thread?
• What does our policy say about carrying leave into the next year?

For this pile the right method is retrieval - the very technology that impressed him at the demo, retrieval-augmented generation, or RAG. The AI understands the question, searches the documents, finds the passage, and reads it back with the source attached. It is AI from end to end, and for find-it work it is genuinely superb.

And I want to be fair to RAG, because pointed at the right job it is the best tool there is. A customer-support assistant answering from your help centre. A new hire querying the staff handbook instead of interrupting a colleague. A lawyer pulling the one relevant clause out of a two-hundred-page contract. That is all pure find-it work: the answer is a passage that already exists, and RAG's whole genius is locating it and handing it back with the receipt. For that, nothing beats it.

But the moment the answer is a number that has to be calculated - a margin, a runway, a concentration - RAG is not merely weaker, it is the wrong instrument by definition. There is no passage to retrieve, because the answer was never written down anywhere. So a retrieval system does the only thing it can: it finds text that looks relevant and improvises a number around it. The exact machinery that makes it flawless on the handbook makes it dangerous on the spreadsheet.

The second pile is work-it-out questions. The answer exists nowhere yet. No document contains it. It has to be computed from the client's own messy numbers. His list was full of them:

• Is any single client more than a third of our revenue?
• How many months of cash do we have left at this burn rate?
• Did our margin actually improve, or did one big one-off flatter it?
• What does it cost us to serve the smallest twenty percent of accounts?

Here the right method is not the language model writing an answer. It is deterministic computation - pull the exact figures out of the actual tables and run the actual arithmetic. And this is the part people get wrong, so let me be precise: this is still an AI solution. It is not "AI versus a spreadsheet." It is AI understanding the question and working out which tables hold the inputs, then a deterministic calculation - ordinary, boring, repeatable arithmetic, no AI in it - doing the actual sum, then AI again explaining the result and citing the cells it used. AI on both ends, a real calculation in the middle. The difference between the two methods is not whether AI is involved. It is whether a genuine computation sits in the centre, or the AI is left to fake one.

Now run your own diagnostics through that lens. Concentration, runway, real margin, cost-to-serve, the line quietly losing money - almost every question that actually drives an engagement sits in the second pile. That was the finding for this owner too. He had been sold a find-it tool, and the questions keeping him up at night were almost all work-it-out. The diagnostic did not just pick a technology for him. It told him the truth about the one he had fallen for.

How we knew: we had already built it wrong

I said I knew at a glance. Here is why, and it is not a flattering story.

When we set out to build the engine behind our own product - the part that takes a client's spreadsheets and answers questions from them - the first version was, naturally, a RAG engine. Ingest every file, index it, let the model retrieve the relevant bits and answer. On find-it questions it was lovely. It could tell you what a tab was called, what a contract said, where a figure appeared. The demo sang, exactly like the one that had hooked our owner.

Then we pointed it at the questions that matter. We asked it for the gross margin in a year where the files held revenue but no cost lines. A human analyst stops there and says "I cannot compute that, the cost data is missing." The retrieval engine did not stop. It found two plausible same-currency numbers, divided one by the other, and handed back a margin. Clean, formatted, sourced-looking, and complete fiction. Not a number that was slightly off - a number that had never existed, delivered in precisely the confident tone the tool uses when it is right.

That was the moment it landed. It was not a tuning problem, not a prompt I could sharpen. It was the wrong method aimed at the wrong kind of question, and no amount of cleverness would fix it, because retrieval finds what exists and a margin nobody has calculated does not exist to be found. So we did the expensive, clarifying thing. We deleted the retrieval answer engine - around nine and a half thousand lines of it - and rebuilt the whole system on one rule we now treat as law.

AI never does the math. Every number the client sees comes from a specific stored cell, or a deterministic calculation over stored cells. The AI understands, matches, and explains. It never calculates.

That rule has a running order, and it is the single most useful thing in this whole piece. The instinct - for a person, for a junior, for nearly every tool on the market - is to reach for retrieval first, because it is the part that works and the part that wins demos. For a find-it question, fine. For a work-it-out question, retrieval-first is precisely how you manufacture a confident wrong number. So you invert the instinct: decide what kind of question it is, compute it deterministically when it needs computing, and fall back to retrieval only when the answer genuinely is written down. Compute first. Retrieve second, as the backstop, never the default.

Take his concentration question - is any single client more than a third of our revenue? Retrieval-first finds a sentence saying "Client A is our largest account," wraps a confident reply around it, maybe even pins on a tidy "about 35 percent." Nothing was added up; the number is decoration. Compute-first recognises a calculation is owed: it pulls every client's billings, sums them, divides, and answers "Client A is 41 percent of last year's revenue - EUR 120,000 of EUR 293,000," with the figures shown underneath. And if a quarter is missing from the data, it says so, instead of guessing. Working shown, or an honest "I cannot answer that." Never a number in the gap between.

And the rebuild was not a quality compromise, it was an upgrade. A 2026 benchmark put the same model on the same business questions two ways: answering straight off the raw tables it was right about ninety percent of the time; routed through a deterministic layer that did the math it climbed past ninety-eight. But the ten points were not the prize. The prize was that the failures changed character. The raw-table mistakes were silent and confident. The deterministic ones became honest refusals - "I cannot answer that from your data" - which is the only kind of failure you can survive in front of a client.

A thirty-second test you can hand anyone

You do not need to see the architecture to vet a tool, or a junior's workflow. You need two questions and half a minute. Ask it a number you already know cold - last year's revenue, your biggest client's share - and watch whether it lands exactly and shows where it pulled the figure from. Vague, rounded, or sourceless is a red flag even when it is close. Then ask it for a figure that is genuinely not in the data - a month never loaded, a metric nobody tracks - and watch what it does. A sound process tells you it does not have that. A dangerous one invents something to fill the silence. That second answer is the entire risk in miniature, and you would far rather trigger it on purpose at your desk than meet it in a client's boardroom.

It was a diagnosis all along

The reflex takeaway is "AI is unreliable," and it is wrong, and it will age badly - the model in that benchmark was right ninety percent of the time. The lesson is not that AI cannot be trusted with a client's numbers. It is that the value was never in the tool. It was in the read we made before any tool ran: which kind of question is this, and therefore which method has to answer it.

That read is exactly what the owner could not do for himself, and it is exactly what we did in the first twenty minutes of meeting him. We did not pick a technology and sell it to him. We diagnosed his questions, told him which ones a retrieval tool would answer cleanly and which ones it would quietly lie about, and pointed each to the method that could answer it honestly. The tools are new. The instrument is the oldest one we have: get the question right before you go looking for the answer.

Two questions for the comments. Where has a confident, well-formatted, completely wrong number nearly made it into a deliverable on your watch - and what caught it? And as more of your analysis runs through AI, how are you drawing the line between the questions you let it retrieve and the ones you insist on computing yourself?