We're building a tool that reads a Confidential Information Memorandum, extracts the financials, and estimates a valuation range — the kind of first-pass analysis a buyer would normally pay an analyst to do. Along the way, we ran an experiment we'd recommend anyone building with AI try before they trust their own output: we fed the exact same CIM into two separate, unrelated test deals and compared what came back.
The results were uncomfortable, and they taught us something worth sharing.
The setup
The document was a real CIM for a 56-year-old security alarm company — call it a hybrid business, with a recurring monitoring-revenue base and a lumpier, project-based installation side. Four years of P&L data showed net income swinging hard: a strong year, two mediocre years, and a near-zero year at the end.
We ran our valuation engine on this CIM in one deal. Separately, in an unrelated deal, we ran it again on the identical document.
What came back
Both runs identified the same business model, the same core risks (a large chunk of revenue tied to government contracts that typically require consent to reassign on a change of ownership), and the same recommended deal structure. That part was reassuring — the qualitative judgment was stable.
The numbers underneath it were not. One run anchored its earnings estimate on a two-year average and landed around $1.6–2.0M. The other anchored on the single most recent (and worst) year and landed around $1.2–1.5M. Same document. Same model. Two answers roughly $500K apart at the low end.
Worse: the arithmetic paths were genuinely different, not just differently worded. One run applied an ad hoc "adjustment to avoid double-counting" that the other didn't apply at all. One used a different revenue base for its sanity-check calculation than the other. Neither was obviously wrong — both were internally consistent, well-reasoned, and confidently stated. They just didn't agree with each other.
Why this happens
Large language models are stochastic by design — asked to reproduce a piece of open-ended reasoning, they'll take a slightly different path each time, even from the same starting point. That's fine for a lot of tasks. It's not fine for something you're about to use to decide what to pay for a business.
The real problem, though, wasn't randomness. It was architecture. We had asked the model to do two fundamentally different jobs in one generation: make the judgment calls (which year to weight, what to add back, which multiple applies) and do the arithmetic to combine them into a final number. Bundling those together meant that small differences in how the model framed its reasoning cascaded into different calculation structures. A model doing genuine analytical work will phrase things differently each time — that's not a bug. Letting that variation leak into the actual math is.
The fix
We separated the two jobs. Now the model's only job is to output the judgment calls as specific, named numbers — this multiple, that add-back, this revenue base for the cross-check — not to compute a final answer. A fixed function in our own code takes those numbers and combines them the same way, every time. No model-generated arithmetic touches the final figure at all.
We went further on the single biggest swing factor we'd found: which year (or years) to anchor the earnings estimate on. Instead of leaving that as an open judgment call, we replaced it with a disclosed, standard technique — a weighted average of the three most recent years, weighted 3:2:1 toward the most recent — computed directly from the extracted figures, not chosen by the model at all. We did the same for the owner-compensation add-back: instead of the model picking "midpoint for two owners" one time and "high end for one owner" the next, it now just extracts the stated facts (the range, the headcount), and one formula turns those into a number.
What actually changed when we re-ran the test
We ran the same CIM through the fixed version, again in two separate deals. The anchor figure came back identical to the dollar, both times. The owner-comp add-back came back identical, both times. The remaining variance — which multiple within a defensible range, exactly how much profit to attribute to the recurring-revenue book — shrank from a five- or six-figure swing to something in the low tens of thousands, and every bit of it was now visible and attributable to a specific, named input instead of buried inside two different narratives.
That's the honest bar to hold an AI-generated valuation to. Not "does it sound confident" — it will, every time. The bar is: if you show it the same facts twice, does it give you the same facts-based answer twice, and can you actually see which parts are computed versus which parts are judgment? If the answer to either is no, the tool isn't ready to be trusted with a number that has real money attached to it.
We'd rather find that out ourselves, on a test document, than have a buyer find out on a real one.
If you're evaluating AI tools for anything involving a number you're going to act on — a valuation, a pricing model, a risk score — this is worth testing yourself before you trust the output: run the same input twice, on two separate instances, and see if you get the same answer for the same reason. If you don't, ask why not.