Is AEO Built on Sand? The Probabilistic-Engine Question

AI engines are probabilistic. Ask the same question twice, get different answers. We've seen it directly, inside our own pipeline: a cybersecurity company scored 86 on one run and 36 on the next, on the identical input, the same evening. A data-infrastructure company named one competitor as its closest rival on one run and a completely different one — a different category of competitor entirely — on the next.

So the honest question a sophisticated buyer should ask is: can AEO ever actually work, or is the whole discipline built on a foundation that shifts every time you measure it?

The question splits into two, and they feel identical but aren't

1. Can AI engines be influenced at all — is there a stable relationship between what a brand does and how AI describes it?
2. Can that influence be measured precisely enough to sell as a product?

Our own data answers the first question with a clear yes. It answers the second with a qualified no — not with a single number. Those are very different conclusions, and the difference is the entire business.

Why the answer to the first question is yes

Across every repeat test we've run, two things held steady while a third did not.

Category held. A well-established endpoint-security company was identified as a category leader on every run. A company with a genuinely thin AI-training footprint scored as a near-ghost on every run — correctly; it's a real company with almost no AI-visible presence. A cloud-security company was consistently described as present in its category but outranked by a specific rival, run after run. A data-infrastructure company was consistently described as strong but not dominant, run after run.

Direction held. Whether a brand was winning, losing, or absent from its category's AI-generated answers was consistent across runs, even when the precise numbers weren't.

What swung was the second decimal — the exact score, and in one case the exact named competitor. The data-infrastructure company's top competitor flipped between two genuinely different rivals: a hyperscale cloud platform on one run, a specialized workflow-orchestration tool on the next.

If AEO were built on nothing — if the engines were pure noise — repeat runs would produce garbage that changed completely each time. They don't. The probabilistic layer sits on top of a real, stable underlying signal.

Why this isn't a flaw: probabilistic doesn't mean random

The engines aren't malfunctioning when they return different answers. They're sampling from a genuine, multi-faceted truth. When a data-infrastructure company comes back with a hyperscaler as its top competitor on one run and a workflow-orchestration tool on the next, that's not noise — the company genuinely competes with both, at different layers of the stack. The variance is the engine surfacing different true facets of a real competitive position, not inventing one.

And critically: the shape of that distribution is governed by exactly the things AEO acts on — citation density, entity authority, structured content, third-party coverage. A brand with deep, authoritative content has its probability mass concentrated: it shows up reliably, run after run. A brand with thin content has that mass diffuse: it shows up sometimes, buried other times. Moving that distribution — concentrating the mass — is precisely what AEO does. The goal was never to make a random process deterministic. It's to shift where the probability mass sits. That's a real, achievable, measurable thing.

The honest hard part: a single measurement is a weak instrument

The variance means one measurement is noisy, and presenting a single number as precise is dishonest. That's not a foundation problem. It's a measurement-design problem, and it has a known solution: measure the distribution, not the point. Run more queries, run repeatedly, average over time, and the noise collapses into a stable trend.

This isn't exotic. It's how every measurement of a probabilistic system works. A poll doesn't ask one voter — it samples thousands and reports a distribution with a margin of error. Climate isn't one day's weather — it's the trend over time. Our free audit is one voter on one day: directional, noisy, and honest about being exactly that. The paid engagement is the poll: enough samples, over enough time, to report a stable position with a known confidence band.

The variance doesn't undermine the product — it defines the line between the free tier and the paid one. The noise is the reason the paid product has to exist.

Verdict: a sound foundation, with a nameable growing pain

This isn't a good idea on a shaky foundation. It's a good idea on a sound foundation, working through the growing pains of a specific, solvable measurement problem: we built a single-snapshot instrument and discovered that single snapshots are noisy. That's a discovery about how AEO must be measured — not about whether it works. Most players in this category haven't reckoned with this yet, and are selling single scores as though they were precise. Knowing the difference puts us ahead of that, not behind it.

The real risks, named without flinching

Measurement-cost risk. Collapsing variance requires multiple runs per measurement — more API spend, more pipeline load. The honest version of this product costs more to deliver than a single-shot version would. A margin question, not a foundation question.

The efficacy risk — the one genuinely unproven thing. Everything above establishes that the signal is real and measurable. It does not yet establish that our interventions reliably move it within a predictable timeframe. That is the actual open question: when we do entity and citation work, does the distribution shift, measurably, inside an engagement window? Our own case study is one data point that it can move — a same-day Perplexity citation shift after a day of focused entity work. One data point is not a pattern. The real test is longitudinal: take a real engagement, do the work, measure the distribution shift over eight weeks. That's the experiment that answers this question definitively, and it's already scoped.

Commoditization risk. Prompt-tracking itself is getting cheap. The durable asset was never the score — it's the longitudinal benchmark dataset and the AI-mediated-perception framing built across real engagements over time. A strategic risk, not a methodological one, and one we're actively navigating.

The one thing that would actually make this shaky

Resolving the variance by making the free single-shot score look more precise than it is — pinning it, hiding the run-to-run swing, presenting one number with false confidence. That's the version built on sand: a sophisticated buyer re-runs it and catches the gap between claim and reality immediately.

The version built on rock is the one we've chosen: claim what's reproducible, disclose what isn't, and let the longitudinal trend carry the precision claim instead of a single snapshot. That choice — made consistently, including when a more impressive-looking number would have been commercially tempting — is the actual foundation here. More than the engines. More than the methodology.

ChatRank measures five signals across AI engines and discloses exactly what's reproducible and what isn't in every engagement. See the methodology for the full framework, or the ChatRank case study for what this looks like on a real (and, in that case, our own) brand.