# Model evaluation should look like product analytics

*Draft — long-form. Companion visual: [monitoring-map.html](monitoring-map.html)*

Academia has spent three years benchmarking medical language models with escalating rigor. HealthBench graded 5,000 conversations against ~48,500 rubric criteria written by 262 physicians. MedHELM built a clinician-validated taxonomy of 121 real clinical tasks across 35 benchmarks and published it in *Nature Medicine*. Stanford's ARISE network now runs MAST, a living leaderboard stitched from clinically-grounded benchmarks. This is serious, careful work — and almost none of it predicts what happens when a model is embedded in someone's working day. The scores tell you what a model *can* do in vitro. They say nothing about whether it gets used, acted on, or quietly ignored in vivo.

I build analytics for a clinical AI assistant — a copilot that drafts pre-visit summaries, answers point-of-care questions, and suggests documentation improvements inside the electronic health record. We track frontier-model benchmarks like everyone else. But after months of instrumenting the deployed product, I've come to a conviction that feels almost embarrassing in its plainness: **most of model evaluation should be product analytics.** Not "should be complemented by." Should *be*. The benchmark is the unit test; the deployment funnel is the exam.

## The literature is already conceding the point

You don't have to take a builder's word for it — the 2025–26 evaluation literature is converging on the same conclusion from the other direction.

Start with the measured gap. A December 2025 systematic review of 39 medical benchmarks in *JMIR* quantified what it called the *knowledge–practice performance gap*: models score 84–90% on USMLE-style examinations — at or above physician level — and drop to 45–69% on practice-based tasks, a 39–45 point fall the authors summarize by calling exam scores "insufficient and misleading proxies for clinical readiness." Bean et al. measured the same phenomenon end-to-end: a model at 95% in evaluation fell to 34% in deployment conditions. A CMU-led analysis argued the gap isn't primarily about model capability or benchmark quality at all, but about *implicit assumptions* embedded in evaluation protocols — how tasks are structured, who interacts with the model, how outputs translate into decisions — that deployment silently violates. Those three assumptions are, nearly verbatim, the exposure, engagement, and action layers below.

Then watch what the benchmark builders themselves are doing. OpenAI's follow-up to HealthBench — HealthBench Professional (April 2026) — abandons simulated scenarios for a set curated from 15,079 *real clinician conversations* with ChatGPT. ARISE's 2026 State of Clinical AI report states flatly that with "the saturation of traditional QA benchmark scores," the field needs evaluation on "multi-turn unstructured real-world data" and "prospective and post-deployment real-world scenarios." MedHELM's differentiator is real EHR data rather than exam questions. Benchmark design is asymptotically approaching the thing it was supposed to substitute for: watching real usage.

Meanwhile, the deployment side has scale the benchmarks will never have. OpenEvidence crossed one million physician consultations in a single day in March 2026 and reports use in ~27 million clinical encounters a month, with roughly 60% of queries being patient-specific decision questions — the unit of work, not the exam item. Doximity's 2026 physician survey has daily AI use jumping from 47% to 63% in a year, with 69% of users *saying* AI improved patient care — while 71% name accuracy as their top concern. Note what that last pair is: an outcome claim collected by survey, sitting next to a trust deficit. Self-report is what you get when nobody has built the reconciliation layer. The richest evaluative datasets in medicine right now are query logs and decision streams, and almost nobody is treating them as evaluation.

Here's the argument, made concrete.

## The unit of analysis is not the prompt

Benchmarks score model outputs one prompt at a time. But a deployed assistant doesn't experience the world as prompts — it experiences it as *units of work*. In our case, the unit is a patient visit. The model generates a summary days before the visit; a clinician may or may not open it before walking into the room; they may chat with the assistant during the encounter; they accept or reject its suggestions, sometimes days later; and the actual verdict — did any of it matter — is written into the EHR by other hands over the following weeks.

The moment you adopt the unit of work as your unit of analysis, evaluation stops looking like a test set and starts looking like a funnel laid out on a clock. Everything we've learned falls into six layers along that clock, and every layer has a failure mode that benchmark thinking cannot see.

## Layer 1: Delivery — the supply side no benchmark has

Consumer analytics has no concept of "content supply" because users generate the content. An embedded assistant inverts this: the *product* pushes work to the user. Which means generation can silently break — for one clinic, one integration, one patient segment — while every usage dashboard still looks alive, because the users who do get output keep using it.

We learned to watch items generated per day, per clinic, per clinician; the pipeline backlog by status; and "last generated" per user. The most important early-warning chart in our stack is one no evaluation paper would recognize as evaluation: *is the model still producing anything for everyone it's supposed to serve?*

## Layer 2: Exposure — delivered is not viewed

Our first accept-rate number was 12%. It was also meaningless. The denominator was everything we *delivered* — including hundreds of summaries no human ever opened. When we rebuilt the funnel so the denominator was items actually *viewed* (screen views unioned with explicit opens, because neither source alone was complete), the accept rate was 44%.

Neither number says anything different about the model. The model was identical. What changed was the question: "how good are the outputs?" became "how good are the outputs *among those that reached a human*?" — and the gap between those questions is an exposure problem, not a capability problem. If your evaluation doesn't separate them, you will alternately blame and credit the model for what is actually a distribution failure. No benchmark can even express this distinction.

## Layer 3: Engagement — normalize by workload, not by calendar

DAU is the most famous metric in consumer analytics and it is actively misleading here. Our user base is a few dozen clinicians, many of whom see patients three days a week. Daily active users mostly measures the clinic schedule.

The fix is to normalize by opportunity: minutes of use per *appointment-provider-day* — a day on which a given clinician actually had patients booked. When we switched, providers who looked disengaged turned out to be the heaviest users per clinical hour, and vice versa. This is the deployment analogue of controlling for exam difficulty, and it took a join against the scheduling system that no analytics SDK will ever do for you.

## Layer 4: Action — the click that matters is in someone else's database

The single most consequential user action in our product — accepting or rejecting a suggestion — is invisible to our product-analytics tool. It happens in a surface the tracking SDK doesn't cover, and the only record is a row in the application database. If we had evaluated adoption from the tracker alone, we would have concluded nobody decided anything.

Worse: once we found the decisions, we had to ask *whose* they were. Our own clinical reviewers pass through the same interface as customers. When we first counted verdicts on the model's coding suggestions, 76 of the first 96 turned out to be our own team's internal review, not clinician adoption. The clinician-only truth was 20 decisions — a fine early signal, but a completely different claim. The platform records no user ID on decisions, so the separation had to be reverse-engineered from action vocabularies. An evaluation program that can't distinguish "our staff approved it" from "a customer adopted it" is not measuring the model; it's measuring itself.

## Layer 5: Outcome — evaluation you have to earn

Consumer funnels end at conversion because conversion happens inside the product. For an embedded assistant, the conversion is a clinical action recorded in *someone else's system*, days or weeks later: the order placed, the medication started, the diagnosis documented. That evidence does not flow into any dashboard by default. It does not exist as data unless you go get it.

This layer is where engagement metrics most often flatter to deceive. A 2026 randomized study of LLM assistance for health workers (Abaluck et al.) found the model genuinely changed how workers deliberated about test ordering — visible engagement, by any product metric — without improving whether tests went to the right patients. Layer 3 moved; layer 5 didn't. Only an outcome layer can tell those apart.

So we built what is essentially a reconciliation pipeline — extract what the assistant said per patient, extract what subsequently appeared in the chart, and attribute: which suggestions were followed by matching evidence? We run LLM attribution over the pairs, score it against manual chart review with citations, and report precision and recall of the attribution itself. This is the layer academics would recognize as evaluation — there's even a held-out human gold standard — but note where it sits: at the *end* of a product funnel, denominated by the exposure and action layers above it, weeks after the model ran. Outcome evaluation without the funnel above it is uninterpretable; you can't attribute an outcome to a suggestion nobody saw.

## Layer 6: Trust — rejection comments are the real error analysis

We collect thumbs up/down like everyone. The signal density is low. The dense signal is free-text rejection reasons: when a reviewer rejects a coding suggestion with *"referenced note doesn't exist — possible hallucination"* or *"condition not documented in the cited encounter,"* that is a labeled failure case with a root-cause hypothesis, produced by a domain expert, attached to a real input — exactly what error analysis on a benchmark tries to approximate with synthetic rubrics. A deployed product generates this continuously, for free, if you bother to store and read it.

## The two epochs: observed vs reconciled

Lay those six layers on the clock and they split into two epochs. Delivery, exposure, engagement, and the front half of action are **usage analytics** — observable live, from product telemetry and the application database. Outcomes, trust signals, and the tail of action are **post-usage analytics** — they only exist if you go back and reconcile: chart review, attribution runs, reading rejection comments.

This split matters because organizations reliably fund the first epoch and neglect the second. The live dashboards feel like accountability. But the second epoch is where the actual evaluative claims live — *did the model change what happened to the patient?* — and it is precisely the part that requires deliberate, ongoing, unglamorous work. Academic evaluation, ironically, lives entirely in a third place: before deployment, on inputs sampled from nowhere in this timeline at all.

## The foundation: data hygiene is your validity threat

Benchmark culture worries intensely about contamination — test items leaking into training data. Deployment evaluation has its own contamination problems, and they are worse because nobody is looking:

- **Your own employees are in the data.** A third of our early chat sessions were internal staff. Every engagement metric was inflated until we resolved session owners against an employee directory.
- **Support impersonation pollutes customer identity.** When our team impersonates a customer account to debug, the events fire under the *customer's* identity. Only ~80% of events in an impersonated session even carry the impersonation flag, so exclusion has to happen at the session level, not the event level. Until we did this, some of our "customer usage" was us.
- **Clocks disagree.** Our product analytics tool returns timestamps in project-local time; the application database speaks UTC; each clinic lives in its own timezone. A four-hour skew made whole activity timelines look misaligned until every source was forced onto one clock.
- **Identity is plural.** The same patient appears under three ID formats across systems. Joins that assume one format silently drop the rest.
- **Events double-count.** Status-transition logs record the same decision multiple ways. One decision per item — the last one — or your adoption numbers drift upward forever.

None of this is glamorous, and all of it is epistemically identical to benchmark contamination: unhandled, it doesn't add noise, it adds *bias with a flattering sign*. Every one of these issues, uncorrected, made our numbers look better than reality.

## What this implies

**Evaluation teams should look like product analytics teams.** The skills that produced trustworthy numbers for us were funnel design, identity resolution, event-schema archaeology, and denominator discipline — not rubric writing. The rubric-writing mattered exactly once, at the outcome layer, and even there it was embedded in a pipeline that looked like ETL.

**Platforms must instrument decisions as first-class events.** The most important gaps in our stack are platform gaps: decisions with no user ID, review status columns that are never written, internal actions indistinguishable from customer actions. If you build AI products: log who decided, log it where the decision happens, and mark your own staff. Your future evaluators will otherwise reverse-engineer it from action vocabularies, badly.

**Benchmarks keep their job — a smaller one.** Capability benchmarks are how you choose which model to deploy and how you regression-test upgrades. That's a unit test: necessary, cheap, run constantly, and utterly insufficient. The claim that matters — *this system, embedded in this workflow, changes outcomes* — is a product-analytics claim with an outcome-reconciliation tail, and it can only be established per deployment, per workflow, on the clock of the work itself.

The academic instinct is that evaluation should be independent of the messy particulars of deployment. For capability, that's right. For everything anyone actually wants to know about an AI product, the messy particulars *are* the evaluation.

---

## References

- [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://arxiv.org/abs/2505.23802) — 121 clinician-validated tasks, 35 benchmarks; [*Nature Medicine* version](https://www.nature.com/articles/s41591-025-04151-2); [leaderboard](https://medhelm.org/)
- [ARISE — AI Research and Science Evaluation network](https://arise-ai.org/); [MAST living benchmark](https://arise-ai.org/blog/mast); [State of Clinical AI Report 2026](https://arise-ai.org/report)
- [Introducing HealthBench (OpenAI, May 2025)](https://openai.com/index/healthbench/) — 5,000 conversations, 262 physicians, 48,562 rubric criteria
- [HealthBench Professional: Evaluating LLMs on Real Clinician Chats (April 2026)](https://arxiv.org/abs/2604.27470) — curated from 15,079 real clinician conversations
- [Knowledge–Practice Performance Gap in Clinical LLMs: Systematic Review of 39 Benchmarks (*JMIR*, Dec 2025)](https://pmc.ncbi.nlm.nih.gov/articles/PMC12706444/) — 84–90% exam vs 45–69% practice
- [Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions (CMU, 2026)](https://arxiv.org/pdf/2605.22612) — incl. Bean et al.'s 95%→34% evaluation-to-deployment drop; [blog version](https://blog.ml.cmu.edu/2026/06/19/healthcare-benchmarks-are-only-as-good-as-their-assumptions/)
- [Abaluck, Pless, Ravi, Sautmann & Schwartz — Does LLM Assistance Improve Healthcare Delivery? (NBER w34660, 2026)](https://www.nber.org/papers/w34660) — on-site physicians + laboratory tests; mixed effects on test appropriateness
- [OpenEvidence: 1M physician consultations in one day (March 2026)](https://www.prnewswire.com/news-releases/openevidence-achieves-historic-milestone-1-million-clinical-consultations-between-verified-doctors-and-an-artificial-intelligence-system-in-a-single-day-302712459.html); [~27M encounters/month, ~2/3 of US physicians](https://boingboing.net/2026/05/18/two-thirds-of-us-doctors-now-ask-an-ai-chatbot-for-treatment-advice.html)
- [Doximity 2026 State of AI in Medicine Report](https://www.doximity.com/reports/state-of-ai-medicine-report/2026) — daily use 47%→63%; 71% cite accuracy as top concern; [press summary](https://www.businesswire.com/news/home/20260317453665/en/Doximity-Study-Finds-Physicians-Rapidly-Adopting-AI-But-Accuracy-Concerns-Persist)