Building AI Systems for High-Stakes Healthcare Information

Most discussion of AI systems is a discussion of capability: what can the model do, how well does it do it, what can it do now that it could not do last year.

In healthcare information, capability is rarely the binding constraint. The binding constraint is what happens when the system is wrong, and specifically what happens when it is wrong in a way that sounds right.

That asymmetry changes the engineering. Everything below follows from it.

A note on scope: this is written at the level of engineering principle, drawn from building production systems in this domain. It describes how I think about the problem, not any employer's architecture, data, or product internals.

The asymmetry that defines the problem

A general-purpose assistant that is wrong 5% of the time is useful, because the user can usually tell. They have context. They notice when the answer does not fit.

A clinical information system that is wrong 5% of the time in a way the user cannot detect is not 95% useful. It may be worse than nothing, because it has replaced an uncertain human with a confident machine, and confidence is exactly the signal people use to decide how hard to check.

The users who most need these systems are the ones least able to verify the output cheaply — that is why they are asking. Any design that quietly depends on the user catching mistakes has inverted its own value proposition.

So the first design question is not "how good can we make the answers?" It is "how does someone know whether to trust this particular answer?"

Grounding is a product requirement, not an optimisation

Retrieval is often framed as a technique for improving accuracy — a way to give the model better context so it produces better text. That framing undersells it.

In this domain, retrieval is what makes the output checkable. A claim with a traceable source can be verified by the person acting on it in seconds. The same claim without one cannot be verified at all, no matter how correct it happens to be.

This has consequences that go beyond adding a citation to the end of a paragraph:

Retrieval quality is the ceiling on everything downstream. No amount of reasoning capability recovers from having the wrong evidence in context. Most quality problems that present as generation problems are retrieval problems wearing a costume.

Citations have to be attached to claims, not to responses. A response with three citations at the bottom tells the reader nothing about which source supports which sentence. Claim-level attribution is harder to build and is the difference between a citation that functions and a citation that decorates.

Sources are not interchangeable. In clinical information, the provenance and recency of evidence are part of its meaning. A system that treats all retrieved text as equally authoritative has thrown away information that domain experts consider primary.

The effect of getting this right is behavioural, not just epistemic. When users can check, they check — and their trust becomes calibrated rather than binary.

Evaluation is the actual product surface

I build the measurement before the capability. Not as discipline for its own sake, but because in a domain where I cannot personally verify every output, evaluation is my only access to whether the system works.

Some things I have found repeatedly true:

Build the eval first and you can move fast afterwards. The instinct is that evaluation slows you down. The opposite holds. With a regression suite you can make aggressive changes, because you will know within minutes what you broke. Without one, every change is a small act of faith and the accumulated caution is far more expensive.

Measuring what is easy to measure is how systems drift. Retrieval hit rate is easy. Answer correctness is hard. A team that optimises the easy metric for six months produces a system with excellent retrieval statistics and no improvement in the thing anyone cares about.

Aggregate metrics conceal the failures that matter. A system with strong overall performance can be systematically wrong about a narrow category — one drug class, one question type — and the aggregate will not move. Stratified evaluation is the only way to see it, and the strata have to come from domain understanding, not from clustering.

Evaluation is continuous, not a launch gate. Sources change. Guidelines are revised. Models are updated underneath you. A system validated once and never again is a system whose correctness is decaying at an unknown rate.

Designing for the failure mode

The useful question about a model in production is not how good it is on average. It is how does it fail.

Fail loudly rather than silently. A system that returns nothing, or flags low confidence, is one a professional can work with. A system that fabricates plausibly is one they cannot. Given the choice between an unhelpful answer and a confidently wrong one, unhelpful wins every time in this domain — and it is not close.

Abstention is a feature. "I do not have sufficient evidence to answer this" is a legitimate and valuable output. Building a system that will not say it — because saying it feels like a product failure — guarantees it will say something else instead, and the something else will be invented.

Make failures recoverable. When the system is wrong, how does the user find out, and what do they do next? If the answer is "they don't" and "nothing", the design is incomplete regardless of benchmark performance.

Agents, applied with restraint

Multi-step agentic systems earn their complexity in specific places:

• Decomposition, when a question genuinely contains several sub-questions that need separate evidence.
• Retrieval planning, when the right search depends on what an earlier search returned.
• Verification, when a separate pass can check a draft answer against its sources — which is one of the highest-value uses of extra inference I know of in this domain.

They fail to earn it when they add steps to a task a single well-constructed call handles. Every additional step is additional failure surface, additional latency, and additional distance between the user and an explanation of what happened. Complexity is a cost that has to be paid for in measured improvement.

From prototype to production

The distance between a working demonstration and a deployed system is where most of the engineering time actually goes, and it is chronically underestimated.

Latency stops being a nice-to-have when it determines whether the tool fits in someone's workflow. Cost stops being theoretical at volume. Reliability becomes a first-class concern the moment people depend on the system being available. Observability becomes the difference between fixing a problem and speculating about it.

None of this is glamorous, and all of it decides whether the good idea at the centre of the system ever reaches anyone.

The systems worth building here are the ones designed for the day they are wrong. That day is coming for every one of them.