Absence of Complaints Is Not Evidence It Worked

I've followed Hannah Fry for a while now. She does a great job explaining complex ideas clearly, which I find a bit unusual for a mathematician. Her latest Google DeepMind podcast episode is with Zoubin Ghahramani, who co-leads frontier AI there, and it is about uncertainty. I listened to it thinking about something else entirely.

A practice puts an AI layer on patient communication and referral coordination. Six months in, the numbers look good. Response times down, backlog clear, no complaints.

Here is what nobody in that building knows: whether it worked.

The patient who didn't get the callback doesn't call back. They just don't come in.

That's the shape of most provider-side admin work. A referral goes out and never closes, and no one reports it. A message gets triaged as routine when it wasn't, and if the patient recovers on their own nobody hears about it, and if they don't, nobody connects the two. A care gap marked closed in the system stays closed until the measure year ends. The work goes out, nothing contradicts it, and it settles into the record as correct.

Absence of complaints is not evidence that it worked. It is usually just absence.

Ghahramani's concern in that conversation was not accuracy. It was that these systems assert things with the same confidence whether or not they have any basis for it, and that people respond by either dismissing them wholesale or by quietly no longer checking. For any problem that matters, he said, he would take a system that knows when it doesn't know over one that is arrogant and overconfident.

Healthcare administration is a problem that matters. But you cannot tell which kind of system you bought by watching the output, because the output never argues back.

That changes what oversight is for. It is not a net catching errors on the way out. It is the only thing producing a signal at all, because nothing downstream is going to produce one for you.

So the question to put to a vendor is not how accurate the system is. It is how anyone would know.