The Design Is Sound. The Verification Is Missing.

I spent a week convinced I had found a governance failure in one of our own workflows. I was wrong, and the thing I found instead is more useful.

What I thought I saw: AI generates a plan that goes into a patient's record, our staff submits it, no provider signs off. A clinical artifact entering a chart with nobody qualified having read it.

So I got the people who do the work on a call and asked them to walk me through it.

The provider oversight is there. It just does not happen where I was looking. The physician sat down with the software vendor and encoded the plan herself. The structure, the rules, what gets monitored and when. What the AI produces is her judgment, executing.

That is a different thing from AI making a clinical decision. She decided in advance what to do for a patient with a given history. The AI is filling in which patient. The output is not a new decision. It is her decision, applied.

Which changes what the downstream job is. Nobody needs to re-make the clinical call. Somebody needs to confirm the algorithm ran correctly on this patient. That is a logic problem, not a medical one, and an administrative person can do it.

Here is the part I would not want anyone to skip. That checking takes a specific capability. Holding a rule in your head, comparing it against an output, and noticing when something does not match. In my experience not all administrative staff can do that work. Some can do it well, and better than the physician would, because it is careful comparison rather than clinical reasoning. Many cannot.

So the arrangement is sound, and it is conditional. Encoded judgment plus capable verification is safe. Encoded judgment plus whoever was available is not, and from the outside those look identical.

Which puts the obligation somewhere specific. If a health organization moves oversight to design time, it owns proving the person doing the verification is capable of it. Not that a person exists in the workflow. That this person can do this job.

Worth saying why this matters more than it would with conventional software. A coded algorithm runs the rule or it fails visibly. An AI executing the same rule is right most of the time and wrong occasionally, with nothing to tell you which. Last night I asked an AI to total a hotel stay. It gave me the answer in the wrong currency, then for the wrong number of nights, and it took four rounds to get a number I trusted. That is arithmetic. If that is the current reliability on arithmetic, you do not assume it on a monitoring instruction without someone checking.

In the workflow I was looking at, nobody is doing that verification at all. The design is sound. The submission is appropriate. Nobody is checking that the algorithm produced what it should have for any given patient.

So the gap is real and narrower than I thought. Not a missing clinician. A missing QA step, and an unanswered question about who is qualified to fill it.