Human oversight has to survive the AI being wrong

After a succession of useful recommendations, trusting an AI system can seem reasonable. The occasional wrong answer may then receive less scrutiny, particularly from someone who is still learning to make the judgement independently.

Research by Junseong Lee and colleagues, published in npj Digital Medicine on 20 October 2026, examines that problem in neonatal diagnosis. It was initially released as accepted, peer-reviewed research ahead of the final version of record. The setting is clinical imaging, not marketing compliance, but the question is recognisable: what happens to the human control when the recommendation is wrong?

An approval count would miss this problem. It records that people reviewed the output without showing whether they could recognise the errors left for them to catch.

Fourteen readers, 1,750 interpretations

The authors’ research repository describes a multi-reader crossover study involving 14 readers: six paediatric radiologists, three neonatologists and five radiology residents. They interpreted 125 external radiographs from 11 tertiary hospitals, producing 1,750 interpretation events. That final number counts readings, not independent participants. Cases were reviewed unaided or with reliable or deliberately error-injected AI assistance.

The publisher’s abstract reports that trainees accepted 52% of incorrect suggestions in the high-reliability condition. Under flawed assistance, the three neonatologists correctly overrode 91.7% of errors. The authors explicitly describe the specialist finding as hypothesis-generating because the subgroup is small; the reported 95% confidence interval is 83% to 96%.

These figures concern different experimental conditions, so they are not a like-for-like comparison of the professional groups. They also reflect particular readers and cases. The paper does not measure financial-promotion review, establish a universal error-acceptance rate or show that experienced people are immune to automation bias.

The full manuscript was unavailable during the source checks on 28 October 2026. The quantitative findings above come from the publisher’s abstract, with the design checked against the authors’ repository. They support examining oversight during AI failure, while leaving the detailed mechanisms and generalisability open to further assessment.

Reviewers need a basis for disagreement

In a compliance review, an objection may sound convincing while citing a rule for the wrong product. Recognising that error takes knowledge of the product and the requirement, as well as access to the source. Reading the explanation carefully will not be enough if the reviewer cannot assess its premise.

This is a particular difficulty with well-written recommendations. A plausible citation gives the reviewer something to check, but its presence can also make the recommendation appear more thoroughly supported than it is. Checking that the source exists and checking that it applies are separate pieces of work.

Even a reviewer who spots the problem may need specialist help to resolve it. If obtaining that help is treated as an avoidable delay, the formal invitation to challenge AI will have little effect on a busy campaign.

Meaningful human oversight in financial services needs an account of what the reviewer can challenge. One implication of the study is to examine that role when the recommendation is wrong, alongside performance during routine assistance.

Redoing every assessment from scratch would sacrifice much of the value of assistance. Reviewers need to learn where reliance is justified and which signs warrant a closer look. Testing that judgement is more informative than asking whether they understand a policy warning about over-reliance.

Test the team with errors it could plausibly encounter

One way to investigate a compliance workflow would be to create a controlled set of realistic cases containing known AI mistakes. The cases could include a missed material issue, an objection based on the wrong requirement and a recommendation that overlooks missing product evidence. Reviewers would need a safe environment in which errors cannot reach customers.

The exercise should distinguish reviewers who miss an error from those who notice it but let it proceed. The latter may understand the problem yet be unsure of their authority or lack time to investigate. An unhurried training exercise would not expose all of those pressures, so workload belongs in the evaluation too.

The neonatal study has not validated this method for compliance work. Specialists would need to assess the test cases and choose errors that the application could plausibly make. Obvious traps might reward test-taking skill while revealing little about routine review.

An independent first assessment may sometimes help. The FCA’s October 2026 examples describe an affordability workflow in which reviewers were not shown the AI’s proposed decision. The approach preserves a comparison, but it does not establish which party is correct when judgements differ. Disagreement still requires adjudication against evidence.

Measuring only agreement with historical approvals would create a weaker test. Past reviewers may have missed issues or applied inconsistent interpretations. A credible evaluation should preserve legitimate ambiguity and distinguish it from a clear error. Otherwise, the business risks training both people and technology to reproduce the same mistakes more consistently.

Expertise needs time and continued practice

If AI supplies the research and first analysis on every case, staff get less practice forming a view before seeing a persuasive answer. That is a difficult arrangement for people who may later be expected to challenge the system independently.

Some manual work contributes little to learning, while AI could help staff explore unfamiliar material or obtain faster feedback. The decision is which activities develop the judgement the business will continue to need and how people will practise them once routine research is automated.

Selected difficult cases could be used for independent analysis followed by discussion of how the evidence changed the decision. Records of the reasoning behind expert decisions would give the discussion more substance than a collection of approved sentences.

A specialist may have all the necessary knowledge and still be unable to apply it across a growing queue of AI output. Increasing production without examining review capacity can weaken the control while leaving its formal description untouched.

Human review deserves the same scrutiny as the technology it is meant to oversee. If the business relies on a person to catch the system’s mistakes, it needs evidence that they can do so under the conditions in which they actually work.