AI chatbot misuse tops ECRI's 2026 hazards. We argue for governed AI

ECRI named misuse of AI chatbots the top health technology hazard for 2026 because general-purpose chatbots produce expert-sounding medical guidance without validation, without regulation as medical devices and without any reliable record of what was said. On 21 January 2026 the independent patient safety organisation published its annual Top 10 Health Technology Hazards report and placed chatbot misuse above every device and system it assessed. The tools it examined are the consumer chatbots that clinicians, patients and healthcare staff already use every day. ECRI's warning is not that AI has no place in healthcare. It is that unvalidated, ungoverned AI already has one.

We built Mickai, a Sovereign Intelligence Operating System, a SIOS, for organisations that cannot accept that trade. It runs on the customer's own hardware, on-premise and air-gapped, and it treats governance as architecture rather than policy. Here we look at what ECRI found and what governed clinical AI has to look like in practice.

What did ECRI actually warn about?

ECRI's warning is specific: general-purpose AI chatbots are being used for medical guidance despite being neither validated for healthcare purposes nor regulated as medical devices. In its assessment, ECRI documented chatbots suggesting incorrect diagnoses, inventing anatomy and offering guidance that could cause direct harm, including advice that would have put a patient at risk of burns through incorrect electrode placement. Scale is what pushed the risk to the top of the list. By one major provider's own analysis, a quarter of its 800 million regular users ask health questions every week. ECRI's advice is to recognise the tools' limitations, to treat them as educational resources rather than decision-making aids, and to keep a human in the loop who confirms important information with trusted sources and qualified professionals.

The rest of the list reinforces the theme. Second place went to healthcare facilities' lack of preparation for a sudden loss of access to electronic systems and patient information, and cybersecurity weaknesses in legacy medical devices also featured. The common thread is dependence on technology that organisations neither control nor fully understand.

Why do fluent chatbots fail in clinical settings?

Because fluency and accuracy are different properties, and a consumer chatbot is optimised for the first. A general-purpose model produces the most plausible continuation of a conversation, and in medicine a plausible wrong answer is more dangerous than an obviously wrong one, because it passes the reader's checks. A single model has no independent second opinion, no validation boundary and no obligation to say it does not know. Consumer tools compound the problem operationally. The conversation happens on someone else's infrastructure, outside clinical governance, with patient details typed into a system the hospital does not control and cannot audit. When something goes wrong, there is no signed record of what was asked, what was answered and who acted on it.

What does governed clinical AI look like?

Governed clinical AI is defined by its controls, not by how good its answers sound. Inside our operating system those controls are enforced by the system itself, as subsystems every studio shares:

• A cooperative multi-model consensus substrate, in which our own models must agree before any sensitive action runs, so no single model's fluent error goes unchallenged.
• Clinical honesty by design: clinical-adjacent output is reference material only, reviewed and signed by the treating clinician, and never presented as a diagnosis.
• A signed entry in the Open Audit Record for every question and answer, cryptographically signed, post-quantum, tamper-evident and verifiable offline.
• Human confirmation and voice-biometric gating on sensitive steps, anchored to a hardware-held root of trust, so accountability attaches to a named person.
• Fully offline, on-premise, air-gapped operation on the organisation's own hardware, so patient data never leaves the clinical network.

Each control answers a failure ECRI documented. Consensus attacks the fluent wrong answer. Clinician sign off makes the human in the loop an enforced step rather than a policy hope. The audit record replaces the missing account of what was said. And on-premise operation keeps patient data inside clinical governance.

How does Mickai put this into practice?

Mickai is one operating system comprising 87 studios, focused environments for functions from documentation and administration to finance and operations. Ten studios are production-ready and we launch with those ten, while the remaining 77 are in development. Every studio shares the same subsystems, so consensus, gating and the Open Audit Record apply to every interaction rather than to one carefully managed pilot. Where a studio's work is clinical-adjacent, its output is reference only, reviewed and signed by the treating clinician. We do not claim regulatory approval of any kind, and nothing our system produces is a diagnosis. That honest boundary is part of the design, exactly as ECRI recommends.

“ECRI's hazard is not artificial intelligence. It is fluent, unvalidated, unrecorded intelligence. Governance is what separates the two, and governance has to be built into the operating system, not written into a policy.”

The methods behind these controls sit within 104 filed UK patent applications across 2,340 claims, owned by Mickai LTD, filed rather than granted. They are the moat beneath the work, not the headline. The headline is that clinical AI can be governed, and provably so.

What should healthcare leaders do now?

Treat ECRI's list as a design brief rather than a warning label. Staff and patients are already using chatbots, so the realistic choice is between ungoverned AI that arrives through the front door on personal phones and governed AI the organisation deploys deliberately, with validation boundaries, clinician sign off and a verifiable record of every interaction. ECRI's second-ranked hazard, unpreparedness for a sudden loss of electronic systems, points the same way. A system that runs entirely on the organisation's own hardware keeps working, and keeps its evidence, when connectivity does not. The organisations that answer chatbot demand with governed capability will be the ones the next hazard list has nothing to say about.

Frequently asked questions

Why did ECRI name AI chatbot misuse the top health hazard for 2026?

Because general-purpose chatbots produce expert-sounding but unvalidated medical guidance, are not regulated as medical devices, and are already used by clinicians, patients and healthcare staff at enormous scale. In its January 2026 report, ECRI documented incorrect diagnoses, invented anatomy and guidance that could cause direct patient harm.

Are AI chatbots regulated as medical devices?

The general-purpose chatbots ECRI assessed are not. They are consumer services, not validated for healthcare purposes, which is central to ECRI's warning. Whether any specific AI system counts as a medical device depends on its intended purpose and the applicable regulator, and we make no claim of regulatory approval for our own system.

Can hospitals use AI safely at all?

Yes, when the AI runs inside clinical governance: validated boundaries, cross-checked outputs, clinician review and sign off on anything clinical-adjacent, and a signed record of every interaction. ECRI's own advice, to keep a human in the loop and treat chatbots as educational aids, describes controls a governed operating system can enforce.

Does Mickai diagnose patients?

No. Mickai's clinical-adjacent capability produces reference material only, which the treating clinician reviews and signs. It is not a diagnosis, and we claim no regulatory approval of any kind. What we add is governance: multi-model consensus, human confirmation and a verifiable audit record, all running on the organisation's own hardware.

What is MICKAI?

MICKAI is a Sovereign Intelligence Operating System, a SIOS, that runs on the customer's own hardware, on-premise and fully air-gapped. Every action is recorded in the Open Audit Record, cryptographically signed, post-quantum secure, tamper-evident and verifiable offline. It comprises 87 studios on one operating system, with ten production-ready at launch and 77 in development, and it is supported by 104 filed UK patent applications across 2,340 claims, filed rather than granted.