The most common question when we talk about artificial intelligence in healthcare is whether it is safe.
It is an essential question. But it is not enough.
An AI can use advanced models, meet technical requirements and hold certifications. And even so, it can generate more work than it removes: forcing professionals to review more, producing irrelevant alerts, or failing to know when to stop.
Because in a hospital it is not enough for a tool to answer correctly.
It has to know what information to use, what it is allowed to do, when it should stop, when a person has to step in, and how to reconstruct afterwards what actually happened.
Trust is not installed, it is designed.
In every piece of data that gets queried, every task that gets automated, every alert that gets raised, and every moment in which a person must stay in control.
This idea is where M de Medicina Real (M for Real Medicine) begins, an Omniloy series about what happens when AI leaves the demo and starts working inside a hospital.
Because an AI can be very good at producing an answer and still not fit the place where it is going to be used.
Before deploying it, there are five questions worth answering.
01 What does the AI know when it starts working? A fast answer is worth little if it does not take into account what matters about the patient, the clinical moment and the hospital's workflow.
This is not just about summarising a medical record or answering a question. It is about knowing which information is relevant to that consultation, that professional and that moment.
Medical history, previous appointments, treatments, recent tests, specialty, reason for consultation or operational context can completely change the meaning of one and the same request.
If the AI does not understand context, it can generate more work: it forces people to review again what it should have helped to organise.
02 Can it act, or only recommend? Rescheduling an appointment, structuring a note or flagging an alert are not the same class of task.
Some tasks can be automated with clear rules. Others require the AI to prepare information or suggest a next step. And some decisions must remain in the hands of the professional.
The question is not how much an AI can automate, it is what it should automate and under what conditions.
Designing that boundary well usually matters more than chasing full automation.
03 When does a person have to step in? Automation works best when it knows when to stop and who to notify.
The goal is not for every case to end up in a review queue. That would simply move the work from one place to another.
The goal is for the system to identify which situations it can resolve autonomously and which ones need clinical judgement, additional context or a response outside protocol.
In patient follow-up, for example, an answer may be completely normal, may require a new interaction, or may need review by the team.
The workflow has to be able to tell those situations apart.
04 What happens when something does not fit? A useful AI does not need to invent a way out for every situation.
In a hospital there will always be exceptions: incomplete data, ambiguous requests, answers that do not fit a protocol, or situations that require referral.
The question is not whether the system can answer everything. It is whether it knows how to recognise when it should not carry on alone.
05 Can what happened be reviewed? If nobody can understand what happened, nobody can correct it or improve it either.
When an AI takes part in a clinical or operational process, the team needs to be able to review what information it used, what action it took, which rule was applied and what happened next.
This is not about turning every interaction into paperwork. It is about keeping enough traceability to learn, correct mistakes and improve processes. Especially when those processes affect patients and professionals.
These five questions do not replace regulation, clinical validation or technical evaluation.
They serve to answer another equally important question:
What happens when this AI actually starts working inside the hospital?
Because that is where it is decided whether a tool adds value or simply adds a new layer of complexity.
In the coming editions of M de Medicina Real we will dig into these decisions through concrete situations in consultation, reception and patient follow-up.
Every Wednesday, a new reflection on artificial intelligence from the place where it really matters: the daily work of professionals and patients.
Want to see how we answer these five questions when we deploy our agents in a hospital? Talk to our team.