Smart camera reduces unexpected complications in hospitals

March 15, 2021

The FORSEE camera uses machine learning algorithms developed at TU/e

Researchers at Eindhoven University of Technology (TU/e) and Catharina Hospital have developed a camera that monitors patients continuously and automatically. Using artificial intelligence, the smart camera spots subtle changes in the face or chest that could indicate possible complications, enabling doctors and nurses to intervene in time. "The camera is a godsend for monitoring patients in the nursing department," says Arthur Bouwman, an anesthesiologist at Catharina Hospital. The camera will be further developed and clinically tested in the coming years.

Suppose you are a heart or cancer patient and you have had surgery. Once the surgery is successful, and you no longer need to stay in the Intensive Care or Medium Care Unit, you are usually transferred to a regular nursing ward to continue your recovery.

There you will not be constantly monitored. However, even then it is vital that doctors and nurses keep an eye out for complications. In most hospitals someone will come to your bedside every six to ten hours to take your pulse and check your breathing.

That might sound like a sensible arrangement. However, the time between those checks is quite long, with the risk that something might happen in the meantime. Moreover, the check-ups can be quite annoying, especially when you are sleeping. Wouldn't it be nice to have a system that could remotely monitor you permanently, without bothering you?

Researcher Sveta Zinger
Researcher Sveta Zinger

Intelligent camera

Researchers at TU/e and the Catharina Hospital in Eindhoven believe they have found the solution: an intelligent camera that constantly checks your heartbeat and breathing, and sends a signal to doctors and nurses when something is wrong.

The researchers believe this will reduce complications and unexpected deaths, about 40 percent of which occur in a regular nursing ward.

"The camera uses artificial intelligence," says Sveta Zinger, associate professor at the Departement of Electrical Engineering who specializes in medical image analysis. "The camera is focused on the patient's chest and face. This allows it to see exactly if there are any subtle changes in the color of your face or the movement of your breast, which could indicate possible complications."

From lab to hospital

The camera has been extensively tested in the laboratory over the past few years. "That showed that a smart camera can remotely measure vital data, such as heart rate, breathing, temperature and the amount of oxygen in the blood. Using machine learning, we can then interpret that data as well," Zinger explains.

"To see if the camera also works in a clinical setting, we are going to conduct an experiment in the ICU department at Catharina Hospital. Ten patients and five healthy volunteers will be monitored non-stop for eight hours, during which time we will combine the data from the camera with the measurement data from the ICU equipment, the 'gold standard' when it comes to monitoring. This will allow us to train the camera in situations it is not yet familiar with, for example when there is less light, or when a patient turns their face away."

Anonymous pixels

For those who might have reservations about having a camera next to their bed, Zinger emphasizes that the privacy of the patient is paramount. All participants in the study have to give their express permission, and there is a button on each camera that allows the patient to turn the device off, if he or she so desires. 

"What’s more, the patients are not personally identifiable in the video recordings. For the algorithms to do their work, they only need anonymous pixels," says the researcher. Zinger also knows from experience that many patients have no problem at all with camera monitoring, because they also see the benefits.

Anesthesiologist Arthur Bouwman
Anesthesiologist Arthur Bouwman

Benefits for patients

The Eindhoven-based Catharina Hospital sees great benefits for the smart camera. "Our hospital specializes in cardiovascular diseases and also treats many cancer patients. It is particularly important for this group that they are closely monitored. Complications can adversely affect recovery and in some cases can even be fatal. Timely intervention is therefore of great importance," says anesthesiologist Arthur Bouwman.

Moreover, a camera is less stressful than regular checks with an electrocardiogram, in which the patient has electrodes taped to his body, or a saturation meter, in which a clip is slipped over the finger to measure the amount of oxygen in your blood. "The patient can now freely move around and is no longer restricted by cables with electrodes. This is great news, because nothing is more damaging to a patient than lying still in bed," says Bouwman.

The doctor expects that the camera can be used mainly on nursing wards. "I don't think the camera can ever replace the classic monitoring in the ICU and medium-care. There is too much 'noise' in those places for that, which distorts the image. But we do want to start testing the camera for home monitoring".

False alarm

The first results of the study are expected in the summer of 2021. Ultimately, the FORSEE project, an initiative of the Eindhoven MedTech Innovation Center (e/MTIC ), and subsidized by ZonMw, the Dutch Research Council, the Heart Foundation and the Dutch CardioVascular Alliance (DCVA), will last five years.

The researchers will also be looking at the effect of the cameras on the patients (do they like the cameras or not?) and on the nurses (are the cameras easy to use and don't they set off to many false alarms?).

The project will also examine how machine learning can help doctors and nurses interpret the vast amounts of data collected by the cameras, so they can make better clinical decisions.


The science behind the smart cameras

The smart cameras in this project apply so-called remote photoplethysmography (PPG) to measure heart rate (also known as pulse), and indirectly, respiration and oxygen saturation. PPG is an optical technology that detects blood volume changes in the small bloodvessels beneath the skin.

Instead of connecting the light source and the detector directly to the patient, as in normal PPG, rPPG uses a remote camera, making it much less invasive. However, this also renders the signals more sensitive to noise: patients move, skin colors vary, and the light conditions are not always optimal.

To remove the unwanted noise, a set of sophisticated algorithms is used. These help the camera to detect the parts of the skin that have enough pulse to generate a significant signal. The researchers then apply another set of algorithms to extract a single pulse signal from the RGB-signals generated by the camera. To ensure speed and efficiency, crucial for real-time monitoring of patients, further data preprocessing algorithms are used to optimize the process.

More information about the technology behind rPPG can be found in the work (pdf, pdf) of TU/e researchers Wenjin Wang and Mark van Gastel.


Media contact

Henk van Appeven
(Communications Adviser)

Latest news

Keep following us