Detecting Visual Cues in the Intensive Care Unit and Association with
  Patient Clinical Status
Subhash Nerella Ziyuan Guan Andrea Davidson Yuanfang Ren Tezcan Baslanti Brooke Armfield Patrick Tighe Azra Bihorac Parisa Rashidi
Intensive Care Units (ICU) provide close supervision and continuous care to
patients with life-threatening conditions. However, continuous patient
assessment in the ICU is still limited due to time constraints and the workload
on healthcare providers. Existing patient assessments in the ICU such as pain
or mobility assessment are mostly sporadic and administered manually, thus
introducing the potential for human errors. Developing Artificial intelligence
(AI) tools that can augment human assessments in the ICU can be beneficial for
providing more objective and granular monitoring capabilities. For example,
capturing the variations in a patient's facial cues related to pain or
agitation can help in adjusting pain-related medications or detecting
agitation-inducing conditions such as delirium. Additionally, subtle changes in
visual cues during or prior to adverse clinical events could potentially aid in
continuous patient monitoring when combined with high-resolution physiological
signals and Electronic Health Record (EHR) data. In this paper, we examined the
association between visual cues and patient condition including acuity status,
acute brain dysfunction, and pain. We leveraged our AU-ICU dataset with 107,064
frames collected in the ICU annotated with facial action units (AUs) labels by
trained annotators. We developed a new "masked loss computation" technique that
addresses the data imbalance problem by maximizing data resource utilization.
We trained the model using our AU-ICU dataset in conjunction with three
external datasets to detect 18 AUs. The SWIN Transformer model achieved 0.57
mean F1-score and 0.89 mean accuracy on the test set. Additionally, we
performed AU inference on 634,054 frames to evaluate the association between
facial AUs and clinically important patient conditions such as acuity status,
acute brain dysfunction, and pain.