Updating Clinical Risk Stratification Models Using Rank-Based
  Compatibility: Approaches for Evaluating and Optimizing Clinician-Model Team
  Performance
Erkin Ötleş Brian T. Denton Jenna Wiens
As data shift or new data become available, updating clinical machine
learning models may be necessary to maintain or improve performance over time.
However, updating a model can introduce compatibility issues when the behavior
of the updated model does not align with user expectations, resulting in poor
user-model team performance. Existing compatibility measures depend on model
decision thresholds, limiting their applicability in settings where models are
used to generate rankings based on estimated risk. To address this limitation,
we propose a novel rank-based compatibility measure, $C^R$, and a new loss
function that aims to optimize discriminative performance while encouraging
good compatibility. Applied to a case study in mortality risk stratification
leveraging data from MIMIC, our approach yields more compatible models while
maintaining discriminative performance compared to existing model selection
techniques, with an increase in $C^R$ of $0.019$ ($95\%$ confidence interval:
$0.005$, $0.035$). This work provides new tools to analyze and update risk
stratification models used in clinical care.