Data Drift Monitoring for Log Anomaly Detection Pipelines
Dipak Wani Samuel Ackerman Eitan Farchi Xiaotong Liu Hau-wen Chang Sarasi Lalithsena
Logs enable the monitoring of infrastructure status and the performance of
associated applications. Logs are also invaluable for diagnosing the root
causes of any problems that may arise. Log Anomaly Detection (LAD) pipelines
automate the detection of anomalies in logs, providing assistance to site
reliability engineers (SREs) in system diagnosis. Log patterns change over
time, necessitating updates to the LAD model defining the `normal' log activity
profile. In this paper, we introduce a Bayes Factor-based drift detection
method that identifies when intervention, retraining, and updating of the LAD
model are required with human involvement. We illustrate our method using
sequences of log activity, both from unaltered data, and simulated activity
with controlled levels of anomaly contamination, based on real collected log
data.