Dependency Aware Incident Linking in Large Cloud Systems
Supriyo Ghosh Karish Grover Jimmy Wong Chetan Bansal Rakesh Namineni Mohit Verma Saravan Rajmohan
Despite significant reliability efforts, large-scale cloud services
inevitably experience production incidents that can significantly impact
service availability and customer's satisfaction. Worse, in many cases one
incident can lead to multiple downstream failures due to cascading effects that
creates several related incidents across different dependent services. Often
time On-call Engineers (OCEs) examine these incidents in silos that lead to
significant amount of manual toil and increase the overall time-to-mitigate
incidents. Therefore, developing efficient incident linking models is of
paramount importance for grouping related incidents into clusters so as to
quickly resolve major outages and reduce on-call fatigue. Existing incident
linking methods mostly leverages textual and contextual information of
incidents (e.g., title, description, severity, impacted components), thus
failing to leverage the inter-dependencies between services. In this paper, we
propose the dependency-aware incident linking (DiLink) framework which
leverages both textual and service dependency graph information to improve the
accuracy and coverage of incident links not only coming from same service, but
also from different services and workloads. Furthermore, we propose a novel
method to align the embeddings of multi-modal (i.e., textual and graphical)
data using Orthogonal Procrustes. Extensive experimental results on real-world
incidents from 5 workloads of Microsoft demonstrate that our alignment method
has an F1-score of 0.96 (14% gain over current state-of-the-art methods). We
are also in the process of deploying this solution across 610 services from
these 5 workloads for continuously supporting OCEs improving incident
management and reducing manual toil.