AIOps-Driven Enhancement of Log Anomaly Detection in Unsupervised
  Scenarios
Daksh Dave Gauransh Sawhney Dhruv Khut Sahil Nawale Pushkar Aggrawal Prasenjit Bhavathankar
Artificial intelligence operations (AIOps) play a pivotal role in
identifying, mitigating, and analyzing anomalous system behaviors and alerts.
However, the research landscape in this field remains limited, leaving
significant gaps unexplored. This study introduces a novel hybrid framework
through an innovative algorithm that incorporates an unsupervised strategy.
This strategy integrates Principal Component Analysis (PCA) and Artificial
Neural Networks (ANNs) and uses a custom loss function to substantially enhance
the effectiveness of log anomaly detection. The proposed approach encompasses
the utilization of both simulated and real-world datasets, including logs from
SockShop and Hadoop Distributed File System (HDFS). The experimental results
are highly promising, demonstrating significant reductions in pseudo-positives.
Moreover, this strategy offers notable advantages, such as the ability to
process logs in their raw, unprocessed form, and the potential for further
enhancements. The successful implementation of this approach showcases a
remarkable reduction in anomalous logs, thus unequivocally establishing the
efficacy of the proposed methodology. Ultimately, this study makes a
substantial contribution to the advancement of log anomaly detection within
AIOps platforms, addressing the critical need for effective and efficient log
analysis in modern and complex systems.