A Critical Review of Common Log Data Sets Used for Evaluation of
  Sequence-based Anomaly Detection Techniques
Max Landauer Florian Skopik Markus Wurzenberger
Log data store event execution patterns that correspond to underlying
workflows of systems or applications. While most logs are informative, log data
also include artifacts that indicate failures or incidents. Accordingly, log
data are often used to evaluate anomaly detection techniques that aim to
automatically disclose unexpected or otherwise relevant system behavior
patterns. Recently, detection approaches leveraging deep learning have
increasingly focused on anomalies that manifest as changes of sequential
patterns within otherwise normal event traces. Several publicly available data
sets, such as HDFS, BGL, Thunderbird, OpenStack, and Hadoop, have since become
standards for evaluating these anomaly detection techniques, however, the
appropriateness of these data sets has not been closely investigated in the
past. In this paper we therefore analyze six publicly available log data sets
with focus on the manifestations of anomalies and simple techniques for their
detection. Our findings suggest that most anomalies are not directly related to
sequential manifestations and that advanced detection techniques are not
required to achieve high detection rates on these data sets.