EMBERSim: A Large-Scale Databank for Boosting Similarity Search in
  Malware Analysis
Dragos Georgian Corlatescu Alexandru Dinu Mihaela Gaman Paul Sumedrea
In recent years there has been a shift from heuristics-based malware
detection towards machine learning, which proves to be more robust in the
current heavily adversarial threat landscape. While we acknowledge machine
learning to be better equipped to mine for patterns in the increasingly high
amounts of similar-looking files, we also note a remarkable scarcity of the
data available for similarity-targeted research. Moreover, we observe that the
focus in the few related works falls on quantifying similarity in malware,
often overlooking the clean data. This one-sided quantification is especially
dangerous in the context of detection bypass. We propose to address the
deficiencies in the space of similarity research on binary files, starting from
EMBER - one of the largest malware classification data sets. We enhance EMBER
with similarity information as well as malware class tags, to enable further
research in the similarity space. Our contribution is threefold: (1) we publish
EMBERSim, an augmented version of EMBER, that includes similarity-informed
tags; (2) we enrich EMBERSim with automatically determined malware class tags
using the open-source tool AVClass on VirusTotal data and (3) we describe and
share the implementation for our class scoring technique and leaf similarity
method.