Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits!
Tirth Patel Fred Lu Edward Raff Charles Nicholas Cynthia Matuszek James Holt
Industry practitioners care about small improvements in malware detection
accuracy because their models are deployed to hundreds of millions of machines,
meaning a 0.1\% change can cause an overwhelming number of false positives.
However, academic research is often restrained to public datasets on the order
of ten thousand samples and is too small to detect improvements that may be
relevant to industry. Working within these constraints, we devise an approach
to generate a benchmark of configurable difficulty from a pool of available
samples. This is done by leveraging malware family information from tools like
AVClass to construct training/test splits that have different generalization
rates, as measured by a secondary model. Our experiments will demonstrate that
using a less accurate secondary model with disparate features is effective at
producing benchmarks for a more sophisticated target model that is under
evaluation. We also ablate against alternative designs to show the need for our
approach.