Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep
  Reinforcement Learning Approach
Brian Etter James Lee Hu Mohammedreza Ebrahimi Weifeng Li Xin Li Hsinchun Chen
Adversarial Malware Generation (AMG), the generation of adversarial malware
variants to strengthen Deep Learning (DL)-based malware detectors has emerged
as a crucial tool in the development of proactive cyberdefense. However, the
majority of extant works offer subtle perturbations or additions to executable
files and do not explore full-file obfuscation. In this study, we show that an
open-source encryption tool coupled with a Reinforcement Learning (RL)
framework can successfully obfuscate malware to evade state-of-the-art malware
detection engines and outperform techniques that use advanced modification
methods. Our results show that the proposed method improves the evasion rate
from 27%-49% compared to widely-used state-of-the-art reinforcement
learning-based methods.