On the Effectiveness of Adversarial Samples against Ensemble
  Learning-based Windows PE Malware Detectors
Trong-Nghia To Danh Le Kim Do Thi Thu Hien Nghi Hoang Khoa Hien Do Hoang Phan The Duy Van-Hau Pham
Recently, there has been a growing focus and interest in applying machine
learning (ML) to the field of cybersecurity, particularly in malware detection
and prevention. Several research works on malware analysis have been proposed,
offering promising results for both academic and practical applications. In
these works, the use of Generative Adversarial Networks (GANs) or Reinforcement
Learning (RL) can aid malware creators in crafting metamorphic malware that
evades antivirus software. In this study, we propose a mutation system to
counteract ensemble learning-based detectors by combining GANs and an RL model,
overcoming the limitations of the MalGAN model. Our proposed FeaGAN model is
built based on MalGAN by incorporating an RL model called the Deep Q-network
anti-malware Engines Attacking Framework (DQEAF). The RL model addresses three
key challenges in performing adversarial attacks on Windows Portable Executable
malware, including format preservation, executability preservation, and
maliciousness preservation. In the FeaGAN model, ensemble learning is utilized
to enhance the malware detector's evasion ability, with the generated
adversarial patterns. The experimental results demonstrate that 100\% of the
selected mutant samples preserve the format of executable files, while certain
successes in both executability preservation and maliciousness preservation are
achieved, reaching a stable success rate.