Large-scale global optimization of ultra-high dimensional non-convex
  landscapes based on generative neural networks
Jiaqi Jiang Jonathan A. Fan
We present a non-convex optimization algorithm metaheuristic, based on the
training of a deep generative network, which enables effective searching within
continuous, ultra-high dimensional landscapes. During network training,
populations of sampled local gradients are utilized within a customized loss
function to evolve the network output distribution function towards one peak at
high-performing optima. The deep network architecture is tailored to support
progressive growth over the course of training, which allows the algorithm to
manage the curse of dimensionality characteristic of high-dimensional
landscapes. We apply our concept to a range of standard optimization problems
with dimensions as high as one thousand and show that our method performs
better with fewer function evaluations compared to state-of-the-art algorithm
benchmarks. We also discuss the role of deep network over-parameterization,
loss function engineering, and proper network architecture selection in
optimization, and why the required batch size of sampled local gradients is
independent of problem dimension. These concepts form the foundation for a new
class of algorithms that utilize customizable and expressive deep generative
networks to solve non-convex optimization problems.