Divide-and-Conquer the NAS puzzle in Resource Constrained Federated
  Learning Systems
Yeshwanth Venkatesha Youngeun Kim Hyoungseob Park Priyadarshini Panda
Federated Learning (FL) is a privacy-preserving distributed machine learning
approach geared towards applications in edge devices. However, the problem of
designing custom neural architectures in federated environments is not tackled
from the perspective of overall system efficiency. In this paper, we propose
DC-NAS -- a divide-and-conquer approach that performs supernet-based Neural
Architecture Search (NAS) in a federated system by systematically sampling the
search space. We propose a novel diversified sampling strategy that balances
exploration and exploitation of the search space by initially maximizing the
distance between the samples and progressively shrinking this distance as the
training progresses. We then perform channel pruning to reduce the training
complexity at the devices further. We show that our approach outperforms
several sampling strategies including Hadamard sampling, where the samples are
maximally separated. We evaluate our method on the CIFAR10, CIFAR100, EMNIST,
and TinyImagenet benchmarks and show a comprehensive analysis of different
aspects of federated learning such as scalability, and non-IID data. DC-NAS
achieves near iso-accuracy as compared to full-scale federated NAS with 50%
fewer resources.