Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real
  World
Bowen Lei Dongkuan Xu Ruqi Zhang Bani Mallick
Sparse training has emerged as a promising method for resource-efficient deep
neural networks (DNNs) in real-world applications. However, the reliability of
sparse models remains a crucial concern, particularly in detecting unknown
out-of-distribution (OOD) data. This study addresses the knowledge gap by
investigating the reliability of sparse training from an OOD perspective and
reveals that sparse training exacerbates OOD unreliability. The lack of unknown
information and the sparse constraints hinder the effective exploration of
weight space and accurate differentiation between known and unknown knowledge.
To tackle these challenges, we propose a new unknown-aware sparse training
method, which incorporates a loss modification, auto-tuning strategy, and a
voting scheme to guide weight space exploration and mitigate confusion between
known and unknown information without incurring significant additional costs or
requiring access to additional OOD data. Theoretical insights demonstrate how
our method reduces model confidence when faced with OOD samples. Empirical
experiments across multiple datasets, model architectures, and sparsity levels
validate the effectiveness of our method, with improvements of up to
\textbf{8.4\%} in AUROC while maintaining comparable or higher accuracy and
calibration. This research enhances the understanding and readiness of sparse
DNNs for deployment in resource-limited applications. Our code is available on:
\url{https://github.com/StevenBoys/MOON}.