SMOOTHIE: A Theory of Hyper-parameter Optimization for Software
  Analytics
Rahul Yedida Tim Menzies
Hyper-parameter optimization is the black art of tuning a learner's control
parameters. In software analytics, a repeated result is that such tuning can
result in dramatic performance improvements. Despite this, hyper-parameter
optimization is often applied rarely or poorly in software analytics--perhaps
due to the CPU cost of exploring all those parameter options can be
prohibitive.
  We theorize that learners generalize better when the loss landscape is
``smooth''. This theory is useful since the influence on ``smoothness'' of
different hyper-parameter choices can be tested very quickly (e.g. for a deep
learner, after just one epoch).
  To test this theory, this paper implements and tests SMOOTHIE, a novel
hyper-parameter optimizer that guides its optimizations via considerations of
``smothness''. The experiments of this paper test SMOOTHIE on numerous SE tasks
including (a) GitHub issue lifetime prediction; (b) detecting false alarms in
static code warnings; (c) defect prediction, and (d) a set of standard ML
datasets. In all these experiments, SMOOTHIE out-performed state-of-the-art
optimizers. Better yet, SMOOTHIE ran 300% faster than the prior state-of-the
art. We hence conclude that this theory (that hyper-parameter optimization is
best viewed as a ``smoothing'' function for the decision landscape), is both
theoretically interesting and practically very useful.
  To support open science and other researchers working in this area, all our
scripts and datasets are available on-line at
https://github.com/yrahul3910/smoothness-hpo/.