A Multi-objective Newton Optimization Algorithm for Hyper-Parameter
  Search
Qinwu Xu
This study proposes a Newton based multiple objective optimization algorithm
for hyperparameter search. The first order differential (gradient) is
calculated using finite difference method and a gradient matrix with
vectorization is formed for fast computation. The Newton Raphson iterative
solution is used to update model parameters with iterations, and a
regularization term is included to eliminate the singularity issue. The
algorithm is applied to search the optimal probability threshold (a vector of
eight parameters) for a multiclass object detection problem of a convolutional
neural network. The algorithm quickly finds the improved parameter values to
produce an overall higher true positive (TP) and lower false positive (FP)
rates, as compared to using the default value of 0.5. In comparison, the
Bayesian optimization generates lower performance in the testing case. However,
the performance and parameter values may oscillate for some cases during
iterations, which may be due to the data driven stochastic nature of the
subject. Therefore, the optimal parameter value can be identified from a list
of iteration steps according to the optimal TP and FP results.