Hyperparameter Adaptive Search for Surrogate Optimization: A
  Self-Adjusting Approach
Nazanin Nezami Hadis Anahideh
Surrogate Optimization (SO) algorithms have shown promise for optimizing
expensive black-box functions. However, their performance is heavily influenced
by hyperparameters related to sampling and surrogate fitting, which poses a
challenge to their widespread adoption. We investigate the impact of
hyperparameters on various SO algorithms and propose a Hyperparameter Adaptive
Search for SO (HASSO) approach. HASSO is not a hyperparameter tuning algorithm,
but a generic self-adjusting SO algorithm that dynamically tunes its own
hyperparameters while concurrently optimizing the primary objective function,
without requiring additional evaluations. The aim is to improve the
accessibility, effectiveness, and convergence speed of SO algorithms for
practitioners. Our approach identifies and modifies the most influential
hyperparameters specific to each problem and SO approach, reducing the need for
manual tuning without significantly increasing the computational burden.
Experimental results demonstrate the effectiveness of HASSO in enhancing the
performance of various SO algorithms across different global optimization test
problems.