<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Hyperparameter optimization using pyGPGO
by José Jiménez (Oct 18, 2017)
In this tutorial, we will learn the basics of the Bayesian optimization (BO) framework through a step-by-step example in the context of optimizing the hyperparameters of a binary classifier. But first of all, where is Bayesian optimization useful?
There are a lot of case scenarios, one would typically use the BO framework in situations like
Step1:  Before going any further, let's visualize it!
Step2:  Let's say that we want to use a Support Vector Machine (SVM) with the radial basis function kernel classifier on this data, which has two usual parameters to optimize, $C$ and $\gamma$. We need to first define a target function that takes these two hyperparameters as input and spits out an error (e.g, using some form of cross validation). Define also a dictionary, specifying parameters and input spaces for each.
Step3:  Now comes the fun part, where we specify our BO framework using pyGPGO. We are going to use a Gaussian Process (GP) model to approximate our true objective function, and a covariance function that measures similarity among training examples. An excellent introduction to Gaussian Process regression can be found in [@Rassmussen-Williams2004]. We are going to use the squared exponential kernel for this example, that takes the form
Step4:  We specify now an acquisition function, that will determine the behaviour of the BO procedure when selecting a new point. For instance, it is very common to use the Expected Improvement (EI) acquisition, that will both take into account the probability of improvement of a point and its magnitude
Step5:  We're almost done! Finally call the GPGO class and put everything together. We'll run the procedure for 20 epochs.
Step6:  Finally retrieve your result!
<ASSISTANT_TASK:>
Python Code:
import numpy as np
from sklearn.datasets import make_moons
np.random.seed(20)
X, y = make_moons(n_samples = 200, noise = 0.3) # Data and target
Explanation: Hyperparameter optimization using pyGPGO
by José Jiménez (Oct 18, 2017)
In this tutorial, we will learn the basics of the Bayesian optimization (BO) framework through a step-by-step example in the context of optimizing the hyperparameters of a binary classifier. But first of all, where is Bayesian optimization useful?
There are a lot of case scenarios, one would typically use the BO framework in situations like:
The objective function has no closed-form
No gradient information is available
In presence of noise
The BO framework uses a surrogate model to approximate the objective function and chooses to optimize it instead according to a chosen criteria. For an in-depth introduction to the topic, we whole-heartedly recommend reading [@Snoek2012, @Jimenez2017].
Let's start by creating some synthetic data that we will use later for classification.
End of explanation
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
cm_bright = ListedColormap(['#fc4349', '#6dbcdb'])
    
fig = plt.figure()
plt.scatter(X[:, 0], X[:, 1], c = y, cmap = cm_bright)
plt.show()
Explanation: Before going any further, let's visualize it!
End of explanation
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
def evaluateModel(C, gamma):
    clf = SVC(C=10**C, gamma=10**gamma)
    return np.average(cross_val_score(clf, X, y))
params = {'C':      ('cont', (-4, 5)),
          'gamma':  ('cont', (-4, 5))
         }
Explanation: Let's say that we want to use a Support Vector Machine (SVM) with the radial basis function kernel classifier on this data, which has two usual parameters to optimize, $C$ and $\gamma$. We need to first define a target function that takes these two hyperparameters as input and spits out an error (e.g, using some form of cross validation). Define also a dictionary, specifying parameters and input spaces for each.
End of explanation
from pyGPGO.surrogates.GaussianProcess import GaussianProcess
from pyGPGO.covfunc import squaredExponential
sexp = squaredExponential()
gp = GaussianProcess(sexp)
Explanation: Now comes the fun part, where we specify our BO framework using pyGPGO. We are going to use a Gaussian Process (GP) model to approximate our true objective function, and a covariance function that measures similarity among training examples. An excellent introduction to Gaussian Process regression can be found in [@Rassmussen-Williams2004]. We are going to use the squared exponential kernel for this example, that takes the form:
$$k(r) = \exp\left(-\dfrac{r^2}{2l^2} \right)$$,
where $r = |x - x'|$ two examples.
End of explanation
from pyGPGO.acquisition import Acquisition
acq = Acquisition(mode = 'ExpectedImprovement')
Explanation: We specify now an acquisition function, that will determine the behaviour of the BO procedure when selecting a new point. For instance, it is very common to use the Expected Improvement (EI) acquisition, that will both take into account the probability of improvement of a point and its magnitude:
End of explanation
from pyGPGO.GPGO import GPGO
gpgo = GPGO(gp, acq, evaluateModel, params)
gpgo.run(max_iter = 20)
Explanation: We're almost done! Finally call the GPGO class and put everything together. We'll run the procedure for 20 epochs.
End of explanation
gpgo.getResult()
Explanation: Finally retrieve your result!
End of explanation
<END_TASK>