<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Playing with Regression
This notebook will help us with testing different regression techniques, and demonstrate the diagnostic class which can be used to find the optimal parameters for COBRA.
So for now we will generate a random data-set and try some of the popular regression techniques on it, after it has been loaded to COBRA.
Imports
Step1:  Setting up data set
Step2:  Setting up COBRA
Let's up our COBRA machine with the data.
Step3:  When we are fitting, we initialise COBRA with an epsilon value of $0.5$ - this is because we are aware of the distribution and 0.5 is a fair guess of what would be a "good" epsilon value, because the data varies from $-1$ to $1$. 
If we do not pass the $\epsilon$ parameter, we perform a CV on the training data for an optimised epsilon.
It can be noticed that the default parameter is set as false
Step4:  Let's load the default machines to COBRA.
Step5:  We note here that further machines can be loaded using either the loadMachine() and loadSKMachine() methods. The only prerequisite is that the machine has a valid predict() method.
Using COBRA's machines
We've created our random dataset and now we're going to use the default sci-kit machines to see what the results look like.
Step6:  Aggregate!
By using the aggregate function we can combine our predictors.
You can read about the aggregation procedure either in the original COBRA paper or look around in the source code for the algorithm.
We start by loading each machine's predictions now.
Step7:  Optimizing COBRA
To squeeze the best out of COBRA we make use of the COBRA diagnostics class. With a grid based approach to optimizing hyperparameters, we can find out the best epsilon value, number of machines (alpha value), and combination of machines.
Let's check the MSE for each of COBRAs machines
Step8:  This error is bound by the value $C\mathscr{l}^{\frac{-2}{M + 2}}$ upto a constant $C$, which is problem dependant. For more details, we refer the user to the original paper.
Step9:  Playing with Data-Splitting
When we initially started to set up COBRA, we split our training data into two further parts - $D_k$, and $D_l$. 
This split was done 50-50, but it is upto us how we wish to do this. 
The following section will compare 20-80, 60-40, 50-50, 40-60, 80-20 and check for which case we get the best MSE values, for a fixed Epsilon (or use a grid).
Step10:  What we saw was the default result, with the optimal split ratio and the corresponding MSE. We can do a further analysis here by enabling the info and graph options, and using more values to split on.
Step11:  Alpha, Epsilon and Machines
The following are methods to idetify the optimal epislon values, alpha values, and combination of machines. 
The grid methods allow for us to predict for a single point the optimal alpha/machines and epsilon combination.
Epsilon
The epsilon paramter controls how "strict" cobra should behave while choosing the points to aggregate.
Step12:  Alpha
The alpha parameter decides how many machines must a point be picked up by before being added to an aggregate. The default value is 4.
Step13:  In this particular case, the best performance is obtained by seeking consensus over all 4 machines.
Machines
Decide which subset of machines to select for the aggregate.
<ASSISTANT_TASK:>
Python Code:
from pycobra.cobra import Cobra
from pycobra.diagnostics import Diagnostics
import numpy as np
%matplotlib inline
Explanation: Playing with Regression
This notebook will help us with testing different regression techniques, and demonstrate the diagnostic class which can be used to find the optimal parameters for COBRA.
So for now we will generate a random data-set and try some of the popular regression techniques on it, after it has been loaded to COBRA.
Imports
End of explanation
# setting up our random data-set
rng = np.random.RandomState(1)
# D1 = train machines; D2 = create COBRA; D3 = calibrate epsilon, alpha; D4 = testing
n_features = 20
D1, D2, D3, D4 = 200, 200, 200, 200
D = D1 + D2 + D3 + D4
X = rng.uniform(-1, 1, D * n_features).reshape(D, n_features)
Y = np.power(X[:,1], 2) + np.power(X[:,3], 3) + np.exp(X[:,10]) 
# Y = np.power(X[:,0], 2) + np.power(X[:,1], 3)
# training data-set
X_train = X[:D1 + D2]
X_test = X[D1 + D2 + D3:D1 + D2 + D3 + D4]
X_eps = X[D1 + D2:D1 + D2 + D3]
# for testing
Y_train = Y[:D1 + D2]
Y_test = Y[D1 + D2 + D3:D1 + D2 + D3 + D4]
Y_eps = Y[D1 + D2:D1 + D2 + D3]
Explanation: Setting up data set
End of explanation
cobra = Cobra(random_state=0, epsilon=0.5)
cobra.fit(X_train, Y_train, default=False)
Explanation: Setting up COBRA
Let's up our COBRA machine with the data.
End of explanation
cobra.split_data(D1, D1 + D2, shuffle_data=True)
Explanation: When we are fitting, we initialise COBRA with an epsilon value of $0.5$ - this is because we are aware of the distribution and 0.5 is a fair guess of what would be a "good" epsilon value, because the data varies from $-1$ to $1$. 
If we do not pass the $\epsilon$ parameter, we perform a CV on the training data for an optimised epsilon.
It can be noticed that the default parameter is set as false: this is so we can walk you through what happens when COBRA is set-up, instead of the deafult settings being used.
We're now going to split our dataset into two parts, and shuffle data points.
End of explanation
cobra.load_default()
Explanation: Let's load the default machines to COBRA.
End of explanation
query = X_test[9].reshape(1, -1)
cobra.machines_
cobra.machines_['lasso'].predict(query)
cobra.machines_['tree'].predict(query)
cobra.machines_['ridge'].predict(query)
cobra.machines_['random_forest'].predict(query)
Explanation: We note here that further machines can be loaded using either the loadMachine() and loadSKMachine() methods. The only prerequisite is that the machine has a valid predict() method.
Using COBRA's machines
We've created our random dataset and now we're going to use the default sci-kit machines to see what the results look like.
End of explanation
cobra.load_machine_predictions()
cobra.predict(query)
Y_test[9]
Explanation: Aggregate!
By using the aggregate function we can combine our predictors.
You can read about the aggregation procedure either in the original COBRA paper or look around in the source code for the algorithm.
We start by loading each machine's predictions now.
End of explanation
cobra_diagnostics = Diagnostics(cobra, X_test, Y_test, load_MSE=True)
cobra_diagnostics.machine_MSE
Explanation: Optimizing COBRA
To squeeze the best out of COBRA we make use of the COBRA diagnostics class. With a grid based approach to optimizing hyperparameters, we can find out the best epsilon value, number of machines (alpha value), and combination of machines.
Let's check the MSE for each of COBRAs machines:
End of explanation
cobra_diagnostics.error_bound
Explanation: This error is bound by the value $C\mathscr{l}^{\frac{-2}{M + 2}}$ upto a constant $C$, which is problem dependant. For more details, we refer the user to the original paper.
End of explanation
cobra_diagnostics.optimal_split(X_eps, Y_eps)
Explanation: Playing with Data-Splitting
When we initially started to set up COBRA, we split our training data into two further parts - $D_k$, and $D_l$. 
This split was done 50-50, but it is upto us how we wish to do this. 
The following section will compare 20-80, 60-40, 50-50, 40-60, 80-20 and check for which case we get the best MSE values, for a fixed Epsilon (or use a grid).
End of explanation
split = [(0.05, 0.95), (0.10, 0.90), (0.20, 0.80), (0.40, 0.60), (0.50, 0.50), (0.60, 0.40), (0.80, 0.20), (0.90, 0.10), (0.95, 0.05)]
cobra_diagnostics.optimal_split(X_eps, Y_eps, split=split, info=True, graph=True)
Explanation: What we saw was the default result, with the optimal split ratio and the corresponding MSE. We can do a further analysis here by enabling the info and graph options, and using more values to split on.
End of explanation
cobra_diagnostics.optimal_epsilon(X_eps, Y_eps, line_points=100)
Explanation: Alpha, Epsilon and Machines
The following are methods to idetify the optimal epislon values, alpha values, and combination of machines. 
The grid methods allow for us to predict for a single point the optimal alpha/machines and epsilon combination.
Epsilon
The epsilon paramter controls how "strict" cobra should behave while choosing the points to aggregate.
End of explanation
cobra_diagnostics.optimal_alpha(X_eps, Y_eps, info=True)
Explanation: Alpha
The alpha parameter decides how many machines must a point be picked up by before being added to an aggregate. The default value is 4.
End of explanation
cobra_diagnostics.optimal_machines(X_eps, Y_eps, info=True)
cobra_diagnostics.optimal_alpha_grid(X_eps[0], Y_eps[0], line_points=100)
cobra_diagnostics.optimal_machines_grid(X_eps[0], Y_eps[0], line_points=100)
Explanation: In this particular case, the best performance is obtained by seeking consensus over all 4 machines.
Machines
Decide which subset of machines to select for the aggregate.
End of explanation
<END_TASK>