<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Running Tune experiments with Skopt
In this tutorial we introduce Skopt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Skopt and, as a result, allow you to seamlessly scale up a Skopt optimization process - without sacrificing performance.
Scikit-Optimize, or skopt, is a simple and efficient library to optimize expensive and noisy black-box functions, e.g. large-scale ML experiments. It implements several methods for sequential model-based optimization. Noteably, skopt does not perform gradient-based optimization, and instead uses computationally cheap surrogate models to
approximate the expensive function. In this example we minimize a simple objective to briefly demonstrate the usage of Skopt with Ray Tune via SkOptSearch. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume scikit-opitmize==0.8.1 library is installed. To learn more, please refer to the Scikit-Optimize website.
Step1:  Click below to see all the imports we need for this example.
You can also launch directly into a Binder instance to run this notebook yourself.
Just click on the rocket symbol at the top of the navigation.
Step2:  Let's start by defining a simple evaluation function. Again, an explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we're running multiple steps of an experiment while tuning three hyperparameters, namely width, height, and activation.
Step3:  Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop,
and uses tune.report to report the score back to Tune.
Step4:  Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
Step5:  The search algorithm is instantiated from the SkOptSearch class. We also constrain the the number of concurrent trials to 4 with a ConcurrencyLimiter.
Step6:  The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
Step7:  Finally, we run the experiment to "min"imize the "mean_loss" of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tune.run().
Step8:  We now have hyperparameters found to minimize the mean loss.
Step9:  Providing an initial set of hyperparameters
While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and
pass this information as a helpful starting point for the SkOptSearch object. We also can pass the known rewards for these initial params to save on unnecessary computation.
Step10:  Now the search_alg built using SkOptSearch takes points_to_evaluate.
Step11:  And again run the experiment, this time with initial hyperparameter evaluations
Step12:  And we again show the ideal hyperparameters.
<ASSISTANT_TASK:>
Python Code:
# !pip install ray[tune]
!pip install scikit-optimize==0.8.1
!pip install sklearn==0.18.2
Explanation: Running Tune experiments with Skopt
In this tutorial we introduce Skopt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Skopt and, as a result, allow you to seamlessly scale up a Skopt optimization process - without sacrificing performance.
Scikit-Optimize, or skopt, is a simple and efficient library to optimize expensive and noisy black-box functions, e.g. large-scale ML experiments. It implements several methods for sequential model-based optimization. Noteably, skopt does not perform gradient-based optimization, and instead uses computationally cheap surrogate models to
approximate the expensive function. In this example we minimize a simple objective to briefly demonstrate the usage of Skopt with Ray Tune via SkOptSearch. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume scikit-opitmize==0.8.1 library is installed. To learn more, please refer to the Scikit-Optimize website.
End of explanation
import time
from typing import Dict, Optional, Any
import ray
import skopt
from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.suggest.skopt import SkOptSearch
ray.init(configure_logging=False)
Explanation: Click below to see all the imports we need for this example.
You can also launch directly into a Binder instance to run this notebook yourself.
Just click on the rocket symbol at the top of the navigation.
End of explanation
def evaluate(step, width, height, activation):
    time.sleep(0.1)
    activation_boost = 10 if activation=="relu" else 0
    return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Explanation: Let's start by defining a simple evaluation function. Again, an explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we're running multiple steps of an experiment while tuning three hyperparameters, namely width, height, and activation.
End of explanation
def objective(config):
    for step in range(config["steps"]):
        score = evaluate(step, config["width"], config["height"], config["activation"])
        tune.report(iterations=step, mean_loss=score)
Explanation: Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop,
and uses tune.report to report the score back to Tune.
End of explanation
search_space = {
    "steps": 100,
    "width": tune.uniform(0, 20),
    "height": tune.uniform(-100, 100),
    "activation": tune.choice(["relu", "tanh"]),
}
Explanation: Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
End of explanation
algo = SkOptSearch()
algo = ConcurrencyLimiter(algo, max_concurrent=4)
Explanation: The search algorithm is instantiated from the SkOptSearch class. We also constrain the the number of concurrent trials to 4 with a ConcurrencyLimiter.
End of explanation
num_samples = 1000
# We override here for our smoke tests.
num_samples = 10
Explanation: The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
End of explanation
analysis = tune.run(
    objective,
    search_alg=algo,
    metric="mean_loss",
    mode="min",
    name="skopt_exp",
    num_samples=num_samples,
    config=search_space
)
Explanation: Finally, we run the experiment to "min"imize the "mean_loss" of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tune.run().
End of explanation
print("Best hyperparameters found were: ", analysis.best_config)
Explanation: We now have hyperparameters found to minimize the mean loss.
End of explanation
initial_params = [
    {"width": 10, "height": 0, "activation": "relu"},
    {"width": 15, "height": -20, "activation": "tanh"}
]
known_rewards = [-189, -1144]
Explanation: Providing an initial set of hyperparameters
While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and
pass this information as a helpful starting point for the SkOptSearch object. We also can pass the known rewards for these initial params to save on unnecessary computation.
End of explanation
algo = SkOptSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(algo, max_concurrent=4)
Explanation: Now the search_alg built using SkOptSearch takes points_to_evaluate.
End of explanation
analysis = tune.run(
    objective,
    search_alg=algo,
    metric="mean_loss",
    mode="min",
    name="skopt_exp_with_warmstart",
    num_samples=num_samples,
    config=search_space
)
Explanation: And again run the experiment, this time with initial hyperparameter evaluations:
End of explanation
print("Best hyperparameters found were: ", analysis.best_config)
ray.shutdown()
Explanation: And we again show the ideal hyperparameters.
End of explanation
<END_TASK>