<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 베이지안 최적화
gaussian process를 사용한 bayesian global optimization library
최대한 적은 iteration으로 최대값을 구하는 것이 목표
원리
최적화하려는 함수를 가장 살 설명하는 함수의 사후 분포(가우시안 프로세스)를 구성해 작동
관측치가 많아지면 사후 분포가 개선되고 파라미터 공간에서 탐색할 가치가 있는 영역과 그렇지 않은 영역이 더 명확해짐
<img src="https
Step1:  1. 최적화할 함수 정의
Step2:  2. 최적화 시작
Step3:  n_iter 
Step4:  최상의 조합은 optimizer.max로 확인 가능
Step5:  2.1 범위 수정
Step6:  3. 최적화 가이드
파라미터 공간에 아이디어가 있는 경우
probe(탐색)
기본적으로 lazy=True로 탐색
다음에 maximize를 호출할 때 평가함
파라미터를 딕셔너리로 넘길 수 있음
Step7:  4. 저장, 로딩, 재시작
BayesianOptimization 객체를 인스턴스화할 때 verbose>0으로 최적화 진행 상황 확인 가능
로깅, 경고 기능을 더 많이 사용하려면 observer를 사용해야 함
advanced tour notebook 참고
여기선 네이티브 JSONLogger 객체를 사용해 파일에 저장하고 진행 상황 로드하는 것을 알려줌
4.1 Saving progress
Step8:  4.2 Loading progress
Step9:  Advanced Part
사실 maximize는 suggest, probe, register의 wrapper임
최적화 루프를 더 잘 쓰고싶으면 위 3개를 알아야 함
Step10:  UtilityFUnction 인스턴스 필요
Step11:  suggest는 언제나 호출할 수 있음
아래 코드를 실행할 때마다 다른 값이 나옴
Step12:  평가
Step13:  어떤 값이 관측되었는지 알려주기
Step14:  1.1 maximize loop
Step15:  2. Discrete 파라미터 다루기
Step16:  3.1 파라미터를 GP로 전달
Step17:  maximize를 여러번 호출하거나 optimizing을 최대화 밖에서 할 경우엔 set_gp_params
Step19:  Observer 계속!
이벤트가 시작되면 콜백 함수가 이벤트와 함께 호출되고 BayesianOptimization 인스턴스가 매개 변수로 전달
Step20:  시각화
Step21:  <img src="https
Step29:  sklearn_example.py
Step33:  분산
<ASSISTANT_TASK:>
Python Code:
!pip3 install bayesian-optimization
Explanation: 베이지안 최적화
gaussian process를 사용한 bayesian global optimization library
최대한 적은 iteration으로 최대값을 구하는 것이 목표
원리
최적화하려는 함수를 가장 살 설명하는 함수의 사후 분포(가우시안 프로세스)를 구성해 작동
관측치가 많아지면 사후 분포가 개선되고 파라미터 공간에서 탐색할 가치가 있는 영역과 그렇지 않은 영역이 더 명확해짐
<img src="https://github.com/fmfn/BayesianOptimization/raw/master/examples/bo_example.png">
반복하면서 알고리즘은 target function에 대해 알고있던 것을 고려해 exploration과 exploitation을 조화시킴
각 단계에서 가우스 프로세스는 알려진 샘플에 fit되고 사후 분포는 탐사 전략(UCB), 또는 EI과 결합됨
<img src="https://github.com/fmfn/BayesianOptimization/raw/master/examples/bayesian_optimization.gif">
이 프로세스는 최적의 조합의 파라미터 조합을 찾는데 필요한 단계 수를 최소화 함
최적화할 함수를 샘플링하는 것이 매우 어려울 경우 적합
End of explanation
def black_box_function(x, y):
    return -x ** 2 - (y - 1) ** 2 + 1
Explanation: 1. 최적화할 함수 정의
End of explanation
from bayes_opt import BayesianOptimization
# 파라미터 경계 정의
pbounds = {'x': (2, 4), 'y': (-3, 3)}
optimizer = BayesianOptimization(
    f=black_box_function,
    pbounds=pbounds,
    verbose=2, # verbose = 1 prints only when a maximum is observed, verbose = 0 is silent
    random_state=1,
)
Explanation: 2. 최적화 시작
End of explanation
optimizer.maximize(
    init_points=2,
    n_iter=3,
)
optimizer.maximize(
    init_points=10,
    n_iter=3,
)
optimizer.maximize(
    init_points=2,
    n_iter=9,
)
Explanation: n_iter : 수행하려는 베이지안 최적화 단계. 더 많은 단계를 거치면 더 좋은 최대치 얻음
init_points : 수행할 무작위 탐색 단계
End of explanation
print(optimizer.max)
Explanation: 최상의 조합은 optimizer.max로 확인 가능
End of explanation
optimizer.set_bounds(new_bounds={"x": (-2, 3)})
optimizer.maximize(
    init_points=0,
    n_iter=5,
)
Explanation: 2.1 범위 수정
End of explanation
optimizer.probe(
    params={"x": 0.5, "y": 0.7},
    lazy=True,
)
print(optimizer.space.keys)
optimizer.probe(
    params=[-0.3, 0.1],
    lazy=True,
)
optimizer.maximize(init_points=0, n_iter=0)
Explanation: 3. 최적화 가이드
파라미터 공간에 아이디어가 있는 경우
probe(탐색)
기본적으로 lazy=True로 탐색
다음에 maximize를 호출할 때 평가함
파라미터를 딕셔너리로 넘길 수 있음
End of explanation
from bayes_opt.observer import JSONLogger
from bayes_opt.event import Events
logger = JSONLogger(path="./bayesian_logs.json")
optimizer.subscribe(Events.OPTMIZATION_STEP, logger)
optimizer.maximize(
    init_points=2,
    n_iter=3,
)
Explanation: 4. 저장, 로딩, 재시작
BayesianOptimization 객체를 인스턴스화할 때 verbose>0으로 최적화 진행 상황 확인 가능
로깅, 경고 기능을 더 많이 사용하려면 observer를 사용해야 함
advanced tour notebook 참고
여기선 네이티브 JSONLogger 객체를 사용해 파일에 저장하고 진행 상황 로드하는 것을 알려줌
4.1 Saving progress
End of explanation
from bayes_opt.util import load_logs
new_optimizer = BayesianOptimization(
    f=black_box_function,
    pbounds={"x": (-2, 2), "y": (-2, 2)},
    verbose=2,
    random_state=7,
)
print(len(new_optimizer.space))
load_logs(new_optimizer, logs=["./bayesian_logs.json"]);
print("New optimizer is now aware of {} points.".format(len(new_optimizer.space)))
new_optimizer.maximize(
    init_points=0,
    n_iter=10,
)
Explanation: 4.2 Loading progress
End of explanation
optimizer = BayesianOptimization(
    f=black_box_function,
    pbounds={'x': (-2, 2), 'y': (-3, 3)},
    verbose=2,
    random_state=1,
)
Explanation: Advanced Part
사실 maximize는 suggest, probe, register의 wrapper임
최적화 루프를 더 잘 쓰고싶으면 위 3개를 알아야 함
End of explanation
from bayes_opt import UtilityFunction
utility = UtilityFunction(kind="ucb", kappa=2.5, xi=0.0)
Explanation: UtilityFUnction 인스턴스 필요
End of explanation
next_point_to_probe = optimizer.suggest(utility)
print("Next point to probe is:", next_point_to_probe)
Explanation: suggest는 언제나 호출할 수 있음
아래 코드를 실행할 때마다 다른 값이 나옴
End of explanation
target = black_box_function(**next_point_to_probe)
print("Found the target value to be:", target)
Explanation: 평가
End of explanation
optimizer.register(
    params=next_point_to_probe,
    target=target,
)
Explanation: 어떤 값이 관측되었는지 알려주기
End of explanation
for _ in range(5):
    next_point = optimizer.suggest(utility)
    target = black_box_function(**next_point)
    optimizer.register(params=next_point, target=target)
    
    print(target, next_point)
print(optimizer.max)
Explanation: 1.1 maximize loop
End of explanation
def func_with_discrete_params(x, y, d):
    # Simulate necessity of having d being discrete.
    assert type(d) == int
    
    return ((x + y + d) // (1 + d)) / (1 + (x + y) ** 2)
def function_to_be_optimized(x, y, w):
    d = int(w)
    return func_with_discrete_params(x, y, d)
optimizer = BayesianOptimization(
    f=function_to_be_optimized,
    pbounds={'x': (-10, 10), 'y': (-10, 10), 'w': (0, 5)},
    verbose=2,
    random_state=1,
)
optimizer.maximize(alpha=1e-3)
Explanation: 2. Discrete 파라미터 다루기
End of explanation
optimizer = BayesianOptimization(
    f=black_box_function,
    pbounds={'x': (-2, 2), 'y': (-3, 3)},
    verbose=2,
    random_state=1,
)
optimizer.maximize(
    init_points=1,
    n_iter=5,
    # What follows are GP regressor parameters
    alpha=1e-3,
    n_restarts_optimizer=5
)
Explanation: 3.1 파라미터를 GP로 전달
End of explanation
optimizer.set_gp_params(normalize_y=True)
Explanation: maximize를 여러번 호출하거나 optimizing을 최대화 밖에서 할 경우엔 set_gp_params
End of explanation
from bayes_opt.event import DEFAULT_EVENTS, Events
optimizer = BayesianOptimization(
    f=black_box_function,
    pbounds={'x': (-2, 2), 'y': (-3, 3)},
    verbose=2,
    random_state=1,
)
class BasicObserver:
    def update(self, event, instance):
        Does whatever you want with the event and `BayesianOptimization` instance.
        print("Event `{}` was observed".format(event))
my_observer = BasicObserver()
optimizer.subscribe(
    event=Events.OPTMIZATION_STEP,
    subscriber=my_observer,
    callback=None, # Will use the `update` method as callback
)
def my_callback(event, instance):
    print("Go nuts here!")
optimizer.subscribe(
    event=Events.OPTMIZATION_START,
    subscriber="Any hashable object",
    callback=my_callback,
)
optimizer.maximize(init_points=1, n_iter=2)
DEFAULT_EVENTS
Explanation: Observer 계속!
이벤트가 시작되면 콜백 함수가 이벤트와 함께 호출되고 BayesianOptimization 인스턴스가 매개 변수로 전달
End of explanation
from bayes_opt import BayesianOptimization
from bayes_opt import UtilityFunction
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import gridspec
%matplotlib inline
Explanation: 시각화
End of explanation
def target(x):
    return np.exp(-(x - 2)**2) + np.exp(-(x - 6)**2/10) + 1/ (x**2 + 1)
x = np.linspace(-2, 10, 10000).reshape(-1, 1)
y = target(x)
plt.plot(x, y);
optimizer = BayesianOptimization(target, {'x': (-2, 10)}, random_state=27)
optimizer.maximize(init_points=2, n_iter=0, kappa=5)
def posterior(optimizer, x_obs, y_obs, grid):
    optimizer._gp.fit(x_obs, y_obs)
    mu, sigma = optimizer._gp.predict(grid, return_std=True)
    return mu, sigma
def plot_gp(optimizer, x, y):
    fig = plt.figure(figsize=(16, 10))
    steps = len(optimizer.space)
    fig.suptitle(
        'Gaussian Process and Utility Function After {} Steps'.format(steps),
        fontdict={'size':30}
    )
    
    gs = gridspec.GridSpec(2, 1, height_ratios=[3, 1]) 
    axis = plt.subplot(gs[0])
    acq = plt.subplot(gs[1])
    
    x_obs = np.array([[res["params"]["x"]] for res in optimizer.res])
    y_obs = np.array([res["target"] for res in optimizer.res])
    
    mu, sigma = posterior(optimizer, x_obs, y_obs, x)
    axis.plot(x, y, linewidth=3, label='Target')
    axis.plot(x_obs.flatten(), y_obs, 'D', markersize=8, label=u'Observations', color='r')
    axis.plot(x, mu, '--', color='k', label='Prediction')
    axis.fill(np.concatenate([x, x[::-1]]), 
              np.concatenate([mu - 1.9600 * sigma, (mu + 1.9600 * sigma)[::-1]]),
        alpha=.6, fc='c', ec='None', label='95% confidence interval')
    
    axis.set_xlim((-2, 10))
    axis.set_ylim((None, None))
    axis.set_ylabel('f(x)', fontdict={'size':20})
    axis.set_xlabel('x', fontdict={'size':20})
    
    utility_function = UtilityFunction(kind="ucb", kappa=5, xi=0)
    utility = utility_function.utility(x, optimizer._gp, 0)
    acq.plot(x, utility, label='Utility Function', color='purple')
    acq.plot(x[np.argmax(utility)], np.max(utility), '*', markersize=15, 
             label=u'Next Best Guess', markerfacecolor='gold', markeredgecolor='k', markeredgewidth=1)
    acq.set_xlim((-2, 10))
    acq.set_ylim((0, np.max(utility) + 0.5))
    acq.set_ylabel('Utility', fontdict={'size':20})
    acq.set_xlabel('x', fontdict={'size':20})
    
    axis.legend(loc=2, bbox_to_anchor=(1.01, 1), borderaxespad=0.)
    acq.legend(loc=2, bbox_to_anchor=(1.01, 1), borderaxespad=0.)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
optimizer.maximize(init_points=0, n_iter=1, kappa=5)
plot_gp(optimizer, x, y)
Explanation: <img src="https://render.githubusercontent.com/render/math?math=f%28x%29%20%3D%20e%5E%7B-%28x%20-%202%29%5E2%7D%20%2B%20e%5E%7B-%5Cfrac%7B%28x%20-%206%29%5E2%7D%7B10%7D%7D%20%2B%20%5Cfrac%7B1%7D%7Bx%5E2%20%2B%201%7D%2C&mode=display">
End of explanation
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier as RFC
from sklearn.svm import SVC
from bayes_opt import BayesianOptimization
from bayes_opt.util import Colours
def get_data():
    Synthetic binary classification dataset.
    data, targets = make_classification(
        n_samples=1000,
        n_features=45,
        n_informative=12,
        n_redundant=7,
        random_state=134985745,
    )
    return data, targets
def svc_cv(C, gamma, data, targets):
    SVC cross validation.
    This function will instantiate a SVC classifier with parameters C and
    gamma. Combined with data and targets this will in turn be used to perform
    cross validation. The result of cross validation is returned.
    Our goal is to find combinations of C and gamma that maximizes the roc_auc
    metric.
    
    estimator = SVC(C=C, gamma=gamma, random_state=2)
    cval = cross_val_score(estimator, data, targets, scoring='roc_auc', cv=4)
    return cval.mean()
def rfc_cv(n_estimators, min_samples_split, max_features, data, targets):
    Random Forest cross validation.
    This function will instantiate a random forest classifier with parameters
    n_estimators, min_samples_split, and max_features. Combined with data and
    targets this will in turn be used to perform cross validation. The result
    of cross validation is returned.
    Our goal is to find combinations of n_estimators, min_samples_split, and
    max_features that minimzes the log loss.
    
    estimator = RFC(
        n_estimators=n_estimators,
        min_samples_split=min_samples_split,
        max_features=max_features,
        random_state=2
    )
    cval = cross_val_score(estimator, data, targets,
                           scoring='neg_log_loss', cv=4)
    return cval.mean()
def optimize_svc(data, targets):
    Apply Bayesian Optimization to SVC parameters.
    def svc_crossval(expC, expGamma):
        Wrapper of SVC cross validation.
        Notice how we transform between regular and log scale. While this
        is not technically necessary, it greatly improves the performance
        of the optimizer.
        
        C = 10 ** expC
        gamma = 10 ** expGamma
        return svc_cv(C=C, gamma=gamma, data=data, targets=targets)
    optimizer = BayesianOptimization(
        f=svc_crossval,
        pbounds={"expC": (-3, 2), "expGamma": (-4, -1)},
        random_state=1234,
        verbose=2
    )
    optimizer.maximize(n_iter=10)
    print("Final result:", optimizer.max)
def optimize_rfc(data, targets):
    Apply Bayesian Optimization to Random Forest parameters.
    def rfc_crossval(n_estimators, min_samples_split, max_features):
        Wrapper of RandomForest cross validation.
        Notice how we ensure n_estimators and min_samples_split are casted
        to integer before we pass them along. Moreover, to avoid max_features
        taking values outside the (0, 1) range, we also ensure it is capped
        accordingly.
        
        return rfc_cv(
            n_estimators=int(n_estimators),
            min_samples_split=int(min_samples_split),
            max_features=max(min(max_features, 0.999), 1e-3),
            data=data,
            targets=targets,
        )
    optimizer = BayesianOptimization(
        f=rfc_crossval,
        pbounds={
            "n_estimators": (10, 250),
            "min_samples_split": (2, 25),
            "max_features": (0.1, 0.999),
        },
        random_state=1234,
        verbose=2
    )
    optimizer.maximize(n_iter=10)
    print("Final result:", optimizer.max)
if __name__ == "__main__":
    data, targets = get_data()
    print(Colours.yellow("--- Optimizing SVM ---"))
    optimize_svc(data, targets)
    print(Colours.green("--- Optimizing Random Forest ---"))
    optimize_rfc(data, targets)
Explanation: sklearn_example.py
End of explanation
import time
import random
from bayes_opt import BayesianOptimization
from bayes_opt.util import UtilityFunction, Colours
import asyncio
import threading
try:
    import json
    import tornado.ioloop
    import tornado.httpserver
    from tornado.web import RequestHandler
    import requests
except ImportError:
    raise ImportError(
        "In order to run this example you must have the libraries: " +
        "`tornado` and `requests` installed."
    )
def black_box_function(x, y):
    Function with unknown internals we wish to maximize.
    This is just serving as an example, however, for all intents and
    purposes think of the internals of this function, i.e.: the process
    which generates its outputs values, as unknown.
    
    time.sleep(random.randint(1, 7))
    return -x ** 2 - (y - 1) ** 2 + 1
class BayesianOptimizationHandler(RequestHandler):
    Basic functionality for NLP handlers.
    _bo = BayesianOptimization(
        f=black_box_function,
        pbounds={"x": (-4, 4), "y": (-3, 3)}
    )
    _uf = UtilityFunction(kind="ucb", kappa=3, xi=1)
    def post(self):
        Deal with incoming requests.
        body = tornado.escape.json_decode(self.request.body)
        try:
            self._bo.register(
                params=body["params"],
                target=body["target"],
            )
            print("BO has registered: {} points.".format(len(self._bo.space)), end="\n\n")
        except KeyError:
            pass
        finally:
            suggested_params = self._bo.suggest(self._uf)
        self.write(json.dumps(suggested_params))
def run_optimization_app():
    asyncio.set_event_loop(asyncio.new_event_loop())
    handlers = [
        (r"/bayesian_optimization", BayesianOptimizationHandler),
    ]
    server = tornado.httpserver.HTTPServer(
        tornado.web.Application(handlers)
    )
    server.listen(9009)
    tornado.ioloop.IOLoop.instance().start()
def run_optimizer():
    global optimizers_config
    config = optimizers_config.pop()
    name = config["name"]
    colour = config["colour"]
    register_data = {}
    max_target = None
    for _ in range(10):
        status = name + " wants to register: {}.\n".format(register_data)
        resp = requests.post(
            url="http://localhost:9009/bayesian_optimization",
            json=register_data,
        ).json()
        target = black_box_function(**resp)
        register_data = {
            "params": resp,
            "target": target,
        }
        if max_target is None or target > max_target:
            max_target = target
        status += name + " got {} as target.\n".format(target)
        status += name + " will to register next: {}.\n".format(register_data)
        print(colour(status), end="\n")
    global results
    results.append((name, max_target))
    print(colour(name + " is done!"), end="\n\n")
if __name__ == "__main__":
    ioloop = tornado.ioloop.IOLoop.instance()
    optimizers_config = [
        {"name": "optimizer 1", "colour": Colours.red},
        {"name": "optimizer 2", "colour": Colours.green},
        {"name": "optimizer 3", "colour": Colours.blue},
    ]
    app_thread = threading.Thread(target=run_optimization_app)
    app_thread.daemon = True
    app_thread.start()
    targets = (
        run_optimizer,
        run_optimizer,
        run_optimizer
    )
    optimizer_threads = []
    for target in targets:
        optimizer_threads.append(threading.Thread(target=target))
        optimizer_threads[-1].daemon = True
        optimizer_threads[-1].start()
    results = []
    for optimizer_thread in optimizer_threads:
        optimizer_thread.join()
    for result in results:
        print(result[0], "found a maximum value of: {}".format(result[1]))
    ioloop.stop()
Explanation: 분산
End of explanation
<END_TASK>