<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Performance Benchmarking for KV Drive
The goal of these set of experiments is to characterize the variability across platforms in a systematic and consistent way in terms of KV drive. The steps of experiments are as follows,
Run Stress-ng benchmarks on one KV drive;
Run Stress-ng benchmarks on machine issdm-6, and get the "without limit" result;
Find all the common benchmarks from both results;
Calculate the speedup (normalized value) of each benchmark based on the one from KV drive (issdm-6 (without limit) / KV drive);
Use torpor to calculate the best cpu quota by minimizing the average speedups. We will later use this parameter to limit the cpu usage in the docker container;
Run Stress-ng benchmarks in the constrained docker container on machine issdm-6, and get the "with limit" result;
Calculate the speedup based on KV drive again (issdm-6 (with limit) / KV drive), then we get a new "speedup range", which should be must smaller than the previous one.
Run a bunch of other benchmarks on both KV drive and constrained docker container to verify if they are all within in the later "speedup range".
Make conclusion.
Step1:  First, we load all test data.
Step2:  Let's have a look at the pattern of data.
Step3:  Show all the test machines.
Step4:  Define some predicates for machines and limits
Step5:  Show the number of stress tests on different machines
Step6:  Because those failed benchmarks are not shown in the result report, we want to know how many common successful stress tests on the target machine and kv3.
Step7:  Read the normalized results.
Step8:  Show some of the data lines. The normalized value is the speedup based on kv3. It becomes a negative value when the benchmark runs on the target machine is slower than on kv3 (slowdown).
Step9:  Show those benchmarks are not both successful completed on the issdm-6 and kv3.
Step10:  Show those benchmarks are not both successful completed on the t2.micro and kv3.
Step11:  We can find the number of benchmarks are speed-up and slowdown, respectively.
Step12:  The average of normalized value for results under CPU restriction
Step13:  Experiment Results from issdm-6
Let's have a look at the histogram of frequency of normalized value based on  stress tests without CPU restriction running on issdm-6.
Step14:  Here is the rank of normalized value from stress tests without CPU restriction
Step15:  Now let's have a look at the histogram of frequency of normalized value based on  stress tests with CPU restriction running on issdm-6.
Step16:  Here is the rank of normalized value from stress tests with CPU restriction
Step17:  We notice that the stressng-cpu-jenkin looks like an outlier. Let's redraw the histogram without this one.
Step18:  Summary
We got the boundary of normalized value on issdm-6 from -29.394675 to 54.266945 by using parameters --cpuset-cpus=1 --cpu-quota=7234 --cpu-period=100000, which means the docker container only uses 7.234ms CPU worth of run-time every 100ms on cpu 1 (See cpu for more details).
Experiment Results from t2.micro
Let's have a look at the histogram of frequency of normalized value based on  stress tests without CPU restriction running on t2.micro.
Step19:  Here is the rank of normalized value from stress tests without CPU restriction
Step20:  Let's have a look at the histogram of frequency of normalized value based on  stress tests with CPU restriction running on t2.micro.
Step21:  Here is the rank of normalized value from stress tests with CPU restriction
Step22:  We notice that the stressng-memory-stack looks like an outlier. Let's redraw the histogram without this one.
Step23:  The stressng-cpu-jenkin benchmark is a collection of (non-cryptographic) hash functions for multi-byte keys. See Jenkins hash function from Wikipedia for more details.
Summary
We got the boundary of normalized value on t2.micro from -198.440535 to 119.904761 by using parameters --cpuset-cpus=0 --cpu-quota=25750 --cpu-period=100000, which means the docker container only uses 7.234ms CPU worth of run-time every 100ms on cpu 0 (See cpu for more details).
Verification
Now we use 9 other benchmark programs to verify this result. These programs are,
- blogbench
Step24:  Show number of test benchmarks.
Step25:  Order the test results by the absolute of normalized value
Step26:  Verification Tests on issdm-6
Histogram of frequency of normalized value.
Step27:  Print the max the min normalized value,
Step28:  The average of noramlized value is,
Step29:  If we remove all nbench tests, the frequency histogram changes to
Step30:  The max the min normalized value changes to,
Step31:  The average of noramlized value changes to,
Step32:  Verification Tests on t2.micro
Histogram of frequency of normalized value.
Step33:  The average of noramlized value of the verification benchmarks is,
Step34:  Let's see the frequency histogram after removing right-most four outliers.
Step35:  Print the max the min normalized value,
Step36:  The average of noramlized value without the four outliners is,
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import random
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
pd.set_option("display.max_rows", 8)
Explanation: Performance Benchmarking for KV Drive
The goal of these set of experiments is to characterize the variability across platforms in a systematic and consistent way in terms of KV drive. The steps of experiments are as follows,
Run Stress-ng benchmarks on one KV drive;
Run Stress-ng benchmarks on machine issdm-6, and get the "without limit" result;
Find all the common benchmarks from both results;
Calculate the speedup (normalized value) of each benchmark based on the one from KV drive (issdm-6 (without limit) / KV drive);
Use torpor to calculate the best cpu quota by minimizing the average speedups. We will later use this parameter to limit the cpu usage in the docker container;
Run Stress-ng benchmarks in the constrained docker container on machine issdm-6, and get the "with limit" result;
Calculate the speedup based on KV drive again (issdm-6 (with limit) / KV drive), then we get a new "speedup range", which should be must smaller than the previous one.
Run a bunch of other benchmarks on both KV drive and constrained docker container to verify if they are all within in the later "speedup range".
Make conclusion.
End of explanation
df = pd.read_csv('stress-ng/third/torpor-results/alltests.csv')
Explanation: First, we load all test data.
End of explanation
df.head()
Explanation: Let's have a look at the pattern of data.
End of explanation
df['machine'].unique()
Explanation: Show all the test machines.
End of explanation
machine_is_issdm_6 = df['machine'] == 'issdm-6'
machine_is_t2_micro = df['machine'] == 't2.micro'
machine_is_kv3 = df['machine'] == 'kv3'
limits_is_with = df['limits'] == 'with'
limits_is_without = df['limits'] == 'without'
Explanation: Define some predicates for machines and limits
End of explanation
df_issdm_6_with_limit  = df[machine_is_issdm_6 & limits_is_with]
df_t2_micro_with_limit = df[machine_is_t2_micro & limits_is_with]
df_kv3_without_limit   = df[machine_is_kv3 & limits_is_without]
print(
    len(df_issdm_6_with_limit),                       # machine issdm-6 with limit
    len(df[machine_is_issdm_6 & limits_is_without]),  # machine issdm-6 without limit
    len(df_t2_micro_with_limit),                      # machine t2.micro with limit
    len(df[machine_is_t2_micro & limits_is_without]), # machine t2.micro without limit
    len(df_kv3_without_limit)                         # machine kv3 without limit
)
Explanation: Show the number of stress tests on different machines
End of explanation
issdm_6_with_limit_merge_kv3 = pd.merge(df_issdm_6_with_limit, df_kv3_without_limit, how='inner', on='benchmark')
t2_micro_with_limit_merge_kv3 = pd.merge(df_t2_micro_with_limit, df_kv3_without_limit, how='inner', on='benchmark')
print(
    # common successful tests from issdm-6 and kv3
    len(issdm_6_with_limit_merge_kv3),
    
    # common successful tests from t2.micro and kv3
    len(t2_micro_with_limit_merge_kv3)
)
Explanation: Because those failed benchmarks are not shown in the result report, we want to know how many common successful stress tests on the target machine and kv3.
End of explanation
df_normalized = pd.read_csv('stress-ng/third/torpor-results/alltests_with_normalized_results_1.1.csv')
Explanation: Read the normalized results.
End of explanation
df_normalized.head()
Explanation: Show some of the data lines. The normalized value is the speedup based on kv3. It becomes a negative value when the benchmark runs on the target machine is slower than on kv3 (slowdown).
End of explanation
df_issdm_6_with_limit[~df_issdm_6_with_limit['benchmark'].isin(issdm_6_with_limit_merge_kv3['benchmark'])]
Explanation: Show those benchmarks are not both successful completed on the issdm-6 and kv3.
End of explanation
df_t2_micro_with_limit[~df_t2_micro_with_limit['benchmark'].isin(t2_micro_with_limit_merge_kv3['benchmark'])]
Explanation: Show those benchmarks are not both successful completed on the t2.micro and kv3.
End of explanation
normalized_limits_is_with = df_normalized['limits'] == 'with'
normalized_limits_is_without = df_normalized['limits'] == 'without'
normalized_machine_is_issdm_6 = df_normalized['machine'] == 'issdm-6'
normalized_machine_is_t2_micro = df_normalized['machine'] == 't2.micro'
normalized_is_speed_up = df_normalized['normalized'] > 0
normalized_is_slow_down = df_normalized['normalized'] < 0
print(
    # issdm-6 without CPU restriction
    len(df_normalized[normalized_limits_is_without & normalized_machine_is_issdm_6 & normalized_is_speed_up]),   # 1. speed-up
    len(df_normalized[normalized_limits_is_without & normalized_machine_is_issdm_6 & normalized_is_slow_down]),  # 2. slowdown
    
    # issdm-6 with CPU restriction
    len(df_normalized[normalized_limits_is_with & normalized_machine_is_issdm_6 & normalized_is_speed_up]),      # 3. speed-up
    len(df_normalized[normalized_limits_is_with & normalized_machine_is_issdm_6 & normalized_is_slow_down]),     # 4. slowdown
    
    # t2.micro without CPU restriction
    len(df_normalized[normalized_limits_is_without & normalized_machine_is_t2_micro & normalized_is_speed_up]),  # 5. speed-up
    len(df_normalized[normalized_limits_is_without & normalized_machine_is_t2_micro & normalized_is_slow_down]), # 6. slowdown
    
    # t2.micro with CPU restriction
    len(df_normalized[normalized_limits_is_with & normalized_machine_is_t2_micro & normalized_is_speed_up]),     # 7. speed-up
    len(df_normalized[normalized_limits_is_with & normalized_machine_is_t2_micro & normalized_is_slow_down])     # 8. slowdown
)
Explanation: We can find the number of benchmarks are speed-up and slowdown, respectively.
End of explanation
print(
    # For issdm-6
    df_normalized[normalized_machine_is_issdm_6 & normalized_limits_is_with]['normalized'].mean(),
    
    # For t2_micro
    df_normalized[normalized_machine_is_t2_micro & normalized_limits_is_with]['normalized'].mean()
)
Explanation: The average of normalized value for results under CPU restriction
End of explanation
df_normalized_issdm_6_without_limit = df_normalized[normalized_machine_is_issdm_6 & normalized_limits_is_without]
df_normalized_issdm_6_without_limit.normalized.hist(bins=150, figsize=(25,12), xlabelsize=20, ylabelsize=20)
plt.title('stress tests run on issdm-6 without CPU restriction', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Experiment Results from issdm-6
Let's have a look at the histogram of frequency of normalized value based on  stress tests without CPU restriction running on issdm-6.
End of explanation
df_normalized_issdm_6_without_limit_sorted = df_normalized_issdm_6_without_limit.sort_values(by='normalized', ascending=0)
df_normalized_issdm_6_without_limit_sorted_head = df_normalized_issdm_6_without_limit_sorted.head()
df_normalized_issdm_6_without_limit_sorted_tail = df_normalized_issdm_6_without_limit_sorted.tail()
df_normalized_issdm_6_without_limit_sorted_head.append(df_normalized_issdm_6_without_limit_sorted_tail)
Explanation: Here is the rank of normalized value from stress tests without CPU restriction
End of explanation
df_normalized_issdm_6_with_limit = df_normalized[normalized_machine_is_issdm_6 & normalized_limits_is_with]
df_normalized_issdm_6_with_limit.normalized.hist(color='Orange', bins=150, figsize=(25,12), xlabelsize=20, ylabelsize=20)
plt.title('stress tests run on issdm-6 with CPU restriction', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Now let's have a look at the histogram of frequency of normalized value based on  stress tests with CPU restriction running on issdm-6.
End of explanation
df_normalized_issdm_6_with_limit_sorted = df_normalized_issdm_6_with_limit.sort_values(by='normalized', ascending=0)
df_normalized_issdm_6_with_limit_sorted_head = df_normalized_issdm_6_with_limit_sorted.head()
df_normalized_issdm_6_with_limit_sorted_tail = df_normalized_issdm_6_with_limit_sorted.tail()
df_normalized_issdm_6_with_limit_sorted_head.append(df_normalized_issdm_6_with_limit_sorted_tail)
Explanation: Here is the rank of normalized value from stress tests with CPU restriction
End of explanation
df_normalized_issdm_6_no_outlier = df_normalized_issdm_6_with_limit['benchmark'] != 'stressng-cpu-jenkin'
df_normalized_issdm_6_with_limit[df_normalized_issdm_6_no_outlier].normalized.hist(color='Green', bins=150, figsize=(25,12), xlabelsize=20, ylabelsize=20)
plt.title('stress tests run on issdm-6 with CPU restriction (no outlier)', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: We notice that the stressng-cpu-jenkin looks like an outlier. Let's redraw the histogram without this one.
End of explanation
df_normalized_t2_micro_without_limit = df_normalized[normalized_machine_is_t2_micro & normalized_limits_is_without]
df_normalized_t2_micro_without_limit.normalized.hist(bins=150,figsize=(30,12), xlabelsize=20, ylabelsize=20)
plt.title('stress tests run on t2.micro without CPU restriction', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Summary
We got the boundary of normalized value on issdm-6 from -29.394675 to 54.266945 by using parameters --cpuset-cpus=1 --cpu-quota=7234 --cpu-period=100000, which means the docker container only uses 7.234ms CPU worth of run-time every 100ms on cpu 1 (See cpu for more details).
Experiment Results from t2.micro
Let's have a look at the histogram of frequency of normalized value based on  stress tests without CPU restriction running on t2.micro.
End of explanation
df_normalized_t2_micro_without_limit_sorted = df_normalized_t2_micro_without_limit.sort_values(by='normalized', ascending=0)
df_normalized_t2_micro_without_limit_sorted_head = df_normalized_t2_micro_without_limit_sorted.head()
df_normalized_t2_micro_without_limit_sorted_tail = df_normalized_t2_micro_without_limit_sorted.tail()
df_normalized_t2_micro_without_limit_sorted_head.append(df_normalized_t2_micro_without_limit_sorted_tail)
Explanation: Here is the rank of normalized value from stress tests without CPU restriction
End of explanation
df_normalized_t2_micro_with_limit = df_normalized[normalized_machine_is_t2_micro & normalized_limits_is_with]
df_normalized_t2_micro_with_limit.normalized.hist(color='Orange', bins=150, figsize=(30,12), xlabelsize=20, ylabelsize=20)
plt.title('stress tests run on t2.micro with CPU restriction', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Let's have a look at the histogram of frequency of normalized value based on  stress tests with CPU restriction running on t2.micro.
End of explanation
df_normalized_t2_micro_with_limit_sorted = df_normalized_t2_micro_with_limit.sort_values(by='normalized', ascending=0)
df_normalized_t2_micro_with_limit_sorted_head = df_normalized_t2_micro_with_limit_sorted.head()
df_normalized_t2_micro_with_limit_sorted_tail = df_normalized_t2_micro_with_limit_sorted.tail()
df_normalized_t2_micro_with_limit_sorted_head.append(df_normalized_t2_micro_with_limit_sorted_tail)
Explanation: Here is the rank of normalized value from stress tests with CPU restriction
End of explanation
df_normalized_t2_micro_no_outlier = df_normalized_t2_micro_with_limit['benchmark'] != 'stressng-memory-stack'
df_normalized_t2_micro_with_limit[df_normalized_t2_micro_no_outlier].normalized.hist(color='Green', bins=150, figsize=(30,12), xlabelsize=20, ylabelsize=20)
plt.title('stress tests run on t2.micro with CPU restriction (no outlier)', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: We notice that the stressng-memory-stack looks like an outlier. Let's redraw the histogram without this one.
End of explanation
df_verification = pd.read_csv('verification/results/2/alltests_with_normalized_results_1.1.csv')
Explanation: The stressng-cpu-jenkin benchmark is a collection of (non-cryptographic) hash functions for multi-byte keys. See Jenkins hash function from Wikipedia for more details.
Summary
We got the boundary of normalized value on t2.micro from -198.440535 to 119.904761 by using parameters --cpuset-cpus=0 --cpu-quota=25750 --cpu-period=100000, which means the docker container only uses 7.234ms CPU worth of run-time every 100ms on cpu 0 (See cpu for more details).
Verification
Now we use 9 other benchmark programs to verify this result. These programs are,
- blogbench: filesystem benchmark.
- compilebench: It tries to age a filesystem by simulating some of the disk IO common in creating, compiling, patching, stating and reading kernel trees.
- fhourstones: This integer benchmark solves positions in the game of connect-4.
- himeno: Himeno benchmark score is affected by the performance of a computer, especially memory band width. This benchmark program takes measurements to proceed major loops in solving the Poisson’s equation solution using the Jacobi iteration method.
- interbench: It is designed to measure the effect of changes in Linux kernel design or system configuration changes such as cpu, I/O scheduler and filesystem changes and options.
- nbench: NBench(Wikipedia) is a synthetic computing benchmark program developed in the mid-1990s by the now defunct BYTE magazine intended to measure a computer's CPU, FPU, and Memory System speed.
- pybench: It is a collection of tests that provides a standardized way to measure the performance of Python implementations.
- ramsmp: RAMspeed is a free open source command line utility to measure cache and memory performance of computer systems.
- stockfish-7: It is a simple benchmark by letting Stockfish analyze a set of positions for a given limit each.
Read verification tests data.
End of explanation
len(df_verification) / 2
Explanation: Show number of test benchmarks.
End of explanation
df_verification_rank = df_verification.reindex(df_verification.normalized.abs().sort_values(ascending=0).index)
df_verification_rank.head(8)
Explanation: Order the test results by the absolute of normalized value
End of explanation
df_verification_issdm_6 = df_verification[df_verification['machine'] == 'issdm-6']
df_verification_issdm_6.normalized.hist(color='y', bins=150,figsize=(20,10), xlabelsize=20, ylabelsize=20)
plt.title('verification tests run on issdm-6', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Verification Tests on issdm-6
Histogram of frequency of normalized value.
End of explanation
print(
    df_verification_issdm_6['normalized'].max(),
    df_verification_issdm_6['normalized'].min()
)
Explanation: Print the max the min normalized value,
End of explanation
df_verification_issdm_6['normalized'].mean()
Explanation: The average of noramlized value is,
End of explanation
df_verification_issdm_6_no_nbench = df_verification_issdm_6[~df_verification_issdm_6['benchmark'].str.startswith('nbench')]
df_verification_issdm_6_no_nbench.normalized.hist(color='greenyellow', bins=150,figsize=(20,10), xlabelsize=20, ylabelsize=20)
plt.title('verification tests run on issdm-6 (no nbench)', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: If we remove all nbench tests, the frequency histogram changes to
End of explanation
print(
    df_verification_issdm_6_no_nbench['normalized'].max(),
    df_verification_issdm_6_no_nbench['normalized'].min()
)
Explanation: The max the min normalized value changes to,
End of explanation
df_verification_issdm_6_no_nbench['normalized'].mean()
Explanation: The average of noramlized value changes to,
End of explanation
df_verification_t2_micro = df_verification[df_verification['machine'] == 't2.micro']
df_verification_t2_micro.normalized.hist(color='y', bins=150,figsize=(20,10), xlabelsize=20, ylabelsize=20)
plt.title('verification tests run on t2.micro', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Verification Tests on t2.micro
Histogram of frequency of normalized value.
End of explanation
df_verification_t2_micro['normalized'].mean()
Explanation: The average of noramlized value of the verification benchmarks is,
End of explanation
df_verification_top_benchmakrs = df_verification_rank[df_verification_rank['machine'] == 't2.micro'].head(4)['benchmark']
df_verification_t2_micro_no_outliers = df_verification_t2_micro[~df_verification_t2_micro['benchmark'].isin(df_verification_top_benchmakrs)]
df_verification_t2_micro_no_outliers.normalized.hist(color='greenyellow', bins=150,figsize=(20,10), xlabelsize=20, ylabelsize=20)
plt.title('verification tests on t2.micro (no outliers)', fontsize=30)
plt.xlabel('Normalized Value (re-execution / original)', fontsize=25)
plt.ylabel('Frequency (# of benchmarks)', fontsize=25)
Explanation: Let's see the frequency histogram after removing right-most four outliers.
End of explanation
print(
    df_verification_t2_micro_no_outliers['normalized'].max(),
    df_verification_t2_micro_no_outliers['normalized'].min()
)
Explanation: Print the max the min normalized value,
End of explanation
df_verification_t2_micro_no_outliers['normalized'].mean()
Explanation: The average of noramlized value without the four outliners is,
End of explanation
<END_TASK>