<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Performance Benchmarking for KV Drive
The goal of these set of experiments is to characterize the variability across platforms in a systematic and consistent way in terms of KV drive. The steps of experiments are as follows,
Run Stress-ng benchmarks on one KV drive;
Run Stress-ng benchmarks on machine issdm-6, and get the "without limit" result;
Find all the common benchmarks from both results;
Calculate the speedup (normalized value) of each benchmark based on the one from KV drive (issdm-6 (without limit) / KV drive);
Use torpor to calculate the best cpu quota by minimizing the average speedups. We will later use this parameter to limit the cpu usage in the docker container;
Run Stress-ng benchmarks in the constrained docker container on machine issdm-6, and get the "with limit" result;
Calculate the speedup based on KV drive again (issdm-6 (with limit) / KV drive), then we get a new "speedup range", which should be must smaller than the previous one.
Run a bunch of other benchmarks on both KV drive and constrained docker container to verify if they are all within in the later "speedup range".
Make conclusion.
Step1:  First, we load all test data.
Step2:  Let's have a look at the pattern of data.
Step3:  Show all the test machines.
Step4:  Define a predicate for machine issdm-6
Step5:  The number of benchmarks we ran on issdm-6 with limit is
Step6:  The number of benchmarks we ran on issdm-6 without limit is
Step7:  The number of benchmarks we ran on kv3
Step8:  Because some benchmarks could fail druing the test suite running, those failed tests are not in the result report. We want to know how many common tests they both complated.
Step9:  Read the normalized results.
Step10:  Show some of the data lines. The normalized value is the speedup based on kv3. It becomes a negative value when the benchmark runs on issdm-6 is slower than on kv3 (slowdown).
Step11:  There is one benchmark not present in both with and without limit result set.
Step12:  Since the number of common benchmarks is 113, we wnat to find the one benchmark less than two results (all from issdm-6).
Step13:  In other words, stressng-memory-oom-pipe should not in the with limit results of issdm-6
Step14:  We can find the number of benchmarks are speed-up and the number of them are slowdown on without limit results.
Step15:  All right, let's draw a bar plot for all results.
Step16:  Which one have the greatest and smallest speedup on without limit banchmark results?
Step17:  Let's have a look at the speedup frequency on without limit benchmark results.
Step18:  Which one have the greatest and smallest speedup on with limit benchmark results?
Step19:  The average speedup of with limit benchmarks is,
Step20:  Let's have a look at the speedup frequency on with limit benchmark results.
Step21:  The stressng-cpu-jenkin benchmark is a collection of (non-cryptographic) hash functions for multi-byte keys. See Jenkins hash function from Wikipedia for more details.
We got the speedup boundary from -276.278268 to 127.716328 by using parameters --cpuset-cpus=1 --cpu-quota=1000 --cpu-period=10000, which means the docker container only uses 1ms CPU worth of run-time every 10ms on cpu 1 (See cpu for more details).
Now we use 9 other benchmark programs to verify this result. These programs are,
- blogbench
Step22:  Show number of test benchmarks.
Step23:  Let's see the speedup of each individual result.
Step24:  Sort the test results set by the absolute value of normalized
Step25:  See the histogram of all of speedups after filtering out the one outlier.
Step26:  The average of speedup of the test benchmarks without the one outlier is,
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import random
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
pd.set_option("display.max_rows", 8)
Explanation: Performance Benchmarking for KV Drive
The goal of these set of experiments is to characterize the variability across platforms in a systematic and consistent way in terms of KV drive. The steps of experiments are as follows,
Run Stress-ng benchmarks on one KV drive;
Run Stress-ng benchmarks on machine issdm-6, and get the "without limit" result;
Find all the common benchmarks from both results;
Calculate the speedup (normalized value) of each benchmark based on the one from KV drive (issdm-6 (without limit) / KV drive);
Use torpor to calculate the best cpu quota by minimizing the average speedups. We will later use this parameter to limit the cpu usage in the docker container;
Run Stress-ng benchmarks in the constrained docker container on machine issdm-6, and get the "with limit" result;
Calculate the speedup based on KV drive again (issdm-6 (with limit) / KV drive), then we get a new "speedup range", which should be must smaller than the previous one.
Run a bunch of other benchmarks on both KV drive and constrained docker container to verify if they are all within in the later "speedup range".
Make conclusion.
End of explanation
df = pd.read_csv('stress-ng/second/results/combo/1/alltests.csv')
Explanation: First, we load all test data.
End of explanation
df.head()
Explanation: Let's have a look at the pattern of data.
End of explanation
df['machine'].unique()
Explanation: Show all the test machines.
End of explanation
machine_is_issdm_6 = df['machine'] == 'issdm-6'
Explanation: Define a predicate for machine issdm-6
End of explanation
limits_is_with = df['limits'] == 'with'
df_issdm_6_with_limit = df[machine_is_issdm_6 & limits_is_with]
len(df_issdm_6_with_limit)
Explanation: The number of benchmarks we ran on issdm-6 with limit is
End of explanation
limits_is_without = df['limits'] == 'without'
len(df[machine_is_issdm_6 & limits_is_without])
Explanation: The number of benchmarks we ran on issdm-6 without limit is
End of explanation
df_kv3 = df[df['machine'] == 'kv3']
len(df_kv3)
Explanation: The number of benchmarks we ran on kv3
End of explanation
df_common = pd.merge(df_issdm_6_with_limit, df_kv3, how='inner', on='benchmark')
len(df_common)
Explanation: Because some benchmarks could fail druing the test suite running, those failed tests are not in the result report. We want to know how many common tests they both complated.
End of explanation
df = pd.read_csv('stress-ng/second/results/combo/1/alltests_with_normalized_results_1.1.csv')
Explanation: Read the normalized results.
End of explanation
df.head()
Explanation: Show some of the data lines. The normalized value is the speedup based on kv3. It becomes a negative value when the benchmark runs on issdm-6 is slower than on kv3 (slowdown).
End of explanation
len(df) / 2
Explanation: There is one benchmark not present in both with and without limit result set.
End of explanation
grouped = df.groupby('benchmark')
df[grouped['benchmark'].transform(len) < 2]
Explanation: Since the number of common benchmarks is 113, we wnat to find the one benchmark less than two results (all from issdm-6).
End of explanation
df_issdm_6_with_limit[df_issdm_6_with_limit['benchmark'] == 'stressng-memory-oom-pipe'].empty
Explanation: In other words, stressng-memory-oom-pipe should not in the with limit results of issdm-6
End of explanation
predicate_without_limits = df['limits'] == 'without'
predicate = predicate_without_limits & (df['normalized'] >= 0)
len(df[predicate])
predicate = predicate_without_limits & (df['normalized'] < 0)
len(df[predicate])
Explanation: We can find the number of benchmarks are speed-up and the number of them are slowdown on without limit results.
End of explanation
sns.set()
sns.set_context("poster")
plt.xticks(rotation=90)
sns.barplot(x='benchmark', y='normalized', hue='limits', data=df)
Explanation: All right, let's draw a bar plot for all results.
End of explanation
df_without_sorted = df[df['limits'] == 'without'].sort_values(by='normalized', ascending=0)
head_without = df_without_sorted.head()
tail_without = df_without_sorted.tail()
head_without.append(tail_without)
Explanation: Which one have the greatest and smallest speedup on without limit banchmark results?
End of explanation
ax = df[df['limits'] == 'without'].groupby('limits').normalized.hist(bins=100,xrot=90,figsize=(20,10),alpha=0.5)
plt.xlabel('Speedup (re-execution / original)')
plt.ylabel('Frequency (# of benchmarks)')
Explanation: Let's have a look at the speedup frequency on without limit benchmark results.
End of explanation
df_with_sorted = df[df['limits'] == 'with'].sort_values(by='normalized', ascending=0)
head_with = df_with_sorted.head()
tail_with = df_with_sorted.tail()
head_with.append(tail_with)
Explanation: Which one have the greatest and smallest speedup on with limit benchmark results?
End of explanation
df[df['limits'] == 'with']['normalized'].mean()
Explanation: The average speedup of with limit benchmarks is,
End of explanation
ax = df[df['limits'] == 'with'].groupby('limits').normalized.hist(bins=100,xrot=90,figsize=(20,10),alpha=0.5)
plt.xlabel('Speedup (re-execution / original)')
plt.ylabel('Frequency (# of benchmarks)')
Explanation: Let's have a look at the speedup frequency on with limit benchmark results.
End of explanation
df = pd.read_csv('verification/results/1/alltests_with_normalized_results_1.0.csv')
Explanation: The stressng-cpu-jenkin benchmark is a collection of (non-cryptographic) hash functions for multi-byte keys. See Jenkins hash function from Wikipedia for more details.
We got the speedup boundary from -276.278268 to 127.716328 by using parameters --cpuset-cpus=1 --cpu-quota=1000 --cpu-period=10000, which means the docker container only uses 1ms CPU worth of run-time every 10ms on cpu 1 (See cpu for more details).
Now we use 9 other benchmark programs to verify this result. These programs are,
- blogbench: filesystem benchmark.
- compilebench: It tries to age a filesystem by simulating some of the disk IO common in creating, compiling, patching, stating and reading kernel trees.
- fhourstones: This integer benchmark solves positions in the game of connect-4.
- himeno: Himeno benchmark score is affected by the performance of a computer, especially memory band width. This benchmark program takes measurements to proceed major loops in solving the Poisson’s equation solution using the Jacobi iteration method.
- interbench: It is designed to measure the effect of changes in Linux kernel design or system configuration changes such as cpu, I/O scheduler and filesystem changes and options.
- nbench: NBench(Wikipedia) is a synthetic computing benchmark program developed in the mid-1990s by the now defunct BYTE magazine intended to measure a computer's CPU, FPU, and Memory System speed.
- pybench: It is a collection of tests that provides a standardized way to measure the performance of Python implementations.
- ramsmp: RAMspeed is a free open source command line utility to measure cache and memory performance of computer systems.
- stockfish-7: It is a simple benchmark by letting Stockfish analyze a set of positions for a given limit each.
Read verification tests data.
End of explanation
len(df)
Explanation: Show number of test benchmarks.
End of explanation
sns.set()
sns.set_context("poster")
plt.xticks(rotation=90)
sns.barplot(x='benchmark', y='normalized', data=df)
Explanation: Let's see the speedup of each individual result.
End of explanation
df.reindex(df.normalized.abs().sort_values(ascending=0).index).head(8)
Explanation: Sort the test results set by the absolute value of normalized
End of explanation
df_t = df[df['benchmark'] != 'nbench_fp']
ax = df_t.normalized.hist(bins=100,xrot=90,figsize=(20,10),alpha=0.5)
plt.xlabel('Speedup (re-execution / original)')
plt.ylabel('Frequency (# of benchmarks)')
Explanation: See the histogram of all of speedups after filtering out the one outlier.
End of explanation
df_t['normalized'].mean()
Explanation: The average of speedup of the test benchmarks without the one outlier is,
End of explanation
<END_TASK>