<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Notebook to classify power plants
Variables include
Step1:  From the figures above, it is difficult to determine an optimal number of clusters. The silhouette score clearly shows that we need more than 5 clusters. 6 looks like a good number, but one of the clusters shows odd behavior when plotting generation change vs net demand change (looks like 2 solid regression lines in the figure).
Step2:  PREVIOUS WORK BELOW
Code above now does everything needed, below is kept for reference
Variable definitions
These variables will be used to cluster the power plants.
- year
Step3:  missing '1-hr ramp rate'
Step4:  Filter out non-fossil plants
Step5:  Start classifying only with all nan values dropped
Step6:  The code below begins by scaling the X-vector data to mean 0 and a standard variance. It then loops through a wide range of k values for the number of clusters (3 to 14), calculating the labels for each plant, the Calinski Harabaz score, and the Silhouette score for each value of k.
Step7:  The two score values are plotted below. They don't agree exactly on the optimal number of centers, but it looks like 6 or 8 is probably best.
Step8:  The integer appended to each of the cluster_id column labels is equal to the number of clusters.
Step9:  Some basic information about each of the clusters
I've created a bunch of tables and figures to better understand each of the clusters (k=6). The tables show mean, count, and variance. The first figure shows the number of power plants in each group (remember that each power plant is included for every year it operates and data is available). The second looks at the size (capacity). Finally, there is a figure with facets for mean Capacity, Capacity Factor, Efficiency, and Ramp Rate.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os
import sklearn as sk
from cluster import Clusters
import os
filename = 'Cluster_Data_2.csv'
path = '../Clean Data'
fullpath = os.path.join(path, filename)
cluster = Clusters(fullpath)
cluster.make_clusters(n_clusters=range(4,26))
cluster.evaluate_clusters()
Explanation: Notebook to classify power plants
Variables include:
- Fuel type
- Historical ramp speeds
- Historical efficiency
- Location?
End of explanation
cluster_labels = cluster.export_cluster_id(k=16)
cluster_labels[:5]
labeled_plants = cluster.label_and_export(k=16)
labeled_plants.to_clipboard()
export_df = fossil_with_ramp.loc[:,['year', 'plant_id', 'cluster_id_6']]
export_df.to_csv('Cluster labels.csv', index = False)
Explanation: From the figures above, it is difficult to determine an optimal number of clusters. The silhouette score clearly shows that we need more than 5 clusters. 6 looks like a good number, but one of the clusters shows odd behavior when plotting generation change vs net demand change (looks like 2 solid regression lines in the figure).
End of explanation
# cols = ['year', 'fuel type', '1-hr ramp rate', '3-hr ramp rate',
#         'efficiency', 'efficiency std', 'CF', 'CF std']
# #Add index of power plant IDs
# df = pd.DataFrame(columns=cols)
filename = 'Cluster_Data.csv'
path = '../Clean Data'
fullpath = os.path.join(path, filename)
cluster_df = pd.read_csv(fullpath)
Explanation: PREVIOUS WORK BELOW
Code above now does everything needed, below is kept for reference
Variable definitions
These variables will be used to cluster the power plants.
- year: year that the data is for
- fuel type: primary fuel type used at the facility (SUB, LIG, NG, etc)
- 1-hr ramp rate: maximum or 95 percentile rate of generation increase over 1 hour
- 3-hr ramp rate: maximum or 95 percentile rate of generation increase over 3 hours
- efficiency: might just change this to heat rate (MMBTU/MWh)
- efficiency std: standard deviation in efficiency based on monthly values for the year
- CF: capacity factor, which is the amount generated in a year divided by the theoretical maximum generation possible $MWh/(Capacity * 8760)$ (or hours per month for monthly values used in calculating the standard deviation)
- CF std: standard deviation of CF based on monthly values for the year
End of explanation
cluster_df[cluster_df.plant_id==127]
cluster_df[cluster_df.plant_id==3466]
cluster_df[cluster_df.plant_id==3584]
cluster_df['fuel_type'].unique()
Explanation: missing '1-hr ramp rate'
End of explanation
fossil_codes = ['SUB', 'LIG', 'NG', 'DFO', 'PC']
fossil_df = cluster_df.loc[cluster_df['fuel_type'].isin(fossil_codes)]
fossil_df.describe()
# Unique plants
len(fossil_df.dropna().loc[:,'plant_id'].unique())
sns.distplot(fossil_df['capacity'].dropna())
fossil_with_ramp = fossil_df.dropna()
sns.distplot(fossil_with_ramp['capacity'])
Explanation: Filter out non-fossil plants
End of explanation
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score, calinski_harabaz_score
from sklearn import preprocessing
fossil_with_ramp.columns
sns.pairplot(fossil_with_ramp, hue='fuel_type', vars=[u'capacity', u'capacity_factor', u'efficiency',
       u'ramp_rate'])
Explanation: Start classifying only with all nan values dropped
End of explanation
cluster_data = pd.DataFrame(index=range(3,15), columns=['n_clusters', 'score', 'silhouette'])
cluster_labels = {}
X = fossil_with_ramp[['capacity', 'capacity_factor', 'efficiency', 'ramp_rate']]
X_scaled = preprocessing.StandardScaler().fit_transform(X)
for idx, n_clusters in enumerate(range(3,15)):
    cluster_data.loc[n_clusters, 'n_clusters'] = n_clusters
    clusterer = KMeans(n_clusters, random_state=42)
    # fit_clusters = clusterer.fit(X)
#     cluster_labels = clusterer.fit_predict(X)
    cluster_labels[n_clusters] = clusterer.fit_predict(X_scaled)
    
    # http://scikit-learn.org/stable/modules/clustering.html#calinski-harabaz-index
    # The score is higher when clusters are dense and well separated
#     score[idx] = metrics.calinski_harabaz_score(X, cluster_labels)
    cluster_data.loc[n_clusters, 'score'] = calinski_harabaz_score(X_scaled, cluster_labels[n_clusters])
     
#     silhouette[idx] = silhouette_score(X, cluster_labels)
    cluster_data.loc[n_clusters, 'silhouette'] = silhouette_score(X_scaled, cluster_labels[n_clusters])
    
#     print 'For ', n_clusters, ' clusters, the average silhouette score is :', silhouette[idx], \
#         ' and the score is :', score[idx]
Explanation: The code below begins by scaling the X-vector data to mean 0 and a standard variance. It then loops through a wide range of k values for the number of clusters (3 to 14), calculating the labels for each plant, the Calinski Harabaz score, and the Silhouette score for each value of k.
End of explanation
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(8,3))
cluster_data.plot(y='score', ax=ax1)
# ax1.plot(range(3,15), cluster_data['score'])
ax1.set_title('Calinski Harabaz score\nHigher is better')
cluster_data.plot(y='silhouette', ax=ax2)
# ax2.plot(range(3,15), silhouette)
ax2.set_title('Silhouette score\nLower is better')
Explanation: The two score values are plotted below. They don't agree exactly on the optimal number of centers, but it looks like 6 or 8 is probably best.
End of explanation
for n_clusters in cluster_labels.keys():
    fossil_with_ramp.loc[:,'cluster_id_{}'.format(n_clusters)] = cluster_labels[n_clusters]
fossil_with_ramp.head()
sns.pairplot(fossil_with_ramp, hue='cluster_id_6', vars=[u'capacity', u'capacity_factor', u'efficiency',
       u'ramp_rate'])
Explanation: The integer appended to each of the cluster_id column labels is equal to the number of clusters.
End of explanation
drop_columns = ['cluster_id_{}'.format(i) for i in [3,4,5,7,8,9,10,11,12,13,14]]
grouped = fossil_with_ramp.drop(drop_columns, axis=1).groupby(['cluster_id_6', 'fuel_type'])
grouped.mean()
grouped.count()
grouped.std()
sns.countplot('cluster_id_6', hue='fuel_type', data=fossil_with_ramp)
plt.title('Count of plants in each cluster')
sns.barplot('cluster_id_6', 'capacity', data=fossil_with_ramp, hue='fuel_type',
           estimator=sum)
plt.title('Total Capacity of plants in each cluster')
# data = fossil_with_ramp.drop(drop_columns, axis=1)
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2,2, figsize=(10,8))
sns.barplot('cluster_id_6', 'capacity', data=fossil_with_ramp, hue='fuel_type', ax=ax1)
ax1.set_title('Mean Capacity')
sns.barplot('cluster_id_6', 'capacity_factor', data=fossil_with_ramp, hue='fuel_type', ax=ax2)
ax2.set_title('Mean Capacity Factor')
sns.barplot('cluster_id_6', 'efficiency', data=fossil_with_ramp, hue='fuel_type', ax=ax3)
ax3.set_title('Mean Efficiency')
sns.barplot('cluster_id_6', 'ramp_rate', data=fossil_with_ramp, hue='fuel_type', ax=ax4)
ax4.set_title('Mean Ramp Rate')
# sns.barplot('Climate', 'HDD65', data=house, ax=ax2)
# # Shrink the point sizes (scale), change the estimator from mean to median
# sns.pointplot('Climate', 'HDD65', data=house, scale=0.7, estimator=np.median, ax=ax3)
# # Adjust the bandwidth (smoothing)
# sns.violinplot('Climate', 'HDD65', data=house, ax=ax4, bw=0.4)
plt.tight_layout()
Explanation: Some basic information about each of the clusters
I've created a bunch of tables and figures to better understand each of the clusters (k=6). The tables show mean, count, and variance. The first figure shows the number of power plants in each group (remember that each power plant is included for every year it operates and data is available). The second looks at the size (capacity). Finally, there is a figure with facets for mean Capacity, Capacity Factor, Efficiency, and Ramp Rate.
End of explanation
<END_TASK>