<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Pre-processing of avalanche warning data for machine learning
Step1:  Check if there are missing values.
Step2:  Fill missing values where necessary.
Step3:  Feature engineering
Re-label og -classifiy variables where necessary.
Add an avalanche problem severity index - based on its attributes size, distribution and sensitivity.
When using shift or filling values using mean or similar, make sure to first sort individual regions and seasons by date.
Step4:  Add historical values, e.g. yesterdays precipitation
Add a tag to the feature name to indicate if it is categorical (c) or numerical (n).
Add a target tag (t).
Add a modelled (m) or observed (o) tag.
_prev1day
_prev3day
n_f_Next24HourChangeInTempFromPrev3DayMax - change of temperature over a certain period.
n_r_Prev7dayMinTemp2InPast - ???
n_r_SNOWDAS_SnowpackAveTemp_k2InPast - modelled average temperature from model SNOWDAS (? https
Step5:  Combine avalanche problem attributes into single parameter
Step6:  Hot encode categorical variables where necessary.
Step7:  Check if there are no weired or missing values.
Step8:  Remove variables we know we do not need. In this case mainly because they are redundant like the avalanche_problem_1_ext_name and avalanche_problem_1_ext_id - in this case we only keep the numeric id variable.
Step9:  Fill missing values where necessary
Step10:  Eventually remove variables with many missing values.
Step11:  Check again if there are still values missing...
need to replace these Nans with meaningful values or remove the feature.
Step12:  We can see that some parameters are highly correlated. These are mainly the parameters belonging to the same avalanche problem. Depending on the ML algorithm we use we have to remove some of them.
Step13:  Save data for further analysis
<ASSISTANT_TASK:>
Python Code:
import sys
import pandas as pd  # check out Modin https://towardsdatascience.com/get-faster-pandas-with-modin-even-on-your-laptops-b527a2eeda74
import numpy as np
import json
import matplotlib.pyplot as plt
import seaborn as sns
from pathlib import Path
import datetime
# Add path to APS modules
aps_pth = Path('.').absolute()
print(aps_pth)
if aps_pth not in sys.path:
    sys.path.append(aps_pth)
sns.set(style="white")
#from sklearn.preprocessing import LabelEncoder
#from pprint import pprint
#pd.set_option("display.max_rows",6)
%matplotlib inline
# analysis of entire data set - collected using varsomdata2.varsomscripts.avalanchewarningscomplete.get_season_17_18()
#data_pth = Path(r'.\aps\data\varsom\norwegian_avalanche_warnings_season_17_18.csv')
data_pth = Path(r'D:\Dev\APS\aps\data\varsom\norwegian_avalanche_warnings_season_16_19.csv')
#varsom_df = pd.read_csv(aps_pth / data_pth, index_col=0)
varsom_df = pd.read_csv(data_pth, index_col=0)
varsom_df.head()
varsom_df.columns.values
varsom_df[varsom_df['region_id']==3012].filter(['avalanche_problem_1_cause_id', 'avalanche_problem_1_cause_name',
       'avalanche_problem_1_destructive_size_ext_id',
       'avalanche_problem_1_destructive_size_ext_name',
       'avalanche_problem_1_distribution_id',
       'avalanche_problem_1_distribution_name',
       'avalanche_problem_1_exposed_height_1',
       'avalanche_problem_1_exposed_height_2',
       'avalanche_problem_1_exposed_height_fill',
       'avalanche_problem_1_ext_id', 'avalanche_problem_1_ext_name',
       'avalanche_problem_1_probability_id',
       'avalanche_problem_1_probability_name',
       'avalanche_problem_1_problem_id',
       'avalanche_problem_1_problem_type_id',
       'avalanche_problem_1_problem_type_name',
       'avalanche_problem_1_trigger_simple_id',
       'avalanche_problem_1_trigger_simple_name',]).head(10)
Explanation: Pre-processing of avalanche warning data for machine learning
End of explanation
# for col in varsom_df.columns.values:
#     print(f'{col}: {varsom_df[col].unique()} \n')
# Find the amount of NaN values in each column
print(varsom_df.isnull().sum().sort_values(ascending=False))
Explanation: Check if there are missing values.
End of explanation
varsom_df['mountain_weather_wind_speed'] = varsom_df['mountain_weather_wind_speed'].fillna('None')
varsom_df['mountain_weather_wind_direction'] = varsom_df['mountain_weather_wind_direction'].fillna('None')
print(varsom_df.isnull().sum().sort_values(ascending=False))
Explanation: Fill missing values where necessary.
End of explanation
varsom_df['date'] = pd.to_datetime(varsom_df['date_valid'], infer_datetime_format=True)
def add_prevday_features(df):
    ### danger level
    df['danger_level_prev1day'] = df['danger_level'].shift(1)
    df['danger_level_name_prev1day'] = df['danger_level_name'].shift(1)
    df['danger_level_prev2day'] = df['danger_level'].shift(2)
    df['danger_level_name_prev2day'] = df['danger_level_name'].shift(2)
    df['danger_level_prev3day'] = df['danger_level'].shift(3)
    df['danger_level_name_prev3day'] = df['danger_level_name'].shift(3)
    ### avalanche problem
    df['avalanche_problem_1_cause_id_prev1day'] = df['avalanche_problem_1_cause_id'].shift(1)
    df['avalanche_problem_1_problem_type_id_prev1day'] = df['avalanche_problem_1_problem_type_id'].shift(1)
    df['avalanche_problem_1_cause_id_prev2day'] = df['avalanche_problem_1_cause_id'].shift(2)
    df['avalanche_problem_1_problem_type_id_prev2day'] = df['avalanche_problem_1_problem_type_id'].shift(2)
    df['avalanche_problem_1_cause_id_prev3day'] = df['avalanche_problem_1_cause_id'].shift(3)
    df['avalanche_problem_1_problem_type_id_prev3day'] = df['avalanche_problem_1_problem_type_id'].shift(3)
    df['avalanche_problem_2_cause_id_prev1day'] = df['avalanche_problem_2_cause_id'].shift(1)
    df['avalanche_problem_2_problem_type_id_prev1day'] = df['avalanche_problem_2_problem_type_id'].shift(1)
    df['avalanche_problem_2_cause_id_prev2day'] = df['avalanche_problem_2_cause_id'].shift(2)
    df['avalanche_problem_2_problem_type_id_prev2day'] = df['avalanche_problem_2_problem_type_id'].shift(2)
    df['avalanche_problem_2_cause_id_prev3day'] = df['avalanche_problem_2_cause_id'].shift(3)
    df['avalanche_problem_2_problem_type_id_prev3day'] = df['avalanche_problem_2_problem_type_id'].shift(3)
    ### weather
    df['mountain_weather_temperature_max_prev1day'] = df['mountain_weather_temperature_max'].shift(1)
    df['mountain_weather_temperature_max_prev2day'] = df['mountain_weather_temperature_max'].shift(2)
    df['mountain_weather_temperature_max_prev3day'] = df['mountain_weather_temperature_max'].shift(3)
    df['mountain_weather_temperature_min_prev1day'] = df['mountain_weather_temperature_min'].shift(1)
    df['mountain_weather_temperature_min_prev2day'] = df['mountain_weather_temperature_min'].shift(2)
    df['mountain_weather_temperature_min_prev3day'] = df['mountain_weather_temperature_min'].shift(3)
    df['mountain_weather_precip_region_prev1day'] = df['mountain_weather_precip_region'].shift(1)
    df['mountain_weather_precip_most_exposed_prev1day'] = df['mountain_weather_precip_most_exposed'].shift(1)
    df['mountain_weather_precip_region_prev3daysum'] = df['mountain_weather_precip_region'].shift(1) + df['mountain_weather_precip_region'].shift(2) + df['mountain_weather_precip_region'].shift(3)
    return df
varsom_df[(varsom_df['date']>=datetime.date(year=2016, month=12, day=1)) & (varsom_df['date']<datetime.date(year=2017, month=6, day=1))]
# grouping by region and season
grouped_df = pd.DataFrame()
for id in varsom_df['region_id'].unique():
#for id in [3003, 3011, 3014, 3028]:
    _tmp_df = varsom_df[varsom_df['region_id']==id].copy()
    _tmp_df.sort_values(by='valid_from')
    
    start, stop = int(_tmp_df['date_valid'].min()[:4]), int(_tmp_df['date_valid'].max()[:4])
    for yr in range(start, stop-1):
        _tmp_df[(_tmp_df['date']>=datetime.date(year=yr, month=12, day=1)) & (_tmp_df['date']<datetime.date(year=yr+1, month=6, day=1))]
        _tmp_df = add_prevday_features(_tmp_df)
        #print(len(_tmp_df), _tmp_df['region_id'].unique())
        if grouped_df.empty:
            print('empty')
            grouped_df = _tmp_df.copy()
        else:
            grouped_df = pd.concat([grouped_df, _tmp_df], ignore_index=True).copy()
        
    #print('g', len(grouped_df), grouped_df['region_id'].unique())
    
grouped_df.filter(['valid_from', 'region_name', 'region_id', 'avalanche_problem_1_problem_type_id', 'avalanche_problem_1_problem_type_id_prev2day'])
varsom_df = grouped_df.copy()
#from aps.notebooks.ml_varsom.regroup_forecast import regroup
from regroup_forecast import regroup
varsom_df = regroup(varsom_df)
Explanation: Feature engineering
Re-label og -classifiy variables where necessary.
Add an avalanche problem severity index - based on its attributes size, distribution and sensitivity.
When using shift or filling values using mean or similar, make sure to first sort individual regions and seasons by date.
End of explanation
# Check if sensitivity transformation worked...
print(varsom_df['avalanche_problem_1_sensitivity_id_class'].value_counts())
varsom_df.filter(['mountain_weather_precip_region', 'mountain_weather_precip_region_prev3daysum']).head(12)
varsom_df[varsom_df['region_id']==3012].filter(['region_id', 'danger_level', 'danger_level_prev1day']).head(40)
Explanation: Add historical values, e.g. yesterdays precipitation
Add a tag to the feature name to indicate if it is categorical (c) or numerical (n).
Add a target tag (t).
Add a modelled (m) or observed (o) tag.
_prev1day
_prev3day
n_f_Next24HourChangeInTempFromPrev3DayMax - change of temperature over a certain period.
n_r_Prev7dayMinTemp2InPast - ???
n_r_SNOWDAS_SnowpackAveTemp_k2InPast - modelled average temperature from model SNOWDAS (? https://nsidc.org/data/g02158)
End of explanation
def get_aval_problem_combined(type_, dist_, sens_, size_):
    return int("{0}{1}{2}{3}".format(type_, dist_, sens_, size_))
def print_aval_problem_combined(aval_combined_int):
    aval_combined_str = str(aval_combined_int)
    #with open(aps_pth / r'aps/config/snoskred_keys.json') as jdata:
    with open(r'D:\Dev\APS\aps\config\snoskred_keys.json') as jdata:
        snoskred_keys = json.load(jdata)
    type_ = snoskred_keys["Class_AvalancheProblemTypeName"][aval_combined_str[0]]
    dist_ = snoskred_keys["Class_AvalDistributionName"][aval_combined_str[1]]
    sens_ = snoskred_keys["Class_AvalSensitivityId"][aval_combined_str[2]]
    size_ = snoskred_keys["DestructiveSizeId"][aval_combined_str[3]]
    
    return f"{type_}:{dist_}:{sens_}:{size_}"
print(print_aval_problem_combined(6221))
    
    
    
varsom_df['aval_problem_1_combined'] = varsom_df.apply(lambda row: get_aval_problem_combined(row['avalanche_problem_1_problem_type_id_class'],
                                                                                     row['avalanche_problem_1_distribution_id'],
                                                                                     row['avalanche_problem_1_sensitivity_id_class'], #avalanche_problem_1_trigger_simple_id_class / avalanche_problem_1_sensitivity_id_class
                                                                                     row['avalanche_problem_1_destructive_size_ext_id']), axis=1)
aval_uni = varsom_df['aval_problem_1_combined'].unique()
print(aval_uni, len(aval_uni))
print(varsom_df['aval_problem_1_combined'].value_counts())
print(varsom_df['avalanche_problem_1_problem_type_id_class'].value_counts())
Explanation: Combine avalanche problem attributes into single parameter
End of explanation
# hot encode
hot_encode_ = ['emergency_warning', 'author', 'mountain_weather_wind_direction']
varsom_df = pd.get_dummies(varsom_df, columns=hot_encode_)
Explanation: Hot encode categorical variables where necessary.
End of explanation
# Check if there are no weired or missing values.
for col in varsom_df.columns.values:
    print(f'{col}: {varsom_df[col].unique()} \n')
Explanation: Check if there are no weired or missing values.
End of explanation
del_list = [
    'utm_zone',
    'utm_east',
    'utm_north',
    'danger_level_name',
    'avalanche_problem_1_exposed_height_fill',
    'avalanche_problem_2_exposed_height_fill',
    'avalanche_problem_3_exposed_height_fill',
    'avalanche_problem_1_valid_expositions',
    'avalanche_problem_2_valid_expositions',
    'avalanche_problem_3_valid_expositions',
    'avalanche_problem_1_cause_name',
    'avalanche_problem_1_problem_type_name',
    'avalanche_problem_1_destructive_size_ext_name',
    'avalanche_problem_1_distribution_name',
    'avalanche_problem_1_ext_name',
    'avalanche_problem_1_probability_name',
    'avalanche_problem_1_trigger_simple_name',
    'avalanche_problem_1_type_name',
    'avalanche_problem_2_cause_name',
    'avalanche_problem_2_problem_type_name',
    'avalanche_problem_2_destructive_size_ext_name',
    'avalanche_problem_2_distribution_name',
    'avalanche_problem_2_ext_name',
    'avalanche_problem_2_probability_name',
    'avalanche_problem_2_trigger_simple_name',
    'avalanche_problem_2_type_name',
    'avalanche_problem_3_cause_name',
    'avalanche_problem_3_problem_type_name',
    'avalanche_problem_3_destructive_size_ext_name',
    'avalanche_problem_3_distribution_name',
    'avalanche_problem_3_ext_name',
    'avalanche_problem_3_probability_name',
    'avalanche_problem_3_trigger_simple_name',
    'avalanche_problem_3_type_name',
    'latest_avalanche_activity',
    'main_text',
    'snow_surface',
    'current_weak_layers',
    'avalanche_danger',
    'avalanche_problem_1_advice',
    'avalanche_problem_2_advice',
    'avalanche_problem_3_advice',
    'mountain_weather_wind_speed',
    'region_type_name',
    'region_name',
    'reg_id',
    'valid_from',
    'valid_to'
]
removed_ = [varsom_df.pop(v) for v in del_list]
removed_
Explanation: Remove variables we know we do not need. In this case mainly because they are redundant like the avalanche_problem_1_ext_name and avalanche_problem_1_ext_id - in this case we only keep the numeric id variable.
End of explanation
fill_list = [
    'mountain_weather_freezing_level',
    'mountain_weather_precip_region',
    'mountain_weather_precip_region_prev1day',
    'mountain_weather_precip_region_prev3daysum',
    'mountain_weather_precip_most_exposed',
    'mountain_weather_precip_most_exposed_prev1day',
    'mountain_weather_temperature_min',
    'mountain_weather_temperature_max',
    'mountain_weather_temperature_elevation',
    'danger_level_prev3day',
    'avalanche_problem_1_problem_type_id_prev3day',
    'avalanche_problem_2_problem_type_id_prev3day',
    'avalanche_problem_2_cause_id_prev3day',
    'avalanche_problem_1_cause_id_prev3day',
    'danger_level_prev2day',
    'avalanche_problem_1_cause_id_prev2day',
    'avalanche_problem_1_problem_type_id_prev2day',
    'avalanche_problem_2_cause_id_prev2day',
    'avalanche_problem_2_problem_type_id_prev2day',
    'avalanche_problem_2_cause_id_prev1day',
    'avalanche_problem_2_problem_type_id_prev1day',
    'avalanche_problem_1_problem_type_id_prev1day',
    'avalanche_problem_1_cause_id_prev1day',
    'danger_level_prev1day'
]
filled_ = [varsom_df[v].fillna(0., inplace=True) for v in fill_list]
filled_
Explanation: Fill missing values where necessary
End of explanation
del_list = [
    'danger_level_name_prev1day', 'danger_level_name_prev2day', 'danger_level_name_prev3day',
    'mountain_weather_change_wind_direction',
    'mountain_weather_change_hour_of_day_start',
    'mountain_weather_change_hour_of_day_stop',
    'mountain_weather_change_wind_speed',
    'mountain_weather_fl_hour_of_day_stop',
    'mountain_weather_fl_hour_of_day_start',
    'latest_observations', 'publish_time', 'date_valid',
    'mountain_weather_temperature_max_prev3day', 'mountain_weather_temperature_min_prev3day',
    'mountain_weather_temperature_max_prev2day',
    'mountain_weather_temperature_min_prev2day',
    'mountain_weather_temperature_max_prev1day',
    'mountain_weather_temperature_min_prev1day'
]
removed_ = [varsom_df.pop(v) for v in del_list]
Explanation: Eventually remove variables with many missing values.
End of explanation
# Find the amount of NaN values in each column
print(varsom_df.isnull().sum().sort_values(ascending=False))
# Compute the correlation matrix - works only on numerical variables.
corr = varsom_df.corr()
# Generate a mask for the upper triangle
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True
# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 11))
# Generate a custom diverging colormap
cmap = sns.diverging_palette(1000, 15, as_cmap=True)
# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.8, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
Explanation: Check again if there are still values missing...
need to replace these Nans with meaningful values or remove the feature.
End of explanation
#corr['avalanche_problem_1_cause_id'].sort_values(ascending=False)
#corr
#sns.pairplot(varsom_df.drop(['date_valid'], axis=1))
# Get all numerical features
num_feat = varsom_df._get_numeric_data().columns
num_feat
# let's see the details about remainig variables 
varsom_df.describe()
Explanation: We can see that some parameters are highly correlated. These are mainly the parameters belonging to the same avalanche problem. Depending on the ML algorithm we use we have to remove some of them.
End of explanation
varsom_df.to_csv('varsom_ml_preproc_3y.csv', index_label='index')
Explanation: Save data for further analysis
End of explanation
<END_TASK>