<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Import useful stuff and define ancillary functions
Step1:  Load train data
Using pandas' read_csv with all the defaults
Step2:  Define columns
Step3:  Parse the weird date format of the date column
Step4:  Drop duplicate rows
Step5:  Drop duplicate columns
Step6:  Remove columns with only NaNs
Step7:  Vectorize Datetime colums
Create dataframes to separately store the year, month and day information of the date columns
Step8:  Delete the original date columns and join the years, months and days DataFrames with the original DataFrame
Step9:  Eliminate columns with 0 variance
See which columns have low standard deviation
Step10:  Eliminate all columns with standard deviation equal to 0
Step11:  Save preprocessed data to another csv file
<ASSISTANT_TASK:>
Python Code:
%pylab inline
%load_ext autoreload
%autoreload 2
from __future__ import division
from collections import defaultdict, namedtuple
import cPickle as pickle
from datetime import datetime, timedelta
import dateutil
from functools import partial
import inspect
import json
import os
import re
import sys
import numpy as np
import pandas as pd
Explanation: Import useful stuff and define ancillary functions
End of explanation
if os.name == 'nt':
    TRAIN_PATH = r'D:\train.csv'
    PTRAIN_PATH = r'D:\train_preprocessed_float_string_date.csv'
    TEST_PATH = r'D:\test.csv'
    GOOGNEWS_PATH = r'D:\GoogleNews-vectors-negative300.bin.gz'
    VOCAB_PATH = r'D:\big.txt'
else:
    TRAIN_PATH = r'/media/mtambos/speedy/train.csv'
    PTRAIN_PATH = r'/media/mtambos/speedy/train_preprocessed_float_string_date.csv'
    TEST_PATH = r'/media/mtambos/speedy/test.csv'
    GOOGNEWS_PATH = r'/media/mtambos/speedy/GoogleNews-vectors-negative300.bin.gz'
    VOCAB_PATH = r'/media/mtambos/speedy/big.txt'
#df_orig = pd.read_csv(TRAIN_PATH, index_col="ID")
df = pd.read_csv(PTRAIN_PATH, index_col="ID")
#df
Explanation: Load train data
Using pandas' read_csv with all the defaults
End of explanation
date_cols = ['VAR_0073', 'VAR_0075', 'VAR_0156',
             'VAR_0157', 'VAR_0158', 'VAR_0159',
             'VAR_0166', 'VAR_0167', 'VAR_0168',
             'VAR_0169', 'VAR_0176', 'VAR_0177',
             'VAR_0178', 'VAR_0179', 'VAR_0204',
             'VAR_0217', 'VAR_0294', 'VAR_0314']
Explanation: Define columns
End of explanation
def parse_date_str(date_val):
    if isinstance(date_val, datetime):
        return date_val
    date_val = str(date_val).lower()
    date_val = None if date_val == 'nan' else date_val
    if date_val is None:
        return pd.NaT
    date_val = date_val[:-2] if date_val[-2:] == '.0' else date_val
    try:
        return datetime.strptime(date_val, '%d%b%y:%H:%M:%S')
    except:
        try:
            return datetime.strptime(date_val, '%d%b%y')
        except:
            try:
                return datetime.strptime(date_val, '%Y')
            except:
                print date_val
                return pd.NaT
df_date_cols = df[date_cols].applymap(parse_date_str).astype('datetime64[ns]')
df_date_cols.describe()
df[date_cols] = df_date_cols
Explanation: Parse the weird date format of the date column
End of explanation
df.drop_duplicates(inplace=True)
Explanation: Drop duplicate rows
End of explanation
cols_to_drop = set()
for i, col in enumerate(date_cols):
    for col2 in date_cols[i+1:]:
        if (df[col] == df[col2]).all():
            cols_to_drop.add(col2)
cols_to_drop
Explanation: Drop duplicate columns
End of explanation
nan_cols = df.isnull().all()
nan_cols = nan_cols.index[nan_cols].tolist()
nan_cols
Explanation: Remove columns with only NaNs
End of explanation
df[date_cols] = df[date_cols].astype('datetime64[ns]')
years = pd.DataFrame(columns=[c+'_year' for c in date_cols], index=df.index, dtype=np.int)
months = pd.DataFrame(columns=[c+'_month' for c in date_cols], index=df.index, dtype=np.int)
days = pd.DataFrame(columns=[c+'_day' for c in date_cols], index=df.index, dtype=np.int)
for c in date_cols:
    dateIndex = pd.DatetimeIndex(df[c])
    years[c+'_year'] = dateIndex.year
    months[c+'_month'] = dateIndex.month
    days[c+'_day'] = dateIndex.day
Explanation: Vectorize Datetime colums
Create dataframes to separately store the year, month and day information of the date columns
End of explanation
df = df.drop(date_cols, axis=1)
df = df.join(years)
df = df.join(months)
df = df.join(days)
date_cols = years.columns.tolist() + months.columns.tolist() + days.columns.tolist()
Explanation: Delete the original date columns and join the years, months and days DataFrames with the original DataFrame
End of explanation
df_desc = df.describe()
df_desc[sorted(df_desc.columns, key=lambda x: df_desc.loc['std', x])]
Explanation: Eliminate columns with 0 variance
See which columns have low standard deviation
End of explanation
std_series = df_desc.loc['std', :]
null_std_cols = std_series[std_series == 0]
df = df.drop(null_std_cols.index, axis=1)
Explanation: Eliminate all columns with standard deviation equal to 0
End of explanation
df.to_csv(PTRAIN_PATH)
with open('date_cols.pickle', 'wb') as fp:
    pickle.dump(date_cols, fp)
Explanation: Save preprocessed data to another csv file
End of explanation
<END_TASK>