<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <table style="width
Step1:  Import and clean data
Set the directory path and look for all netcdf files that correspond to the model drivers and target.
Step2:  Define a function to extract the variable values from each netCDF4 file. Variables are flattened from a 3 dimensional array to 1 dimensional version, pooling all values both spatially and temporily. 
Don't know if this is the correct way to do this, but will come back to it once I understand the model (and its optimisation) better.
Step3:  Execute the above function on all netCDF4 file paths.
Step4:  Turn this into a dataframe for the analysis.
Step5:  Check that we've built it correctly.
Step6:  Export this to disk to be used by the analysis notebook - used gzip compression to save on space. Beware, because of there are approximation 10 million rows of data, this may take some time.
<ASSISTANT_TASK:>
Python Code:
# data munging and analytical libraries 
import re
import os
import numpy as np
import pandas as pd
from netCDF4 import Dataset 
# graphical libraries
import matplotlib.pyplot as plt
%matplotlib inline
# set paths
outPath = "../data/globfire.csv"
Explanation: <table style="width: 100%; border-collapse: collapse;" border="0">
<tr>
<td><b>Created:</b> Monday 30 January 2017</td>
<td style="text-align: right;"><a href="https://www.github.com/rhyswhitley/fire_limitation">github.com/rhyswhitley/fire_limitation</td>
</tr>
</table>
<div>
<center>
<font face="Times">
<br>
<h1>Quantifying the uncertainity of a global fire limitation model using Bayesian inference</h1>
<h2>Part 1: Staging data for analysis</h2>
<br>
<br>
<sup>1,* </sup>Douglas Kelley, 
<sup>2 </sup>Ioannis Bistinas, 
<sup>3, 4 </sup>Chantelle Burton, 
<sup>1 </sup>Tobias Marthews, 
<sup>5 </sup>Rhys Whitley
<br>
<br>
<br>
<sup>1 </sup>Centre for Ecology and Hydrology, Maclean Building, Crowmarsh Gifford, Wallingford, Oxfordshire, United Kingdom
<br>
<sup>2 </sup>Vrije Universiteit Amsterdam, Faculty of Earth and Life Sciences, Amsterdam, Netherlands
<br>
<sup>3 </sup>Met Office United Kingdom, Exeter, United Kingdom
<br>
<sup>4 </sup>Geography, University of Exeter, Exeter, United Kingdom
<br>
<sup>5 </sup>Natural Perils Pricing, Commercial & Consumer Portfolio & Pricing, Suncorp Group, Sydney, Australia
<br>
<br>
<h3>Summary</h3>
<hr>
<p> 
This notebook aims to process the separate netCDF4 files for the model drivers (X<sub>i=1, 2, ... M</sub>) and model target (Y) into a unified tabular data frame, exported as a compressed comma separated value (CSV) file. This file is subsequently used in the Bayesian inference study that forms the second notebook in this experiment. The advantage of the pre-processing the data separately to the analysis allows for it be quickly staged on demand. Of course other file formats may be more advantageous for greater compression (e.g. SQLite3 database file).
</p>
<br>
<b>You will need to run this notebook to prepare the dataest before you attempt the Bayesian analysis in Part 2</b>.
<br>
<br>
<br>
<i>Python code and calculations below</i>
<br>
<hr>
</font>
</center>
</div>
Load libraries
End of explanation
driver_paths = [os.path.join(dp, f) for (dp, _, fn) in os.walk("../data/raw/") for f in fn if f.endswith('.nc')]
driver_names = [re.search('^[a-zA-Z_]*', os.path.basename(fp)).group(0) for fp in driver_paths]
file_table = pd.DataFrame({'filepath': driver_paths, 'file_name': driver_names})
file_table
Explanation: Import and clean data
Set the directory path and look for all netcdf files that correspond to the model drivers and target.
End of explanation
def nc_extract(fpath):
    print("Processing: {0}".format(fpath))
    with Dataset(fpath, 'r') as nc_file:
        gdata = np.array(nc_file.variables['variable'][:,:,:])
        gflat = gdata.flatten()
        if type(gdata) == np.ma.core.MaskedArray:
            return gflat[~gflat.mask].data
        else:
            return gflat.data
Explanation: Define a function to extract the variable values from each netCDF4 file. Variables are flattened from a 3 dimensional array to 1 dimensional version, pooling all values both spatially and temporily. 
Don't know if this is the correct way to do this, but will come back to it once I understand the model (and its optimisation) better.
End of explanation
values = [nc_extract(dp) for dp in driver_paths]
Explanation: Execute the above function on all netCDF4 file paths.
End of explanation
# turn list into a dataframe
fire_df = pd.DataFrame(np.array(values).T, columns=driver_names)
# replace null flags with pandas null
fire_df.replace(-3.4e38, np.nan, inplace=True)
# drop all null rows (are ocean and not needed in optim)
fire_df.dropna(inplace=True)
Explanation: Turn this into a dataframe for the analysis.
End of explanation
fire_df.head()
Explanation: Check that we've built it correctly.
End of explanation
savepath = os.path.expanduser(outPath)
fire_df.to_csv(savepath, index=False)
Explanation: Export this to disk to be used by the analysis notebook - used gzip compression to save on space. Beware, because of there are approximation 10 million rows of data, this may take some time.
End of explanation
<END_TASK>