<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Calibrate all floats that have oxygen data
Create a local copy of Argo data, match it to World Ocean Atlas, and correct it to match climatology
Use the load_biofloat_cache.py script to create a local cache file of all the data we want to work with
Step1:  That's over 10 million measurements from over 42,000 profiles from 301 floats. The load_biofloat_cache.py script examined 559 floats for valid oxygen data. All of the profile data are in this file and the data from any float can be explored as demonstrated in the Notebooks explore_cached_biofloat_data.ipynb and explore_surface_oxygen_and_WOA.ipynb.
To compare these data against the World Ocean Atlas database the woa_calibration.py script can be executed in the home directory
Step2:  The inf and NaN values in the statistics of the gain indicate bad data, let's restrict data to a reasonably valid range of the measured o2sat.
Step3:  Plot the distribution of gains from this minimally qc'ed data &mdash; comparible to Figure 4 in Takeshita et al. (2013).
Step4:  Plot time series of gain for all the floats and fit an ordinary least squares regression to detect any drift in time of the gain for all the floats.
Step5:  Make a list of all the floats in qdf and assign a color to each.
Step6:  Make scatter plot of float oxygen saturation vs. World Ocean Atlas oxygen saturation.
Step7:  This first look at WOA oxygen calibration for all the Argo floats reveals that there are some issues with the data that warrant further exploration.
Get most recent profiles of all the floats in the cache file so that we can see their locations.
<ASSISTANT_TASK:>
Python Code:
from biofloat import ArgoData
from os.path import join, expanduser
ad = ArgoData(cache_file = join(expanduser('~'), 
     'biofloat_fixed_cache_age365_variablesDOXY_ADJUSTED-PSAL_ADJUSTED-TEMP_ADJUSTED.hdf'))
ocdf = ad.get_cache_file_oxy_count_df()
print ocdf.groupby('wmo').sum().sum()
print 'Float having DOXY_ADJUSTED data:', ocdf.wmo.count()
acdf = ad.get_cache_file_all_wmo_list()
print 'Number of floats examined:', len(acdf)
Explanation: Calibrate all floats that have oxygen data
Create a local copy of Argo data, match it to World Ocean Atlas, and correct it to match climatology
Use the load_biofloat_cache.py script to create a local cache file of all the data we want to work with:
bash
load_biofloat_cache.py --age 365 -v &gt;&gt; age365.out 2&gt;&amp;1 &amp;
This script takes several days to complete and needs to be monitored as it depends on Internet resources which occaisonally fail to respond. The load_biofloat_cache.py can be killed and restarted as necessary to deal with these interruptions (a scripts/load_biofloat_cache_watchdog.sh bash script is provided to do this automatically for you from cron on Unix systems). We save the logger output to a log file (age365.out) to facilitate analysis of the data load. When this script was executed in December of 2015 it produced a 1.7 GB file (biofloat_fixed_cache_age365_variablesDOXY_ADJUSTED-PSAL_ADJUSTED-TEMP_ADJUSTED.hdf) which we can explore using biofloat's ArgoData.
If you'd rather not wait the several days to create this file you may download it to your home directory from MBARI's anonymous FTP server: ftp://ftp.mbari.org/pub/biofloat/. An advantage of using this file is that it is updated nightly with new profile data.
End of explanation
import pandas as pd
df = pd.DataFrame()
with pd.HDFStore(join(expanduser('~'), 'woa_lookup_age365.hdf')) as s:
    wmo_list = ocdf.wmo
    for wmo in wmo_list:
        try:
            fdf = s.get(('/WOA_WMO_{}').format(wmo))
        except KeyError:
            pass
        if not fdf.dropna().empty:
            df = df.append(fdf)
print df.head()
print df.describe()
print 'Number of floats with corresponding WOA data:', len(df.index.get_level_values('wmo').unique())
Explanation: That's over 10 million measurements from over 42,000 profiles from 301 floats. The load_biofloat_cache.py script examined 559 floats for valid oxygen data. All of the profile data are in this file and the data from any float can be explored as demonstrated in the Notebooks explore_cached_biofloat_data.ipynb and explore_surface_oxygen_and_WOA.ipynb.
To compare these data against the World Ocean Atlas database the woa_calibration.py script can be executed in the home directory:
bash
woa_calibration.py --cache_file \
biofloat_fixed_cache_age365_variablesDOXY_ADJUSTED-PSAL_ADJUSTED-TEMP_ADJUSTED.hdf \
--results_file woa_lookup_age365.hdf -v
This script takes a few hours to complete, but once it's done we have surface monthly oxygen saturation data from Argo floats together with WOA 2013 oxygen saturarion data.
Let's load in the woa_calibration.py generated data into a DataFrame.
End of explanation
qdf = df.query('(o2sat > 50 ) & (o2sat < 200)')
qdf.describe()
Explanation: The inf and NaN values in the statistics of the gain indicate bad data, let's restrict data to a reasonably valid range of the measured o2sat.
End of explanation
%pylab inline
import pylab as plt
plt.rcParams['figure.figsize'] = (18.0, 4.0)
plt.style.use('ggplot')
ax = qdf.groupby('wmo').mean().gain.hist(bins=100)
ax.set_xlabel('Gain')
ax.set_ylabel('Count')
floats = qdf.index.get_level_values('wmo').unique()
ax.set_title(('Distribution of WOA calibrated gains from {} floats').format(len(floats)))
Explanation: Plot the distribution of gains from this minimally qc'ed data &mdash; comparible to Figure 4 in Takeshita et al. (2013).
End of explanation
qdf.head()
plt.rcParams['figure.figsize'] = (18.0, 8.0)
ax = qdf.unstack(level='wmo').gain.plot()
ax.set_ylabel('Gain')
ax.set_title(('Calculated gain factor for {} floats').format(len(floats)))
ax.legend_.remove()
Explanation: Plot time series of gain for all the floats and fit an ordinary least squares regression to detect any drift in time of the gain for all the floats.
End of explanation
wmo_list = qdf.index.get_level_values('wmo').unique()
colors = cm.spectral(np.linspace(0, 1, len(wmo_list)))
print 'Number of floats with reasonable oxygen saturation values:', len(wmo_list)
Explanation: Make a list of all the floats in qdf and assign a color to each.
End of explanation
plt.rcParams['figure.figsize'] = (18.0, 8.0)
fig, ax = plt.subplots(1, 1)
for wmo, c in zip(wmo_list, colors):
    ax.scatter(qdf.xs(wmo, level='wmo')['o2sat'], qdf.xs(wmo, level='wmo')['woa_o2sat'], c=c)
ax.set_xlim([40, 200])
ax.set_ylim([40, 200])
ax.set_xlabel('Float o2sat (%)')
ax.set_ylabel('WOA o2sat (%)')
Explanation: Make scatter plot of float oxygen saturation vs. World Ocean Atlas oxygen saturation.
End of explanation
%%time
ad.set_verbosity(0)
df1 = ad.get_float_dataframe(wmo_list, update_cache=False, max_profiles=4)
from mpl_toolkits.basemap import Basemap
m = Basemap(llcrnrlon=15, llcrnrlat=-90, urcrnrlon=390, urcrnrlat=90, projection='cyl')
m.fillcontinents(color='0.8')
df1m = df1.groupby(level=['wmo','lon','lat']).mean()
for wmo, c in zip(wmo_list, colors):
    try:
        lons = df1m.xs(wmo, level='wmo').index.get_level_values('lon')
        lats = df1m.xs(wmo, level='wmo').index.get_level_values('lat')
        try:
            m.scatter(lons, lats, latlon=True, color=c)
        except IndexError:
            # Some floats have too few points
            pass
        lon, lat = lons[0], lats[0]
        if lon < 0:
            lon += 360
        plt.text(lon, lat, wmo)
    except KeyError:
        pass
Explanation: This first look at WOA oxygen calibration for all the Argo floats reveals that there are some issues with the data that warrant further exploration.
Get most recent profiles of all the floats in the cache file so that we can see their locations.
End of explanation
<END_TASK>