<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Work with different data formats (binary, netCDF, .mat)
Nikolay Koldunov
koldunovn@gmail.com
This is part of Python for Geosciences notes.
================
Step1:  Binary data
Open binary
Data from Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data.
Step2:  Create file id
Step3:  We use uint8 data type. 
List of numpy data types
The file format consists of a 300-byte descriptive header followed by a two-dimensional array.
Step4:  Reshape
Step5:  Simple visualisation of array with imshow (Matplotlib function)
Step6:  To convert to the fractional parameter range of 0.0 to 1.0, divide the scaled data in the file by 250.
Step7:  Let's mask all land and missing values
Step8:  Masking in this case is similar to using NaN in Matlab. More about NumPy masked arrays
Save binary
Step9:  In order to work with other data formats we need to use one of the SciPy submodules
Step10:  We are going to download Polar science center Hydrographic Climatology (PHC) for January in Matlab format.
Step11:  Open file
Step12:  We can look at the names of variables stored in the file
Step13:  We need only PTEMP1 (3d potential temperature).
Step14:  Check variable's shape
Step15:  Show surface level
Step16:  Open netCDF files
Scipy have function for working with netCDF files, and you can import it with
Step17:  I am going to download NCEP reanalysis data. Surface 4 daily air temperature for 2012.
Step18:  Create file id
Step19:  It's not really file id, it's netcdf_file object, that have some methods and attributes
Step20:  list variables
Step21:  Access information about variables
Step22:  This time we create netcdf_variable object, that contain among other things attributes of the netCDF variable as well as data themselves.
Step23:  We can access the data by simply using array syntax. Here we show first time step of our data set
Step24:  Save netCDF file
Minimalistic variant 
Step25:  More descriptive variant
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
Explanation: Work with different data formats (binary, netCDF, .mat)
Nikolay Koldunov
koldunovn@gmail.com
This is part of Python for Geosciences notes.
================
End of explanation
!wget ftp://sidads.colorado.edu/pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/monthly/nt_200709_f17_v01_n.bin
Explanation: Binary data
Open binary
Data from Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data.
End of explanation
ice = np.fromfile('nt_200709_f17_v01_n.bin', dtype='uint8')
Explanation: Create file id:
End of explanation
ice = ice[300:]
Explanation: We use uint8 data type. 
List of numpy data types
The file format consists of a 300-byte descriptive header followed by a two-dimensional array.
End of explanation
ice = ice.reshape(448,304)
Explanation: Reshape
End of explanation
plt.imshow(ice)
plt.colorbar();
Explanation: Simple visualisation of array with imshow (Matplotlib function):
End of explanation
ice = ice/250.
plt.imshow(ice)
plt.colorbar();
Explanation: To convert to the fractional parameter range of 0.0 to 1.0, divide the scaled data in the file by 250.
End of explanation
ice_masked = np.ma.masked_greater(ice, 1.0)
plt.imshow(ice_masked)
plt.colorbar();
Explanation: Let's mask all land and missing values:
End of explanation
fid = open('My_ice_2007.bin', 'wb')
ice.tofile(fid)
fid.close()
Explanation: Masking in this case is similar to using NaN in Matlab. More about NumPy masked arrays
Save binary
End of explanation
from scipy.io import loadmat
Explanation: In order to work with other data formats we need to use one of the SciPy submodules:
SciPy
<img  height="100" src="files/scipy.png" >
General purpose scientific library (that consist of bunch of sublibraries) and builds on NumPy arrays.
Special functions (scipy.special)
Integration (scipy.integrate)
Optimization (scipy.optimize)
Interpolation (scipy.interpolate)
Fourier Transforms (scipy.fftpack)
Signal Processing (scipy.signal)
Linear Algebra (scipy.linalg)
Sparse Eigenvalue Problems (scipy.sparse)
Statistics (scipy.stats)
Multi-dimensional image processing (scipy.ndimage)
File IO (scipy.io)
We are going to use only scipy.io library.
scipy.io
Open .mat files
First we have to load function that works with Matlab files:
End of explanation
!wget https://www.dropbox.com/s/0kuzvz03gw6d393/PHC_jan.mat
Explanation: We are going to download Polar science center Hydrographic Climatology (PHC) for January in Matlab format.
End of explanation
all_variables = loadmat('PHC_jan.mat')
Explanation: Open file:
End of explanation
all_variables.keys()
Explanation: We can look at the names of variables stored in the file:
End of explanation
temp = np.array(all_variables['PTEMP1'])
Explanation: We need only PTEMP1 (3d potential temperature).
End of explanation
temp.shape
Explanation: Check variable's shape:
End of explanation
plt.imshow(temp[0,:,:])
plt.colorbar();
Explanation: Show surface level:
End of explanation
from netCDF4 import Dataset
Explanation: Open netCDF files
Scipy have function for working with netCDF files, and you can import it with:
    from scipy.io import netcdf
However it only supports netCDF3 format. It is better to use python netcdf4 module that have a lot of nice functionality. Moreover NCEP reanalysis data, that we are going to work with are in netCDF4 format.
Import nessesary function:
End of explanation
!wget ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis/surface/air.sig995.2012.nc
#Alternative for the times of US goverment shutdowns:
#!wget http://database.rish.kyoto-u.ac.jp/arch/ncep/data/ncep.reanalysis/surface/air.sig995.2012.nc
Explanation: I am going to download NCEP reanalysis data. Surface 4 daily air temperature for 2012.
End of explanation
fnc = Dataset('air.sig995.2012.nc')
Explanation: Create file id:
End of explanation
fnc.description
fnc.history
Explanation: It's not really file id, it's netcdf_file object, that have some methods and attributes:
End of explanation
fnc.variables
Explanation: list variables
End of explanation
air = fnc.variables['air']
Explanation: Access information about variables
End of explanation
air.actual_range
air.long_name
air.units
air.shape
Explanation: This time we create netcdf_variable object, that contain among other things attributes of the netCDF variable as well as data themselves.
End of explanation
plt.imshow(air[0,:,:])
plt.colorbar();
Explanation: We can access the data by simply using array syntax. Here we show first time step of our data set:
End of explanation
!rm test_netcdf.nc
fw = Dataset('test_netcdf.nc', 'w')
fw.createDimension('t', 1464)
fw.createDimension('y', 73)
fw.createDimension('x', 144)
air_var = fw.createVariable( 'air','float32', ('t', 'y', 'x'))
air_var[:] = air[:]
fw.close()
Explanation: Save netCDF file
Minimalistic variant :)
End of explanation
!rm test_netcdf.nc
fw = Dataset('test_netcdf.nc', 'w')
fw.createDimension('TIME', 1464)
fw.createDimension('LATITUDE', 73)
fw.createDimension('LONGITUDE', 144)
time = fw.createVariable('TIME', 'f', ('TIME',))
time[:] = fnc.variables['time'][:]
time.units = 'hours since 1-1-1 00:00:0.0' 
lat  = fw.createVariable('LATITUDE', 'f', ('LATITUDE',))
lat[:] = fnc.variables['lat'][:]
lon = fw.createVariable('LONGITUDE', 'f', ('LONGITUDE',))
lon[:] = fnc.variables['lon'][:]
ha = fw.createVariable('New_air','f', ('TIME', 'LATITUDE', 'LONGITUDE'))
ha[:] = air[:]
ha.missing_value = -9999.
fw.close()
Explanation: More descriptive variant:
End of explanation
<END_TASK>