<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <style>
.answers { 
    visibility
Step1:  Objectives
There will be hands-on exercises for a range of GEOS-Chem analysis tools, including plotting maps with overlays, converting bpch to netcdf, editing bpch/netcdf files, and evaluating GEOS-Chem against AQSD or ICARTT campaign files.
Produce publication quality graphics
Perform standard model performance evaluations
Create emission perturbations
Add custom modifications to each exercise
Requirements
Some scripting experience (R, Python, Perl, bash, or csh). Attendees who do not have experience can follow my on-line Python-primer (http
Step2:  Open Notebook
Intro to python/ipython/jupyter
Options
python - least burden on system
ipython - least burden on user
jupyter - least burden on next user
Running interactively python/ipython
Terminal
Integrated Development Environments
Notebook
Running saved scripts
from terminal $ python /path/to/file.py
from python `>>> execfile('/path/to/file.py')
from ipython In[1] %run /path/to/file.py
from spyder open in editor
*Windows replace $ with C
Step3:  Basic Types
int, float
Step5:  strings
Step6:  slices
slice([start, ]stop[, step]) or [start]
Step7:  ANSWERS Hidden
<div style="visibility
Step8:  Intro to numpy/scipy/matplotlib
numpy - numerical python (aka, do math fast)
scipy - a collection of scientific tools (aka, do stats or solve equations)
matplotlib - a library for plotting (aka, make figures for my article)
numpy
numpy - Numerical Python (numpy.org)
Fast library for numerical mathematics
Large array processing (element by element)
Large matrix processing (using standard matrix)
Support for masked arrays
LAPACK for linear algebra
Make a random "ozone" array
Step9:  Which dimension is 2 long? 3 long? 4 long? 5 long?
Make an "averaging" kernel matrix
Step10:  Which dimension is the input layer and which is the output layer?
Product sums and matrices?
Step11:  ozone[0, 
Step12:  CHECK POINT
Change the code to operate on all times, latitude, and longitudes without a loop.
Step13:  ANSWERS Hidden
<div style="visibility
Step14:  XKCD, Jelly beans, and Type I Error?
<ASSISTANT_TASK:>
Python Code:
# Prepare my slides
%pylab inline
%cd working
Explanation: <style>
.answers { 
    visibility: hidden;
}
</style>
Python Analysis for GEOS-Chem
Author: Barron H. Henderson
Attendees will learn how to leverage Python to interact with air pollution-related model and observational data. Air research and application relies on big data. In addition to the challenge presented by data size, researchers must understand a multitude of formats and meta-data standards. For example, CMAQ, CAMx, and GEOS-Chem all use different formats and different meta-data conventions. This tutorial provides format-independent and convention-independent tools.
End of explanation
# ipython
!curl -kLO http://github.com/barronh/GCandPython/archive/master.zip
import zipfile
zf = zipfile.ZipFile('master.zip')
zf.extractall()
%mv GCandPython-master GCandPython
Explanation: Objectives
There will be hands-on exercises for a range of GEOS-Chem analysis tools, including plotting maps with overlays, converting bpch to netcdf, editing bpch/netcdf files, and evaluating GEOS-Chem against AQSD or ICARTT campaign files.
Produce publication quality graphics
Perform standard model performance evaluations
Create emission perturbations
Add custom modifications to each exercise
Requirements
Some scripting experience (R, Python, Perl, bash, or csh). Attendees who do not have experience can follow my on-line Python-primer (http://www.barronh.com/software/tutorials/python-tutorial) to satisfy the requiremnt.
A computer with either
Windows, Linux, or Mac; a text editor; and Anaconda 3.5 installed.
or a computer and an account on wakari.io
See detailed install instructions at https://github.com/barronh/GCandPython/blob/master/PythonInstallation.ipynb
Download Workshop Materials
https://github.com/barronh/GCandPython
"Clone or Download"
"Download Zip"
Move it somewhere good
Unzip it
Fork it.
With Anaconda
Open Anaconda Navigator
Open ipython
Run the code below
End of explanation
def foo(bar):
    return 'foo.' + bar
print(foo('bar'))
try:
    print(foo(1))
except Exception as e:
    print(e)
Explanation: Open Notebook
Intro to python/ipython/jupyter
Options
python - least burden on system
ipython - least burden on user
jupyter - least burden on next user
Running interactively python/ipython
Terminal
Integrated Development Environments
Notebook
Running saved scripts
from terminal $ python /path/to/file.py
from python `>>> execfile('/path/to/file.py')
from ipython In[1] %run /path/to/file.py
from spyder open in editor
*Windows replace $ with C:\&gt;
Basic Functions
def foo(bar1, bar2, foo = 1, a = 2):
    return 'foo' + bar1
created with def name and call signature followed by colon
defined by commands following signature until the first unindented command
Most Useful Function... ever.
help:
- ? in ipython or jupyter
- it will return a docstring for that element or type
Really Useful Functions
type:
- All things in python are objects (even classes).
- type is a function object that returns the type of another object.
dir:
- List of methods and properties
print:
- display representation of object
End of explanation
1 + 1
1 + 1.
1 / 2.
1 // 2
Explanation: Basic Types
int, float:
Literals 1 is int, 1. is float
64-bit precision values (i.e., float is real*8)
str:
literal '...' or "..."
multi-line uses triple quotes
indexed (aka subset) with square braces and "slices"
version 3 uses system dependent string types (use encode or decode methods0
*A method is a bound function...
Integers and Floats
End of explanation
.encode()
"a b c".split()
''.join(['h', 'ow', ' ', 'c', 'an', ' ', 'I ' 'su', 'bse', 't?'])
Explanation: strings
End of explanation
teststr = 'this is not fun.'
print(teststr[:8])
Explanation: slices
slice([start, ]stop[, step]) or [start]:stop[:step]
slices define the subset in terms of 0-based indices
stop is the only required element.
Missing starts are implied (start: 0, end: len(utterable))
CHECK POINT
How can you rearange the string below to say "this is so fun"?
The final answer will fill in the ? with numbers (positive or negative).
print(teststr[:?]+teststr[?:?:?]+teststr[?:])
End of explanation
def foo(pieces):
    out = type(pieces[0])()
    for i, piece in enumerate(pieces):
        out += piece
    return out
a = foo('12345')
b = foo([1, 2, 3, 4, 5])
c = foo({0: 1, 1: 2, 2: 3, 3: 4, 4: 5})
print(a, b, c)
Explanation: ANSWERS Hidden
<div style="visibility: hidden">
```
print(teststr[:7], end = ' ')
print(teststr[3:-1:6], end = '')
print(teststr[-5:])
```
</div>
Iterable Types
lists : [item1, item2, item3, ...]
created with square braces
indexed with square braces (indexing is always square braces, not just lists) and "slices"
tuples are like lists, but defined with parentheses and the items are immutable.
dict : {key1: value1, key2: value2, ...}
indexed with keys
iterate keys
iterables
strings, lists, dict, and tuples are iterables.
For any iterable, you can loop over the elements with a for loop
Each character in a string...
Each item in a list or tuple
Each key in a dictionary
enumerate is a special function that returns the element number (0-based) and the element as a tuple
CHECK POINT
The code below has three parts:
first cell define a function
subsequent cells call function
What are the types and values of a, b, and c?
End of explanation
%pylab tk
np.random.seed(50)
ozone = (np.random.normal(size = 2*3*4*5) + 40).reshape(2,3,4,5)
print(ozone.ndim)
print(ozone.shape, ozone[0, :, 2, 3].shape, ozone[0, :, 2, 3])
print(ozone.mean())
print(np.percentile(ozone, [5, 95]))
Explanation: Intro to numpy/scipy/matplotlib
numpy - numerical python (aka, do math fast)
scipy - a collection of scientific tools (aka, do stats or solve equations)
matplotlib - a library for plotting (aka, make figures for my article)
numpy
numpy - Numerical Python (numpy.org)
Fast library for numerical mathematics
Large array processing (element by element)
Large matrix processing (using standard matrix)
Support for masked arrays
LAPACK for linear algebra
Make a random "ozone" array
End of explanation
averagekernel = np.array([[0.5, 0.35, 0.15],
                          [0.25, 0.5, 0.25],
                          [0.1, 0.4, 0.5]])
averagekernelm = np.matrix(averagekernel)
print(averagekernel.shape)
print(averagekernelm.shape)
print(averagekernelm.T.shape)
averagekernel = np.array([0.25, 0.5, 0.25])
averagekernelm = np.matrix(averagekernel)
Explanation: Which dimension is 2 long? 3 long? 4 long? 5 long?
Make an "averaging" kernel matrix
End of explanation
(ozone[0, :, 2, 3] * averagekernel).sum()
Explanation: Which dimension is the input layer and which is the output layer?
Product sums and matrices?
End of explanation
ozone[0, :, 2, 3] * averagekernelm.T
Explanation: ozone[0, :, 2, 3] * averagekernelm
End of explanation
out = np.zeros_like(ozone[:, 0])
for t in range(ozone.shape[0]):
    for j in range(ozone.shape[2]):
        for i in range(ozone.shape[3]):
            out[t, j, i] = ozone[t, :, j, i] * averagekernelm.T
out
Explanation: CHECK POINT
Change the code to operate on all times, latitude, and longitudes without a loop.
End of explanation
from scipy import constants
?constants
from scipy import stats
?stats
from scipy.stats import mstats
?mstats
Explanation: ANSWERS Hidden
<div style="visibility: hidden">
newout = (ozone * averagekernel[None, :, None, None])
(out == out).all()
</div>
Hint: Add singleton dimensions to force broadcasting.
scipy
scipy - Scientific Python (scipy.org)
Good for physical constants
Large statistical library
Large interpolation library
Including masked array statistics
Ordinary Differential Equation Solving
End of explanation
%%timeit -n 20
from scipy.stats import ttest_ind, mannwhitneyu
mdifferent = []
tdifferent = []
n = 100
for i in range(n):
    a = np.exp(np.random.normal(size = 20))
    b = np.exp(np.random.normal(size = 20))
    tresult = ttest_ind(a, b)
    mresult = mannwhitneyu(a, b)
    if tresult.pvalue < 0.05:
        tdifferent.append(i);
    if mresult.pvalue < 0.05:
        mdifferent.append(i);
        
print(len(mdifferent)/n, end = '/')
print(len(tdifferent)/n, end = ', ')
np.array([0.06, 0.06, 0.01, 0.04, 0.05, 0.02, 0.06, 0.07, 0.02, 0.07, 0.04, 0.05, 0.03, 0.08, 0.04, 0.03, 0.03, 0.02, 0.04, 0.01, 0.06, 0.04, 0.05, 0.01, 0.03, 0.04, 0.02, 0.04, 0.03, 0.02, 0.04, 0.03, 0.06, 0.03, 0.02, 0.07, 0.05, 0.02, 0.01, 0.05, 0.03, 0.04, 0.04, 0.03, 0.03, 0.04, 0.04, 0.06, 0.03, 0.05, 0.05, 0.03, 0.02, 0.0,\
          0.04, 0.05, 0.02, 0.07, 0.05, 0.05]).mean()
Explanation: XKCD, Jelly beans, and Type I Error?
End of explanation
<END_TASK>