<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 To determine what is in the file, check the header attribute on the file object
Step1:  Ok, lets pull in the name variable to see what we have
Step2:  Now obtain per capital incomes in 1929 which is in the column associated with 1929
Step3:  And now 2009
Step4:  These are read into regular Python lists which are not particularly well suited to efficient data analysis. So let's convert them to numpy arrays
Step5:  Much better. But pulling these in and converting them a column at a time is tedious and error prone. So we will do all of this in a list comprehension.
Step6:  Spaghetti Plot
Step7:  Kernel Density (univariate, aspatial)
Step8:  So the distribution is becoming less dispersed over time.
But what about internal mixing? Do poor (rich) states remain poor (rich), or is there movement within the distribuiton over time?
Markov Chains
Step9:  State Per Capita Incomes
Step10:  Put series into cross-sectional quintiles (i.e., quintiles for each year)
Step11:  we are looping over the rows of y which is ordered Txn (rows are cross sections, row 0 is the cross-section for period 0
Step12:  For a state with income in the first quintile, it takes on average 11.5 years for it to first enter the second quintile, 29.6 to get to the third quintile, 53.4 years to enter the fourth, and 103.6 years to reach the richest quintile.
But, this approach assumes the movement of a state in the income distribution is independent of the movement of its neighbors or the position of the neighbors in the distribution. Does spatial context matter?
Dynamics of Spatial Dependence
Read in a GAL file to construct our W
Step13:  Spatial Markov
Step14:  LISA Markov
Step15:  Test of independence of own chains and lag chains
<ASSISTANT_TASK:>
Python Code:
f.header
Explanation: To determine what is in the file, check the header attribute on the file object:
End of explanation
name = f.by_col('Name')
name
Explanation: Ok, lets pull in the name variable to see what we have
End of explanation
y1929 = f.by_col('1929')
y1929
Explanation: Now obtain per capital incomes in 1929 which is in the column associated with 1929
End of explanation
y2009 = f.by_col("2009")
y2009
Explanation: And now 2009
End of explanation
y2009 = np.array(y2009)
y2009
Explanation: These are read into regular Python lists which are not particularly well suited to efficient data analysis. So let's convert them to numpy arrays
End of explanation
Y = np.array( [ f.by_col(str(year)) for year in range(1929,2010) ] ) * 1.0
Y.shape
Y = Y.transpose()
Y.shape
years = np.arange(1929,2010)
plot(years,Y[0])
RY = Y / Y.mean(axis=0)
plot(years,RY[0])
name = np.array(name)
np.nonzero(name=='Ohio')
plot(years, RY[32], label='Ohio')
plot(years, RY[0], label='Alabama')
legend()
Explanation: Much better. But pulling these in and converting them a column at a time is tedious and error prone. So we will do all of this in a list comprehension.
End of explanation
for row in RY:
    plot(years, row)
Explanation: Spaghetti Plot
End of explanation
from scipy.stats.kde import gaussian_kde
density = gaussian_kde(Y[:,0])
Y[:,0]
density = gaussian_kde(Y[:,0])
minY0 = Y[:,0].min()*.90
maxY0 = Y[:,0].max()*1.10
x = np.linspace(minY0, maxY0, 100)
plot(x,density(x))
d2009 = gaussian_kde(Y[:,-1])
minY0 = Y[:,-1].min()*.90
maxY0 = Y[:,-1].max()*1.10
x = np.linspace(minY0, maxY0, 100)
plot(x,d2009(x))
minR0 = RY.min()
maxR0 = RY.max()
x = np.linspace(minR0, maxR0, 100)
d1929 = gaussian_kde(RY[:,0])
d2009 = gaussian_kde(RY[:,-1])
plot(x, d1929(x))
plot(x, d2009(x))
plot(x, d1929(x), label='1929')
plot(x, d2009(x), label='2009')
legend()
for cs in RY.T: # take cross sections
    plot(x, gaussian_kde(cs)(x))
    
sigma = RY.std(axis=0)
plot(years, sigma)
ylabel('s')
xlabel('year')
title("Sigma-Convergence")
Explanation: Kernel Density (univariate, aspatial)
End of explanation
c = np.array([['b','a','c'],
['c','c','a'],
['c','b','c'],
['a','a','b'],
['a','b','c']])
c
m = ps.Markov(c)
m.classes
m.transitions
m.p
Explanation: So the distribution is becoming less dispersed over time.
But what about internal mixing? Do poor (rich) states remain poor (rich), or is there movement within the distribuiton over time?
Markov Chains
End of explanation
f = ps.open(ps.examples.get_path("usjoin.csv"))
pci = np.array([f.by_col[str(y)] for y in range(1929,2010)])
pci.shape
Explanation: State Per Capita Incomes
End of explanation
q5 = np.array([ps.Quantiles(y).yb for y in pci]).transpose()
q5.shape
q5[:,0]
pci.shape
pci[0]
Explanation: Put series into cross-sectional quintiles (i.e., quintiles for each year)
End of explanation
m5 = ps.Markov(q5)
m5.classes
m5.transitions
m5.p
m5.steady_state
fmpt = ps.ergodic.fmpt(m5.p)
fmpt
Explanation: we are looping over the rows of y which is ordered Txn (rows are cross sections, row 0 is the cross-section for period 0
End of explanation
w = ps.open(ps.examples.get_path("states48.gal")).read()
w.n
w.transform = 'R'
mits = [ps.Moran(cs, w) for cs in RY.T]
res = np.array([(m.I, m.EI, m.p_sim, m.z_sim) for m in mits])
plot(years, res[:,0], label='I')
plot(years, res[:,1], label='E[I]')
title("Moran's I")
legend()
plot(years, res[:,-1])
ylim(0,7.0)
title('z-values, I')
Explanation: For a state with income in the first quintile, it takes on average 11.5 years for it to first enter the second quintile, 29.6 to get to the third quintile, 53.4 years to enter the fourth, and 103.6 years to reach the richest quintile.
But, this approach assumes the movement of a state in the income distribution is independent of the movement of its neighbors or the position of the neighbors in the distribution. Does spatial context matter?
Dynamics of Spatial Dependence
Read in a GAL file to construct our W
End of explanation
pci.shape
pci = pci.T
pci.shape
rpci = pci / pci.mean(axis=0)
rpci[:,0]
rpci[:,0].mean()
sm = ps.Spatial_Markov(rpci, w, fixed=True, k=5)
sm.p
for p in sm.P:
    print p
sm.S
for f in sm.F:
    print f
Explanation: Spatial Markov
End of explanation
lm = ps.LISA_Markov(pci,w)
lm.classes 
lm.transitions
np.set_printoptions(3, suppress=True)
lm.transitions
lm.p
lm.steady_state
ps.ergodic.fmpt(lm.p)
Explanation: LISA Markov
End of explanation
lm.transitions
lm.expected_t
lm.chi_2
Explanation: Test of independence of own chains and lag chains
End of explanation
<END_TASK>