<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 LDA on vertebrates
Notes on the data
In this example the tree is contstrained
In this example we have to extract position,transition, and branch. 
The total position are broken into N 'splits'                      
Each split contains 30,000 files (except the last file that has less)
So that means that there are 30,000 * N total positions
Browsing the data
Step1:  a sentence of words is represented as the transitions for a given position
Step2:  Simple test run with lda package
Step3:  Recall that the Dirichlet Process (DP) (Ferguson, 1973) is essentially a distribution over distributions, where each draw from a DP is itself a distribution and importantly for clustering applications it serves as a natural prior that lets the number of clusters grow as the data grows.  The DP has a base distribution parameter $\beta$ and a strength or concentration parameter $\alpha$.   
$\alpha$ is a hyperprior for the DP over per-document topic distributions
$\beta$ is the hyperprior for the DP over per-topic word distributions
$\theta_{m}$ is the topic distribution for document $m$
$\phi_{k}$ is the word distribution for topic $k$
$z_{m,n}$ is the topic for the $n$th word in document $m$
$w_{m,n}$ is the specific word 
The generative story for phylogenetics
We are still modeling topics. However, documents become sites and words become transitions.  Transitions may defined in nucleotide, amino acid or codon space.  Perhaps more logically though a document would be all of the sites for a given gene.
$\alpha$ is a hyperprior for the DP over per-site topic distributions
$\beta$ is the hyperprior for the DP over per-topic transition distributions
$\theta_{m}$ is the topic distribution for gene $m$
$\phi_{k}$ is the nucleotide transition distribution for topic $k$
$z_{m,n}$ is the topic for the $n$th nucleotide transition in gene $m$
$w_{m,n}$ is the specific transition
The generative process
Choose $\theta_{m} \sim \textrm{Dir}(\alpha)$, where $m \in {1,...M}$ and $\textrm{Dir}(\alpha)$ is the Dirichlet distribtion for $\alpha$
Choose $\phi_{k} \sim \textrm{Dir}(\beta)$, where $k \in {1,...K}$
For each of the transition positions ($m$,$n$), where $n \in {1,...N}$, and $m \in {1,...M}$  
Choose a topic $z_{m,n} \sim \textrm{Multinomial}(\theta_{m})$
Choose a transition $w_{m,n} \sim \textrm{Multinomial}(\phi_{m,n})$
$\phi$ is a $K*V$ Markov matrix each row of which denotes the transition distribution of a topic.
The type of data to expect
Here I am borrowing from the package LDA, which uses a collapsed version of Gibbs sampling.
In this example.
Positions are documents
First 1000 positions
We consider in place transitions
20 topics
1500 MCMC iterations
7 words in a topic (transitions)
topics
Topic 0
<ASSISTANT_TASK:>
Python Code:
import os
import numpy as np
from vertebratesLib import *
split = "SPLIT1"
summaryTree,summarySpecies,splitPositions = get_split_data(split)
print summaryTree.shape
Explanation: LDA on vertebrates
Notes on the data
In this example the tree is contstrained
In this example we have to extract position,transition, and branch. 
The total position are broken into N 'splits'                      
Each split contains 30,000 files (except the last file that has less)
So that means that there are 30,000 * N total positions
Browsing the data
End of explanation
def get_sentence(position,splitPositions,summary,ignore=False):
    splitIndex = np.where(splitPositions==position)[0]
    nonZero = np.where(summary[splitIndex,:] != 0)[1]
    sentence = []
    for nz in nonZero:
        if ignore and TRANSITIONS[nz].count(TRANSITIONS[nz][0]) == 2:
            continue
        
        count = int(summary[splitIndex,nz][0])
        sentence.extend([TRANSITIONS[nz]] * count)
    
    return sentence
        
position = '8500'
sentence1 = get_sentence(position,splitPositions,summaryTree,ignore=False)
sentence2 = get_sentence(position,splitPositions,summaryTree,ignore=True)
print("with same AA transition")
print(sentence1)
print("without same AA transition")
print(sentence2)
Explanation: a sentence of words is represented as the transitions for a given position
End of explanation
import lda
## the data matrix are the sentences by vocabulary
vocab = TRANSITIONS
#inPlaceTransitions = []
#for t in TRANSITIONS:
Explanation: Simple test run with lda package
End of explanation
from IPython.display import Image
dataDir = None
for ddir in [os.path.join("..","data","herve-vertebrates"),\
             os.path.join("/","media","ganda","mojo","phylogenetic-models","herve-vertebrates")]:
    if os.path.isdir(ddir):
        dataDir = ddir
split = "SPLIT1"
position = "0"
treeList = get_trees(split,position,dataDir)
countMatrix = np.zeros((len(treeList),len(TRANSITIONS)),)
t = 0
for t,pbTree in enumerate(treeList):
    fixedTree,treeSummary = fix_tree(pbTree)
    tlist = []
    for item in treeSummary.itervalues():
        tlist.extend(item['pairs'])
    counts = transitions_to_counts(tlist)
    countMatrix[t,:] = counts
figName1 = os.path.join("figures","lda-bplot-check.png")
profile_box_plot(countMatrix,figName1,figTitle='position - %s'%position)
Image(filename=figName1)
Explanation: Recall that the Dirichlet Process (DP) (Ferguson, 1973) is essentially a distribution over distributions, where each draw from a DP is itself a distribution and importantly for clustering applications it serves as a natural prior that lets the number of clusters grow as the data grows.  The DP has a base distribution parameter $\beta$ and a strength or concentration parameter $\alpha$.   
$\alpha$ is a hyperprior for the DP over per-document topic distributions
$\beta$ is the hyperprior for the DP over per-topic word distributions
$\theta_{m}$ is the topic distribution for document $m$
$\phi_{k}$ is the word distribution for topic $k$
$z_{m,n}$ is the topic for the $n$th word in document $m$
$w_{m,n}$ is the specific word 
The generative story for phylogenetics
We are still modeling topics. However, documents become sites and words become transitions.  Transitions may defined in nucleotide, amino acid or codon space.  Perhaps more logically though a document would be all of the sites for a given gene.
$\alpha$ is a hyperprior for the DP over per-site topic distributions
$\beta$ is the hyperprior for the DP over per-topic transition distributions
$\theta_{m}$ is the topic distribution for gene $m$
$\phi_{k}$ is the nucleotide transition distribution for topic $k$
$z_{m,n}$ is the topic for the $n$th nucleotide transition in gene $m$
$w_{m,n}$ is the specific transition
The generative process
Choose $\theta_{m} \sim \textrm{Dir}(\alpha)$, where $m \in {1,...M}$ and $\textrm{Dir}(\alpha)$ is the Dirichlet distribtion for $\alpha$
Choose $\phi_{k} \sim \textrm{Dir}(\beta)$, where $k \in {1,...K}$
For each of the transition positions ($m$,$n$), where $n \in {1,...N}$, and $m \in {1,...M}$  
Choose a topic $z_{m,n} \sim \textrm{Multinomial}(\theta_{m})$
Choose a transition $w_{m,n} \sim \textrm{Multinomial}(\phi_{m,n})$
$\phi$ is a $K*V$ Markov matrix each row of which denotes the transition distribution of a topic.
The type of data to expect
Here I am borrowing from the package LDA, which uses a collapsed version of Gibbs sampling.
In this example.
Positions are documents
First 1000 positions
We consider in place transitions
20 topics
1500 MCMC iterations
7 words in a topic (transitions)
topics
Topic 0: EE ED EQ EK EG DE EA
Topic 1: YY YF FY FF YH YC YS
Topic 2: RR KK RK QQ RQ HH KR
Topic 3: AA AS AT AV SS VA SA
Topic 4: II MM IV IL IM MI ML
Topic 5: SS ST SA SN SP TT SG
Topic 6: WW WL YW WS SW WV WG
Topic 7: KK KR RR KQ RK KE KN
Topic 8: HH HY HQ HR HN QH YH
Topic 9: CC CS SC CY SS CF LC
Topic 10: VV VI IV VA VL II VM
Topic 11: TT TA TS TV TI ST TM
Topic 12: DD DE DN ED EE DG ND
Topic 13: QQ QK QE QH QR QL QP
Topic 14: FF FL LF FY FI FV FC
Topic 15: PP PS SP PA PL PT PQ
Topic 16: NN NS SS ND NK SN NT
Topic 17: LL LI LV LM LF MM LQ
Topic 18: RR RK RQ RT RW RY RV
Topic 19: GG GS GA GE GN SG GD
top topics
position - 0 (top topic: 3)
position - 1 (top topic: 19)
position - 10 (top topic: 3)
position - 100 (top topic: 18)
position - 1000 (top topic: 7)
position - 10000 (top topic: 7)
position - 10001 (top topic: 7)
position - 10002 (top topic: 19)
position - 10003 (top topic: 7)
position - 10004 (top topic: 5)
End of explanation
<END_TASK>