<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 IPA to IMAGE model
Prerequisites
In principle, this should work on a computer without a GPU. It will help if you have a lot of RAM.
Install the latest versions of the following Python modules
Step1:  Load the model
(It will take a couple of minutes)
Step2:  Symbol embeddings
Step3:  The table of IPA symbols corresponding to  the 49 dimensions
Step4:  Let's display the embeddings projected to 2D via PCA
Step5:  Seems mostly random...
Let's try the same with a Language Model.
Symbol embeddings for a language model
Step6:  There is clear structure in the embeddings for the LM model.
Load MSCOCO validation data
Step7:  Project sentences to state space
Step8:  Find similar sentences in state space
Compute the pairwise cosine distance matrix.
Step9:  Display neighbors for a sentence
Step10:  State space for lower layers
We can also access the states in the whole set of layers using the pile function. We'll do it in batches to avoid memory problems on the GPU.
Step11:  What is encoded in different layers
The prediction is that the lowest layer will mostly encode phonetic similarity, while the top layer will encode mostly semantic similarity. Let's see if this is the case.
Step12:  Tracing the evolution of states
Step13:  Use espeak to convert graphemes to phonemes
Step14:  Plot traces of example sentences
<ASSISTANT_TASK:>
Python Code:
import imaginet.defn.visual as visual
import imaginet.task
Explanation: IPA to IMAGE model
Prerequisites
In principle, this should work on a computer without a GPU. It will help if you have a lot of RAM.
Install the latest versions of the following Python modules:
https://github.com/gchrupala/funktional
https://github.com/gchrupala/reimaginet
Download http://grzegorz.chrupala.me/data/coco.zip and unzip it in the reimaginet/data/coco directory.
  You should have the following files:
dataset.json - MSCOCO sentences
vgg_feats.mat - MSCOCO image vectors
dataset.ipa.jsonl.gz  - IPA transcriptions of MSCOCO sentences
(Copy pretrained models from yellow.uvt.nl:/home/gchrupala/repos/reimaginet/examples/vis and yellow.uvt.nl:/home/gchrupala/repos/reimaginet/examples/vis/lm)
End of explanation
model = imaginet.task.load(path="vis/model.10.zip")
Explanation: Load the model
(It will take a couple of minutes)
End of explanation
reload(visual)
emb = visual.embeddings(model)
print(emb.shape)
Explanation: Symbol embeddings
End of explanation
symb = visual.symbols(model)
print " ".join(symb.values())
Explanation: The table of IPA symbols corresponding to  the 49 dimensions
End of explanation
%pylab inline
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
xy = pca.fit_transform(emb)
pylab.rc('font', family='DejaVu Sans')
pylab.figure(figsize=(8,8))
pylab.scatter(xy[:,0], xy[:,1], alpha=0.1)
for j,symb_j in symb.items():
    if symb_j not in ["<BEG>","<END>", "<UNK>"]:
        pylab.text(xy[j,0], xy[j,1], symb_j)
Explanation: Let's display the embeddings projected to 2D via PCA
End of explanation
import imaginet.defn.lm
reload(imaginet.defn.lm)
model2 = imaginet.task.load("lm/model.10.zip")
emb2 = imaginet.defn.lm.embeddings(model2)
pca2 = PCA(n_components=2)
xy = pca.fit_transform(emb2)
pylab.rc('font', family='DejaVu Sans')
pylab.figure(figsize=(8,8))
pylab.scatter(xy[:,0], xy[:,1], alpha=0.1)
for j,symb_j in symb.items(): 
    if symb_j not in ["<BEG>","<END>", "<UNK>"]:
        pylab.text(xy[j,0], xy[j,1], symb_j)
Explanation: Seems mostly random...
Let's try the same with a Language Model.
Symbol embeddings for a language model
End of explanation
from imaginet.data_provider import getDataProvider
# Adjust the root to point to the directory above data
prov = getDataProvider('coco', root="..")
sents = list(prov.iterSentences(split='val'))
from imaginet.simple_data import phonemes
sents_ipa = [ phonemes(sent) for sent in sents ]
Explanation: There is clear structure in the embeddings for the LM model.
Load MSCOCO validation data
End of explanation
reps = imaginet.task.representation(model, sents_ipa)
Explanation: Project sentences to state space
End of explanation
from scipy.spatial.distance import cdist
distance = cdist(reps, reps, metric='cosine')
Explanation: Find similar sentences in state space
Compute the pairwise cosine distance matrix.
End of explanation
import numpy
def neighbors(k, distance=distance, size=5):
    nn =  numpy.argsort(distance[k,:])[1:size]
    print sents[k]['raw'], ''.join(sents_ipa[k])
    for n in nn:
        print u"✔" if sents[n]['imgid']==sents[k]['imgid'] else u"✘", \
        sents[n]['raw'], ''.join(sents_ipa[n])
import random
random.seed(41)
for _ in range(10):
    neighbors(random.randint(0, len(sents)), distance=distance)
    print
Explanation: Display neighbors for a sentence
End of explanation
reload(visual)
reps2 = []
for i in range(0,len(sents_ipa),512):
    r = [ ri[-1] for ri in imaginet.task.pile(model, sents_ipa[i:i+512], batch_size=256) ]
    reps2.extend(r)
reps2 = numpy.array(reps2)
distance1 = cdist(reps2[:,1,:], reps2[:,1,:], metric='cosine')
distance0 = cdist(reps2[:,0,:], reps2[:,0,:], metric='cosine')
%reset_selective reps2 # Free memory
Explanation: State space for lower layers
We can also access the states in the whole set of layers using the pile function. We'll do it in batches to avoid memory problems on the GPU.
End of explanation
random.seed(41)
for _ in range(10):
    i = random.randint(0, len(sents))
    print "Layer 1"
    neighbors(i, distance=distance0, size=2)
    print "Layer 2"
    neighbors(i, distance=distance1, size=2)
    print "Layer 3"
    neighbors(i, distance=distance, size=2)
    print
Explanation: What is encoded in different layers
The prediction is that the lowest layer will mostly encode phonetic similarity, while the top layer will encode mostly semantic similarity. Let's see if this is the case.
End of explanation
import imaginet.tracer
tr = imaginet.tracer.Tracer()
tr.fit(reps)
tr.proj.explained_variance_
Explanation: Tracing the evolution of states
End of explanation
from subprocess import check_output
def espeak(words):
    return phon(check_output(["espeak", "-q", "--ipa=3",
                        '-v', 'en',
                        words]).decode('utf-8'))
def phon(inp):
    return [ ph.replace(u"ˈ","") for word in inp.split() for ph in word.split("_") ]
%pylab inline --no-import-all
def trace(orths, tracer=tr, model=model, eos=True, size=(6,6)):
    ipas = [ espeak(orth) for orth in orths ]
    states = imaginet.task.states(model, ipas)
    pylab.figure(figsize=size)
    tracer.traces(ipas, states, eos=eos)
Explanation: Use espeak to convert graphemes to phonemes
End of explanation
trace(["A bowl of salad","A plate of pizza","A brown dog", "A black cat"])
trace(["a cow", "a baby cow","a tiny baby cow"])
orths = ["A cow","A baby","A tiny"]
ipas = [ espeak(orth) for orth in orths ]
states = imaginet.task.states(model, ipas)
states[0][0] - states[1][0]
trace(["some food on a table","a computer on a table","a table with food"])
pylab.axis('off')
trace(["a bear in a cage", "a brown bear in the zoo","a teddy bear on a chair"])
Explanation: Plot traces of example sentences
End of explanation
<END_TASK>