<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 QVEC
This notebook is a replication of Tsvetkov et al. (2015) Evaluation of Word Vector Representations by Subspace Alignment, which introduces QVEC. QVEC is an intrinsic evaluation method of word embeddings, measuring the correlation between dimensions of the embeddings and linguistic features. The original code is available, but I'm replicating it for two reasons
Step1:  QVEC model
QVEC finds an alignment between dimensions of learnt word embeddings and dimensions (features) of linguistic features by maximising the cumulative correlation.
$N$ is the size of the vocabulary (in common between the embeddings and the linguistic features).
$D$ is the dimensionality of the embeddings.
$X \in \mathbb{R}^{D \times N}$ is the matrix of embeddings. Note that a word's embedding is a column, rows are individual dimensions.
$P$ is the number of linguistic features.
$S \in \mathbb{R}^{P \times N}$ is the matrix of linguistic features, created above. Again, each word is a column and rows are individual features.
QVEC finds an alignment between the rows of $X$ and the rows of $S$ that maximises the correlation between the aligned rows. Each row of $X$ is aligned to at most one row of $S$, but each row of $S$ may be aligned to more than one row of $X$.
$A \in {0,1}^{D \times P}$ holds the alignments. $x_{ij}$ is 1 if dimension $i$ of $X$ is aligned with linguistic feature $j$.
The sum of correlations (which can be arbitrarily large with more dimensions or features) is their measure of the quality of the word embeddings in $X$.
$QVEC = \max_{A|\sum_{j}a_{ij} \leq 1}\sum_{i=1}^{D}\sum_{j=1}^{P}r(x_i, s_j) \times a_{ij}$
In words, for any possible alignment $A$, subject to the constraint that each embedding dimension is aligned to 0 or 1 linguistic features, sum up the correlations. The sum for the best alignment is the measure of the embeddings.
Crucially, this assumes that the dimensions of the embeddings end up encoding linguistic features. The authors justify this by the effectiveness of using word embeddings in linear models in downstream tasks.
First things first, transform my linguistic features into the format mentioned above (i.e., the matrix $S$).
Step2:  Learnt word embeddings
The original paper trains various different models of varying sizes. At a later stage I could do that, but for now I'm happy with using pre-trained embeddings.
Step3:  The Python variables S and X refer to $S$ and $X$ exactly as above.
Step4:  Now we want the correlation between the rows of S and the rows of X. This may not be the easiest way to do it but it works.
Step5:  For each row of this correlation matrix (i.e. for each of the dimenions of the embeddings), we want the linguistic feature that it is most correlated with. We also get the value of that correlation.
Step6:  The score of the embeddings relative to the linguistic features is the sum of the maximum correlations. Note how this value depends on how many dimensions in the embeddings there are. For 300 dimension vectors trained (by them) from GloVe, the authors get 34.4, while I get 32.4. Note that our linguistic features are still different, so the fact that the discrepancy here is not too big is encouraging.
Step7:  We don't really need it, but just to be explicit let's get the matrix $A$ of alignments.
Step8:  The rest of the paper is a series of experiments training large models and evaluating them on both instrinic and extrinsic tasks, including QVEC. I'm not going to replicate that here, but the QVEC implementation is complete.
Canonical Correlation Analysis
In a follow-up 2016 paper, a subset of the original authors introduce QVEC-CCA. It's really just QVEC except instead of summing the highest row-wise correlations, they use canonical correlation analysis. I didn't know what that was, but after reading a bit I have a reasonable grasp of it. I'm going to replicate that 2016 paper, or at least the most important part of it which is the use of CCA. Note that the other new thing in the 2016 paper is the use of syntactic features, in addition to semantic, which I won't do right now.
Scikit-learn has an implementation of CCA. It took me a while to figure out what are the learnt parameters that I want, and I'm only 80% confident I have it right.
Step9:  I believe the linear combinations I want are stored in the x_weights_ and y_weights_ attributes.
Step11:  Succint implementation
<ASSISTANT_TASK:>
Python Code:
import os
import csv
import pandas as pd
import numpy as np
from scipy import stats
data_path = '../../data'
tmp_path = '../../tmp'
feature_path = os.path.join(data_path, 'evaluation/semcor/tsvetkov_semcor.csv')
subset = pd.read_csv(feature_path, index_col=0)
subset.columns = [c.replace('semcor.', '') for c in subset.columns]
subset.head()
Explanation: QVEC
This notebook is a replication of Tsvetkov et al. (2015) Evaluation of Word Vector Representations by Subspace Alignment, which introduces QVEC. QVEC is an intrinsic evaluation method of word embeddings, measuring the correlation between dimensions of the embeddings and linguistic features. The original code is available, but I'm replicating it for two reasons: i) as a learning exercise and ii) the original implementation looks messy.
To implement QVEC, I'm going to need two things:
- Gold standard linguistic features
- The QVEC model
The linguistic features used in the original paper come from SemCor, a WordNet annotated subset of the Brown corpus. This is done in here.
End of explanation
subset.set_index('words', inplace=True)
#subset.drop('count_in_semcor', inplace=True, axis=1)
subset = subset.T
subset.head()
Explanation: QVEC model
QVEC finds an alignment between dimensions of learnt word embeddings and dimensions (features) of linguistic features by maximising the cumulative correlation.
$N$ is the size of the vocabulary (in common between the embeddings and the linguistic features).
$D$ is the dimensionality of the embeddings.
$X \in \mathbb{R}^{D \times N}$ is the matrix of embeddings. Note that a word's embedding is a column, rows are individual dimensions.
$P$ is the number of linguistic features.
$S \in \mathbb{R}^{P \times N}$ is the matrix of linguistic features, created above. Again, each word is a column and rows are individual features.
QVEC finds an alignment between the rows of $X$ and the rows of $S$ that maximises the correlation between the aligned rows. Each row of $X$ is aligned to at most one row of $S$, but each row of $S$ may be aligned to more than one row of $X$.
$A \in {0,1}^{D \times P}$ holds the alignments. $x_{ij}$ is 1 if dimension $i$ of $X$ is aligned with linguistic feature $j$.
The sum of correlations (which can be arbitrarily large with more dimensions or features) is their measure of the quality of the word embeddings in $X$.
$QVEC = \max_{A|\sum_{j}a_{ij} \leq 1}\sum_{i=1}^{D}\sum_{j=1}^{P}r(x_i, s_j) \times a_{ij}$
In words, for any possible alignment $A$, subject to the constraint that each embedding dimension is aligned to 0 or 1 linguistic features, sum up the correlations. The sum for the best alignment is the measure of the embeddings.
Crucially, this assumes that the dimensions of the embeddings end up encoding linguistic features. The authors justify this by the effectiveness of using word embeddings in linear models in downstream tasks.
First things first, transform my linguistic features into the format mentioned above (i.e., the matrix $S$).
End of explanation
size = 50
fname = 'embeddings/glove.6B.{}d.txt'.format(size)
embedding_path = os.path.join(data_path, fname)
embeddings = pd.read_csv(embedding_path, sep=' ', header=None, index_col=0, quoting=csv.QUOTE_NONE).T
embeddings.head()
common_words = embeddings.columns.intersection(subset.columns)
embeddings = embeddings[common_words]
fname = os.path.join(tmp_path, 'glove_embeddings.csv')
embeddings.to_csv(fname)
from sklearn.metrics.pairwise import cosine_similarity as cos
pairwise = cos(embeddings.T)
distances = pd.DataFrame(pairwise, columns=common_words, index=common_words)
distances.to_csv(os.path.join(data_path, 'pairwise_sim.csv'))
Explanation: Learnt word embeddings
The original paper trains various different models of varying sizes. At a later stage I could do that, but for now I'm happy with using pre-trained embeddings.
End of explanation
S = subset[common_words]
X = embeddings[common_words]
Explanation: The Python variables S and X refer to $S$ and $X$ exactly as above.
End of explanation
correlations = pd.DataFrame({i:X.corrwith(S.iloc[i], axis=1) for i in range(len(S))})
correlations.columns = S.index
Explanation: Now we want the correlation between the rows of S and the rows of X. This may not be the easiest way to do it but it works.
End of explanation
alignments = correlations.idxmax(axis=1)
correlations.max(axis=1).head()
Explanation: For each row of this correlation matrix (i.e. for each of the dimenions of the embeddings), we want the linguistic feature that it is most correlated with. We also get the value of that correlation.
End of explanation
qvec = correlations.max(axis=1).sum()
qvec
Explanation: The score of the embeddings relative to the linguistic features is the sum of the maximum correlations. Note how this value depends on how many dimensions in the embeddings there are. For 300 dimension vectors trained (by them) from GloVe, the authors get 34.4, while I get 32.4. Note that our linguistic features are still different, so the fact that the discrepancy here is not too big is encouraging.
End of explanation
A = pd.DataFrame(0, index=range(len(X)), columns=S.index)
for dim, feat in alignments.iteritems():
    A[feat][dim] = 1
A.head()
Explanation: We don't really need it, but just to be explicit let's get the matrix $A$ of alignments.
End of explanation
from sklearn.cross_decomposition import CCA
cca = CCA(n_components=1)
cca = cca.fit(X.T, S.T)
Explanation: The rest of the paper is a series of experiments training large models and evaluating them on both instrinic and extrinsic tasks, including QVEC. I'm not going to replicate that here, but the QVEC implementation is complete.
Canonical Correlation Analysis
In a follow-up 2016 paper, a subset of the original authors introduce QVEC-CCA. It's really just QVEC except instead of summing the highest row-wise correlations, they use canonical correlation analysis. I didn't know what that was, but after reading a bit I have a reasonable grasp of it. I'm going to replicate that 2016 paper, or at least the most important part of it which is the use of CCA. Note that the other new thing in the 2016 paper is the use of syntactic features, in addition to semantic, which I won't do right now.
Scikit-learn has an implementation of CCA. It took me a while to figure out what are the learnt parameters that I want, and I'm only 80% confident I have it right.
End of explanation
a = np.dot(X.T, cca.x_weights_)
b = np.dot(S.T, cca.y_weights_)
stats.pearsonr(a, b)
Explanation: I believe the linear combinations I want are stored in the x_weights_ and y_weights_ attributes.
End of explanation
def qvec(features, embeddings):
    
    Returns correlations between columns of `features` and `embeddings`.
    
    The aligned feature is the one with the highest correlation.
    The qvec score is the sum of correlations of aligned features.
    
    common_words = embeddings.columns.intersection(subset.columns)
    S = features[common_words]
    X = embeddings[common_words]
    correlations = pd.DataFrame({i:X.corrwith(S.iloc[i], axis=1) for i in range(len(S))})
    correlations.columns = S.index
    return correlations
qvec(subset, embeddings).head()
Explanation: Succint implementation
End of explanation
<END_TASK>