<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Evaluating Pre-trained Word Embeddings - Extended results
This notebook contains the extended results on word embeddings evaluation.
Step1:  Similarity task
We can see that the performance varies between the different embeddings on the different datasests.
Please see the API page for more information about the respective datasets.
Step2:  Analogy task
For the analogy task, we report the results per category in the dataset.
Note that the analogy task is a open vocabulary task
Step3:  We then present the results using the threecosadd analogy function.
Step4:  Bigger Analogy Test Set
We then display the results on the Bigger Analogy Test Set (BATS).
Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection
  of morphological and semantic relations with word embeddings
Step5:  We then present the results for the threecosmul analogy function.
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function
import pandas as pd
pd.options.display.max_rows = 999
pd.set_option('display.width', 1000)
import glob
header = ["evaluation_type", "dataset", "kwargs", "evaluation", "value", "num_skipped"]
similarity_dfs = []
similarity_names = []
similarity_glob = './results/similarity*'
for similarity_file in glob.glob(similarity_glob):
    df = pd.read_table(similarity_file, header=None, names=header).set_index(["dataset", "kwargs"]).drop(["evaluation_type"], axis=1)
    similarity_dfs.append(df)
    similarity_names.append(similarity_file[len(similarity_glob):])
analogy_dfs = []
analogy_names = []
analogy_glob = './results/analogy*'
for analogy_file in glob.glob(analogy_glob):
    df = pd.read_table(analogy_file, header=None, names=header).set_index(["dataset", "kwargs", "evaluation"]).drop(["evaluation_type"], axis=1)
    analogy_dfs.append(df)
    analogy_names.append(analogy_file[len(analogy_glob):])
similarity_df = pd.concat(similarity_dfs, keys=similarity_names, names=['embedding']).reorder_levels(["dataset", "kwargs", "embedding"]).sort_index()
analogy_df = pd.concat(analogy_dfs, keys=analogy_names, names=['embedding']).reorder_levels(["dataset", "evaluation", "kwargs", "embedding"]).sort_index()
Explanation: Evaluating Pre-trained Word Embeddings - Extended results
This notebook contains the extended results on word embeddings evaluation.
End of explanation
for (dataset, kwargs), df in similarity_df.groupby(level=[0,1]):
    print('Performance on', dataset, kwargs)
    print(df.loc[dataset, kwargs].sort_values(by='value', ascending=False))
    print()
    print()
Explanation: Similarity task
We can see that the performance varies between the different embeddings on the different datasests.
Please see the API page for more information about the respective datasets.
End of explanation
for kwargs, df in analogy_df.loc['GoogleAnalogyTestSet', 'threecosmul'].groupby(level=0):
    print(kwargs)
    print(df.loc[kwargs].sort_values(by='value', ascending=False))
    print()
    print()
Explanation: Analogy task
For the analogy task, we report the results per category in the dataset.
Note that the analogy task is a open vocabulary task: Given a query of 3 words, we ask the model to select a 4th word from the whole vocabulary. Different pre-trained embeddings have vocabularies of different size. In general the vocabulary of embeddings pretrained on more tokens (indicated by a bigger number before the B in the embedding source name) include more tokens in their vocabulary. While training embeddings on more tokens improves their quality, the larger vocabulary also makes the analogy task harder.
In this experiment all results are reported with reducing the vocabulary to the 300k most frequent tokens. Questions containing Out Of Vocabulary words are ignored.
Google Analogy Test Set
We first display the results on the Google Analogy Test Set.
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient
  estimation of word representations in vector space. In Proceedings of
  the International Conference on Learning Representations (ICLR).
The Google Analogy Test Set contains the following categories.
All analogy questions per category follow the pattern specified by the category name.
We first present the results using the threecosmul analogy function.
End of explanation
for kwargs, df in analogy_df.loc['GoogleAnalogyTestSet', 'threecosadd'].groupby(level=0):
    print(kwargs)
    print(df.loc[kwargs].sort_values(by='value', ascending=False))
    print()
    print()
Explanation: We then present the results using the threecosadd analogy function.
End of explanation
for kwargs, df in analogy_df.loc['BiggerAnalogyTestSet', 'threecosadd'].groupby(level=0):
    print(kwargs)
    print(df.loc[kwargs].sort_values(by='value', ascending=False))
    print()
    print()
Explanation: Bigger Analogy Test Set
We then display the results on the Bigger Analogy Test Set (BATS).
Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection
  of morphological and semantic relations with word embeddings: what works
  and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San
  Diego, California, June 12-17, 2016: ACL. Retrieved from
  https://www.aclweb.org/anthology/N/N16/N16-2002.pdf
Unlike the Google Analogy Test Set, BATS is balanced across 4 types of relations (inflectional morphology, derivational morphology, lexicographic semantics, encyclopedic semantics).
We first present the results for the threecosadd analogy function.
End of explanation
for kwargs, df in analogy_df.loc['BiggerAnalogyTestSet', 'threecosmul'].groupby(level=0):
    print(kwargs)
    print(df.loc[kwargs].sort_values(by='value', ascending=False))
    print()
    print()
Explanation: We then present the results for the threecosmul analogy function.
End of explanation
<END_TASK>