<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Information Retrieval
This is a tutorial for the npfl103 package for Information Retrieval assignments.
Big picture
In simple IR systems that we'll build in this lab session, two major things are happening more or less independently on each other. One
Step1:  Tutorial data
The data for the tutorial lives in the tutorial_assignment subfolder. It mirrors the assignment folder in structure and file types.
Step2:  Loading documents and queries
For loading data, use the npfl103.io module. From the top down, there is a class for representing the entire collection of documents (Collection), a Document class for representing one document and an equivalent Topic class for representing one query topic, a VText class for representing a block of text in one zone of a document/query, and a VToken class for representing one word (or equivalent token) in a zone text.
These classes are nothing interesting
Step3:  Notice that creating the Collection was fast. This is because the whole pipeline in npfl103 is lazy
Step4:  Caching
The constant memory footprint is not exactly true
Step5:  The purpose of a Vectorizer is to take a stream of a document's tokens and convert it into one vector representing this document in some vector space. Each token is used as a dimension of the vector space. If your tokens are just part of speech tags (noun, verb, etc.), then your space will have just some 10 dimensions; if your tokens are word forms, then there will be thousands of dimensions.
(The vectors will be sparse vectors, only remembering the nonzero elements -- implemented simply as a dict.) 
When we initialize a Vectorizer, we need to specify two things
Step6:  The vectorizer provides a transform method that does the processing.
Step7:  (Notice that we're still using generator expressions, so nothing really gets computed so far.)
After running through a Vectorizer, there is no implementation difference between what a Document and a Topic look like.
Which tokens?
Not all tokens are relevant. For instance, you might only want to represent a document using its title and headings, not the texts themselves. Or you might want to only use tokens which are "content words" -- usually defined as nouns, adjectives, verbs, or adverbs. The Vectorizers accept some arguments to specify which tokens should be used
Step8:  We can compare the results of the "plain" vectorizer and the content word vectorizer
Step9:  We can see that token filtering can make a pretty large difference.
Transforming vectors
While the Vectorizers got us from the raw data to a vector space, we might not be particularly happy with the immediate results. For instance, in the above example, we see very general words like "new" or "be", and we might wish to apply the Inverse Document Frequency transform. Or we want to normalize the frequencies to sum to 1, or use pivot normalization, or... whatever.
To do operations on vector spaces, we use TransformCorpus objects as "pipeline sections" that operate on the flow of data.
Step10:  These "pipeline" components get two parameters
Step11:  The corpus is an iterable that contains dictionary-like objects as sparse document vectors.
The transform parameter is a callable
Step12:  Vectorization as transformation
It is maybe more elegant to implement vectorization also as a pipeline component instead of having lists or generators floating around.
Step13:  Chaining transformations
The pipelines can, of course, build on top of each other. Using the previous pipeline stages cw_docs and cw_queries objects of TransformCorpus class as the corpus parameter, we can put the normalization on top of these
Step14:  How would you implement TF-IDF in this system?
Vectorize with TermFrequencyVectorizer
Implement IDF as a class with a __call__ method that can be used as a transformation. (Hint
Step15:  Recapitulation
At this point, the retrieval pipeline is set up. We have
Step16:  Evaluation
You should already have compiled trec_eval using the instructions in the README in npfl103/eval.
The npfl103.evaluation package provides a do_eval() and print_eval() function to run evaluation
from within the package.
Step17:  Since trec_eval (which is called inside these functions) needs an input file, not a stream,
we have to dump our results to a file.
Step18:  The tutorial assignment has its ground truth file
Step19:  You can also break down the results by query, by setting results_by_query=True
Step20:  If you want to do further processing with the results, use do_eval().
Instead of printing results, it returns them as a dictionary. Again, you can
request the results by query (it will come in an OrderedDict of OrderedDicts,
see do_eval() docstring).
<ASSISTANT_TASK:>
Python Code:
import npfl103
Explanation: Information Retrieval
This is a tutorial for the npfl103 package for Information Retrieval assignments.
Big picture
In simple IR systems that we'll build in this lab session, two major things are happening more or less independently on each other. One: the similarity index of documents to retrieve has to be built. Two: the queries are processed and documents get returned.
The first part is all about representing the documents in your collection as points in a vector space.
In the second part, you then convert the queries into the same vector space, and return the documents according to how close to the query point they are represented.
Your job will be mostly to deal with designing a clever vector space, so that the closest documents to a query happen to be the right ones to retrieve.
There are some pesky technicalities that have to be taken care of: reading the documents, writing the outputs, evaluating, etc. This package does its best to help you not to have to deal with these parts, but you kind of have to be aware of them, so the tutorial does go through how they are handled.
The plan
The tutorial goes through the following steps:
Loading documents and queries (topics)
Processing documents and queries into a vector space (!!!)
Transforming the document vectors from one space to another (!!!)
Making queries
Writing the outputs
Evaluating
Points 2 and 3 are where you're supposed to modify things and come up with ideas. The linguistic stuff (lemmatization/stemming, part-of-speech filtering, etc.) comes in step 2, the math (TF-IDF, pivot normalization, topic models...) come up in step 3.
Note that the Python classes you're supposed to use have documentation inside, with quite detailed examples.
We don't cover all of that here -- the tutorial focuses on how the library fits together.
End of explanation
import os
dpath = os.path.join('.', 'tutorial-assignment')
dlist = os.path.join(dpath, 'documents.list')
qlist = os.path.join(dpath, 'topics.list')
Explanation: Tutorial data
The data for the tutorial lives in the tutorial_assignment subfolder. It mirrors the assignment folder in structure and file types.
End of explanation
from npfl103.io import Collection
coll_docs = Collection(dlist)
Explanation: Loading documents and queries
For loading data, use the npfl103.io module. From the top down, there is a class for representing the entire collection of documents (Collection), a Document class for representing one document and an equivalent Topic class for representing one query topic, a VText class for representing a block of text in one zone of a document/query, and a VToken class for representing one word (or equivalent token) in a zone text.
These classes are nothing interesting: they merely represent the data (both the queries and the documents that should be retrieved). Check out their documentation strings for details on how to operate them.
The main entry point to data loading is the Collection class. Initialize it with the documents.list (or topics.list) file:
End of explanation
from npfl103.io import Topic
coll_queries = Collection(qlist, document_cls=Topic)
Explanation: Notice that creating the Collection was fast. This is because the whole pipeline in npfl103 is lazy: no document is read until it is actually requested. This helps keep time and especially memory requirements down; the library is designed to have a constant memory footprint.
Collection item classes
A Collection consists of individual documents. There are two implemented document types: the Document class from npfl103.io, and the Topic class. Collections are by default created as collections of Documents; however, for reading the queries, we use the same Collection mechanism and explicitly supply the Topic class.
End of explanation
from npfl103.io import BinaryVectorizer, TermFrequencyVectorizer
Explanation: Caching
The constant memory footprint is not exactly true: for speeding up repeated document reads, the Collection caches the documents it read. Eventually, you may run out of memory. In that case, try creating a Collection with nocache=True.
Processing documents and queries into a vector space
To run a vector space information retrieval experiment, we now have to convert the loaded representations of documents and queries into a vector space. For this, we provide the Vectorizer classes.
End of explanation
vectorizer = TermFrequencyVectorizer(field='lemma')
Explanation: The purpose of a Vectorizer is to take a stream of a document's tokens and convert it into one vector representing this document in some vector space. Each token is used as a dimension of the vector space. If your tokens are just part of speech tags (noun, verb, etc.), then your space will have just some 10 dimensions; if your tokens are word forms, then there will be thousands of dimensions.
(The vectors will be sparse vectors, only remembering the nonzero elements -- implemented simply as a dict.) 
When we initialize a Vectorizer, we need to specify two things:
What the stream of tokens should contain (what the dimensions of the space will be),
How the values of the vector items will be computed (binary? frequencies? etc.)
Let's build a term frequency vectorizer that iterates over lemmas.
End of explanation
vdocs = (vectorizer.transform(d) for d in coll_docs)
Explanation: The vectorizer provides a transform method that does the processing.
End of explanation
cw_vectorizer = TermFrequencyVectorizer(field='lemma', token_filter=lambda t: t.pos in 'NADV')
Explanation: (Notice that we're still using generator expressions, so nothing really gets computed so far.)
After running through a Vectorizer, there is no implementation difference between what a Document and a Topic look like.
Which tokens?
Not all tokens are relevant. For instance, you might only want to represent a document using its title and headings, not the texts themselves. Or you might want to only use tokens which are "content words" -- usually defined as nouns, adjectives, verbs, or adverbs. The Vectorizers accept some arguments to specify which tokens should be used:
Zones (TITLE, HEADING, ...)
Field (form, lemma, pos, full_pos, ... - see format specification in the assignment README)
Start and end token (e.g. you might only want to read the beginnings of documents)
Token filter
We've already seen an example using field. The zones for documents are TITLE, HEADING, and TEXT. The zones for queries (topics) are title, narr, and desc. Take a moment to look into the data, to make sure you know what the roles of the zones are.
Except for zones, make sure you use the same vectorization settings for both the Documents and the Topics!
Token filter
The token filter argument is a function that returns True or False when called with a VToken object. This enables filtering out tokens based on a different field than the one used to build the vector space. For instance, the aforementioned content word filtering would be iplemented as:
End of explanation
vdocs = [vectorizer.transform(d) for d in coll_docs]   # This actually parses the documents.
cw_docs = [cw_vectorizer.transform(d) for d in coll_docs]
d = vdocs[0]
cw_d = cw_docs[0]
# Print the top 10 most frequent tokens
import pprint, operator
print('All words:')
pprint.pprint({w: n for w, n in sorted(d.items(), key=operator.itemgetter(1), reverse=True)[:10]})
print('----------------------\nContent words:')
pprint.pprint({w: n for w, n in sorted(cw_d.items(), key=operator.itemgetter(1), reverse=True)[:10]})
Explanation: We can compare the results of the "plain" vectorizer and the content word vectorizer:
End of explanation
from npfl103.transform import TransformCorpus
Explanation: We can see that token filtering can make a pretty large difference.
Transforming vectors
While the Vectorizers got us from the raw data to a vector space, we might not be particularly happy with the immediate results. For instance, in the above example, we see very general words like "new" or "be", and we might wish to apply the Inverse Document Frequency transform. Or we want to normalize the frequencies to sum to 1, or use pivot normalization, or... whatever.
To do operations on vector spaces, we use TransformCorpus objects as "pipeline sections" that operate on the flow of data.
End of explanation
# This is the transformation we want to apply.
def normalize(vec):
    doc_length = sum(vec.values())
    return {k: v / doc_length for k, v in vec.items()}
normalized_docs = TransformCorpus(corpus=cw_docs, transform=normalize, name='normalized_docs')
Explanation: These "pipeline" components get two parameters: the transformation they should be applying, and the source of the data to apply it on. Let's make an example transformation: normalizing the frequencies to 1.
End of explanation
cw_queries = (cw_vectorizer.transform(q) for q in coll_queries)   # Generator, again
normalized_queries = TransformCorpus(corpus=cw_queries, transform=normalize, name='normalized_queries')
Explanation: The corpus is an iterable that contains dictionary-like objects as sparse document vectors.
The transform parameter is a callable: either a function, or a class that implements a __call__ method.
Notice also the name parameter: this is for yourself, to be able to keep track of what each pipeline component does.
Let's do the same thing for queries:
End of explanation
cw_docs = TransformCorpus(corpus=coll_docs, transform=cw_vectorizer.transform, name='vectorized_docs')
cw_queries = TransformCorpus(corpus=coll_queries, transform=cw_vectorizer.transform, name='vectorized_queries')
Explanation: Vectorization as transformation
It is maybe more elegant to implement vectorization also as a pipeline component instead of having lists or generators floating around.
End of explanation
normalized_docs = TransformCorpus(corpus=cw_docs, transform=normalize, name='normalized_docs')
normalized_queries = TransformCorpus(corpus=cw_queries, transform=normalize, name='normalized_docs')
Explanation: Chaining transformations
The pipelines can, of course, build on top of each other. Using the previous pipeline stages cw_docs and cw_queries objects of TransformCorpus class as the corpus parameter, we can put the normalization on top of these:
End of explanation
from npfl103.similarity import Similarity
# The similarity is initialized with the document corpus.
similarity = Similarity(corpus=normalized_docs, k=10)       # Returning the top 10 documents. Use None for all docs.
similarity_corpus = TransformCorpus(corpus=normalized_queries, transform=similarity, name='sim')
Explanation: How would you implement TF-IDF in this system?
Vectorize with TermFrequencyVectorizer
Implement IDF as a class with a __call__ method that can be used as a transformation. (Hint: it needs to see the training corpus of documents at initialization time, to initialize the inverse document frequencies for the individual terms.)
Add a TransformCorpus that gets this IDF transformer as a transform method on top of the vectorized corpus (that was also used as input for the IDF transformer's initialization).
Similarity queries
Assuming we're happy with the vector space in which our documents now live, we want to find for a query the similarity scores for all documents.
The same transformation mechanism is used. This time, we transform a query from the same space as the documents into a similarity space: the dimensions of this space are the documents which should be retrieved, and the values are the similarity scores for the query and that given document.
End of explanation
import io   # The system io, not npfl103.io
hdl = io.StringIO()  # Technical workaround, so that the tutorial does not create files at this point.
# This is what writes the output. In practice, you'll probably use "with open(...) as hdl:" to write to a file.
Similarity.write_trec(similarity_corpus, similarity, hdl)
Explanation: Recapitulation
At this point, the retrieval pipeline is set up. We have:
Vectorized and processed the documents which we want to retrieve,
We can vectorize and process an incoming query in the same way,
We can use the query to compute similarities and return the top K documents.
Now, we only have to worry about recording our retrieval results and evaluating them against human judgments of relevant vs. non-relevant documents.
Writing the outputs
In order to record the outputs, use the Similarity.write_trec static method:
End of explanation
from npfl103.evaluation import do_eval, print_eval
Explanation: Evaluation
You should already have compiled trec_eval using the instructions in the README in npfl103/eval.
The npfl103.evaluation package provides a do_eval() and print_eval() function to run evaluation
from within the package.
End of explanation
results_file = 'tutorial-assignment/tutorial-output.dat'
with open(results_file, 'w') as outstream:
    Similarity.write_trec(similarity_corpus, similarity, outstream)
Explanation: Since trec_eval (which is called inside these functions) needs an input file, not a stream,
we have to dump our results to a file.
End of explanation
qrels_file = 'tutorial-assignment/qrels.txt'
print_eval(qrels_file, results_file)
Explanation: The tutorial assignment has its ground truth file:
End of explanation
print_eval(qrels_file, results_file, results_by_query=True)
Explanation: You can also break down the results by query, by setting results_by_query=True:
End of explanation
results = do_eval(qrels_file, results_file, results_by_query=True)
pprint.pprint([q for q in results])
pprint.pprint(results['10.2452/401-AH'])
Explanation: If you want to do further processing with the results, use do_eval().
Instead of printing results, it returns them as a dictionary. Again, you can
request the results by query (it will come in an OrderedDict of OrderedDicts,
see do_eval() docstring).
End of explanation
<END_TASK>