<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Mini-Assignment 1
Step1:  To make it easier to access the data, we convert here paper entries into named tuples. This will allow us to refer to fields by keyword (like var.year), rather than index (like var[2]).
Step2:  Dataset Statistics
Plotting relies on matplotlib and NumPy. If your installation doesn't have them included already, you can download them here and here, respectively.
Step3:  Papers per Year
Here, we will get information on how many papers in the dataset were published per year.
We'll be using the Counter class to determine the number of papers per year.
Step4:  Filtering results, to obain only papers since 1940
Step5:  Creating a bar plot to visualize the results (using matplotlib.pyplot.bar)
Step6:  Alternatively, you can get the same result in a more direct manner by plotting it as a histogram with matplotlib.pyplot.hist
Step7:  Papers per Author
Here, we will obtain the distribution characterizing the number of papers published by an author.
Step8:  Creating a histogram to visualize the results
Step9:  Authors per Paper
Step10:  Words in Titles
assemble list of words in paper titles, convert them to lowercase, and remove trailing '.'
Step11:  Assignments
Your name
Step12:  Task 2
Based on the dictionary authors_at_year from exercise 1 above, create a plot for the years from 1940 until 2016 that shows how many authors published at least one paper. (You can retrieve the number of unique items in a set s with len(s).)
Step13:  Task 3
Calculate and plot (e.g. using plt.plot) a graph of the frequency of the 100 most frequent words in titles of papers, from most frequent to least frequent. (You can make use of the data structures created above.)
Step14:  Task 4
Print out the top 50 most often occurring words in the paper's titles. (You can again make use of the data structures created above.)
<ASSISTANT_TASK:>
Python Code:
import pickle, bz2
Summaries_file = 'data/malaria__Summaries.pkl.bz2'
Summaries = pickle.load( bz2.BZ2File( Summaries_file, 'rb' ) )
Explanation: Mini-Assignment 1: Inspecting the PubMed Paper Dataset
In this code for the first mini-assignment, we will get to know the dataset that we will be using throughout. You can find the assignment tasks at the very bottom of this document.
Our dataset consists of short texts (article abstracts) from the PubMed database of scientific publications in the Life Science domain. As the full dataset consists of millions of documents, we are using just a small subset, namely all publications that contain the word "malaria" in their title or abstract. You can download that dataset in the form of four files (malaria__Summaries.pkl.bz2, etc.) from Blackboard. Save these four files in a directory called data, which should be a sub-directory of the one that contains this notebook file (or adjust the file path in the code)
Loading the Dataset
End of explanation
from collections import namedtuple
paper = namedtuple( 'paper', ['title', 'authors', 'year', 'doi'] )
for (id, paper_info) in Summaries.items():
    Summaries[id] = paper( *paper_info )
Summaries[24130474]
Summaries[24130474].title
Explanation: To make it easier to access the data, we convert here paper entries into named tuples. This will allow us to refer to fields by keyword (like var.year), rather than index (like var[2]).
End of explanation
import matplotlib.pyplot as plt
# show plots inline within the notebook
%matplotlib inline
# set plots' resolution
plt.rcParams['savefig.dpi'] = 100
Explanation: Dataset Statistics
Plotting relies on matplotlib and NumPy. If your installation doesn't have them included already, you can download them here and here, respectively.
End of explanation
from collections import Counter
paper_years = [ p.year for p in Summaries.values() ]
papers_per_year = sorted( Counter(paper_years).items() )
print('Number of papers in the dataset per year for the past decade:')
print(papers_per_year[-10:])
Explanation: Papers per Year
Here, we will get information on how many papers in the dataset were published per year.
We'll be using the Counter class to determine the number of papers per year.
End of explanation
papers_per_year_since_1940 = [ (y,count) for (y,count) in papers_per_year if y >= 1940 ]
years_since_1940 = [ y for (y,count) in papers_per_year_since_1940 ]
nr_papers_since_1940 = [ count for (y,count) in papers_per_year_since_1940 ]
print('Number of papers in the dataset published since 1940:')
print(sum(nr_papers_since_1940))
Explanation: Filtering results, to obain only papers since 1940:
End of explanation
plt.bar(left=years_since_1940, height=nr_papers_since_1940, width=1.0)
plt.xlim(1940, 2017)
plt.xlabel('year')
plt.ylabel('number of papers');
Explanation: Creating a bar plot to visualize the results (using matplotlib.pyplot.bar):
End of explanation
plt.hist( x=[p.year for p in Summaries.values()], bins=range(1940,2018) );
plt.xlim(1940, 2017)
plt.xlabel('year')
plt.ylabel('number of papers');
Explanation: Alternatively, you can get the same result in a more direct manner by plotting it as a histogram with matplotlib.pyplot.hist:
End of explanation
# flattening the list of lists of authors:
authors_expanded = [ auth for paper in Summaries.values() for auth in paper.authors ]
nr_papers_by_author = Counter( authors_expanded )
print('Number of authors in the dataset with distinct names:')
print(len(nr_papers_by_author))
print('Top 50 authors with greatest number of papers:')
print(sorted(nr_papers_by_author.items(), key=lambda i:i[1], reverse=True)[:50])
Explanation: Papers per Author
Here, we will obtain the distribution characterizing the number of papers published by an author.
End of explanation
plt.hist( x=list(nr_papers_by_author.values()), bins=range(51), log=True )
plt.xlabel('number of papers authored')
plt.ylabel('number of authors');
Explanation: Creating a histogram to visualize the results:
End of explanation
plt.hist( x=[ len(p.authors) for p in Summaries.values() ], bins=range(20), align='left', normed=True )
plt.xlabel('number of authors in one paper')
plt.ylabel('fraction of papers')
plt.xlim(-0.5, 15.5);
Explanation: Authors per Paper
End of explanation
words = [ word.lower() for paper in Summaries.values() for word in paper.title.split(' ') ]
word_counts = Counter(words)
print('Number of distinct words in the paper titles:')
print(len(word_counts))
Explanation: Words in Titles
assemble list of words in paper titles, convert them to lowercase, and remove trailing '.':
End of explanation
# Add your code here
Explanation: Assignments
Your name: ...
Task 1
Create a Python dictionary object that returns sets of author names for a given year. You can name this dictionary, for example, authors_at_year. (You can use a defaultdict with a default value of set.) Demonstrate the working of this dictionary by showing the author set for the year 1941.
End of explanation
# Add your code here
Explanation: Task 2
Based on the dictionary authors_at_year from exercise 1 above, create a plot for the years from 1940 until 2016 that shows how many authors published at least one paper. (You can retrieve the number of unique items in a set s with len(s).)
End of explanation
# Add your code here
Explanation: Task 3
Calculate and plot (e.g. using plt.plot) a graph of the frequency of the 100 most frequent words in titles of papers, from most frequent to least frequent. (You can make use of the data structures created above.)
End of explanation
# Add your code here
Explanation: Task 4
Print out the top 50 most often occurring words in the paper's titles. (You can again make use of the data structures created above.)
End of explanation
<END_TASK>