<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <!-- Some HTML for ncie picture -->
<p>
    <a href="https
Step1:  A. The semantic connections
Step2:  A. Web of Science - Recursion 1.
Search details
Date
Step3:  Keyword analysis
Step4:  Journal analysis
Step5:  Scopus - Recursive search
Step6:  Journal analysis
Step7:  C. Scopus Recursive search
Step8:  Journal analysis
<ASSISTANT_TASK:>
Python Code:
maketimeseries() # Load this function from bottom of notebook to print.
Explanation: <!-- Some HTML for ncie picture -->
<p>
    <a href="https://commons.wikimedia.org/wiki/File:Open_Science_-_Prinzipien.png#/media/File:Open_Science_-_Prinzipien.png"><img src="https://upload.wikimedia.org/wikipedia/commons/9/9c/Open_Science_-_Prinzipien.png" alt="Open Science - Prinzipien.png" width="400"></a>
    <br>
</p>
<center>Image by Andreas E. Neuhold, CC BY 3.0</center>
Bibliometric probing of the concept 'open science' - a notebook
by Christopher Kullenberg<sup>1</sup> (2017)
E-mail: christopher#kullenberg§gu#se 
Abstract. To date there is only a small number of scientific articles that have been written on the topic of "open science", at least when considering the recent trend in science policy and public discourse. The Web of Science only contains 544 records with the term "open science" in the title, keywords or abstract, and Scopus ammounts to 769 records. Based on article keywords, open science is primarily connected with the concepts of "open access", "open data", "reproducible research", and "data sharing". It occurs most frequently in biomedical-, interdisciplinary- and natural science journals, however, some of the most cited articles have been published in science policy journals.
Contents
Introduction
Method
Software
Main results, a summary
The semantic connections: keywords
Most cited articles
Most common journals
Code, data and figures. 
Web of Science, recursion 1. 
Scopus, recursion 1. 
Scopus, recursion 2.
1. Introduction
The purpose of this notebook is to produce an overview of scientific literature related to the phenomenon of open science. This is a preliminary research note, not a definite result. Feel free to use and modify this notebook in anyway you want, as long as you cite it as follows:
<sup>1</sup> Kullenberg, Christopher (2017) "Bibliometric probing of the concept 'open science' - a notebook", https://github.com/christopherkullenberg/openscienceliterature, accessed YY-MM-DD. 
This notebook uses bibliometric data from the Web of Science and Scopus databases. Since these databases are proprietary and require a subscription, the source data cannot be redistributed here. However, the search strings and dates of retrieval are noted each time a dataset is used, so if you have access to these databases, it should be quite possible to replicate every step. 
2. Method
Since open science is very loosely defined and can have multiple meanings, it is difficult to craft a precise search string in the bibliometric databases. Thus, in this notebook, I probe recursively into the Web of Science and Scopus databases with recursive searches (See Kullenberg & Kasperowski 2016) to generate search terms that can be relevant for getting a better picture of the open science phenomenon. 
|Database|Records|
| --- | --- |
| Web of Science | 544 |
| Scopus 1st search| 769 |
| Scopus 2nd search| 14.146 |
The recursive search works as follows. First each database is queried in the abstract, title and keywords with the search term "open science". Then the keywords of this first database is analysed in terms of frequency and co-occurrence. This gives a first overview of what other keywords are relevant. 
In the next step the keywords from the first iteration are inspected and sorted. I filter out those keywords that are too general in scope, for example "open source" and "reproducibility", in order to craft a more precise combination of search terms. This is a qualitative process, so for every keyword, I make choices on what is relevant and what is not. As this notebook is explorative in nature, this should be regarded as an interpretative practice, where one might return at a later stage and then make other decisions due to increased knowledge about the field. 
A. Software
This Python 3 notebook uses the Pandas dataframe as a method for parsing the bibliometric records into a convenient data structure. For plotting, the Seaborn high-level library for interfacing with Matplotlib is used. Wielding such software for bibliometric analyses might seem a bit strange, but there are many advantages when it comes to reproducibility, since every step can be traced and the software is free. To install all software swiftly, I 
recommend warmly the Anaconda distribution. 
3. Main results, a summary
As the figure below shows, the notion of "open science" is practically non-existent before the turn of the millennium (in quantitative terms). During the past two or tree years, however, the concept seems to have gained a little traction. Note - the slight decrease in publications for 2016 is only a "lag effect" of the update process of the databases.
End of explanation
# General libraries
%matplotlib inline
import pandas as pd
import warnings
warnings.simplefilter(action = "ignore", category = FutureWarning) # Supress some meaningless warnings. 
#from tabulate import tabulate
from collections import Counter
import seaborn as sns
import numpy as np
from itertools import combinations
import matplotlib.pyplot as plt
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 12, 7 # Make figures a little bigger
# Load up this cell to enable the keywordcooccurrence function. 
def cooccurrence(column):
    '''
    Input: a dataframe column containing keywords that are separated by semicolons
           Example: df.DE (Web of Science) or df[Author keywords] (Scopus)
    Output: A list of co-occurring keywords that can be ranked with the Counter function
    '''
    cooccurrencelist = []
    for keyword in column:
        k = str(keyword)
        keywordsperarticle = k.split('; ')
        keywordsperarticle = [word.lower() for word in keywordsperarticle] # Lowers all keywords in each list.
        cooccurrence = list(combinations(keywordsperarticle, 2))
        for c in cooccurrence:
            cooccurrencelist.append(c)
    return(cooccurrencelist)
# This function returns a list of journals that can be easily counted
def frequentjournals(column):
    '''
    Input: a dataframe column containing journal names.
           Example: df.SO (Web of Science) or df[Source title] (Scopus)
    Output: A list of journal names that can be ranked with the Counter function
    '''
    journallist = []
    for journal in column:
        #print(len(journal))
        journallist.append(journal.lower()) # Lower names. Looks bad, but computes well. 
    return(journallist)
Explanation: A. The semantic connections: keywords
Based on a first iterative search on "open science", the term frequently occurs in connection with:
open access
open data
reproducible research
data sharing
reproducibility
big data
collaboration
metadata
meta-analysis
research data
Here it is reasonable to conclude that "open science" is largely defined from an epistemic point of view, as these terms refer to the scientific practice of sharing research findings and data openly (open access, open data), ensuring the reproducibility of research and achieving a better collaboration between scientists. Also, the notions of "meta-data" and "meta-analysis" indicate a concern for standardizing data for the purpose of aggregation (big data). 
B. Most cited articles
Finding the most cited articles can be a good way of finding obligatory points of passage in the literature. However, one must be careful not to draw too far-fetched conclusions from such numbers. After all, the practice of citation cannot be reduced to a single motivational factor. 
Web of Science, 10 most cited articles.
| Author | Year | Title | Journal | Times Cited | DOI/URL |
| --- | --- | --- | --- | --- | --- |
| Nooner, KB; Colcombe, SJ; Tobe, RH; Mennes, M;... | 2012.0 | The NKI-Rockland sample: a model for accelerat... | FRONTIERS IN NEUROSCIENCE | 82.0 | https://dx.doi.org/10.3389/fnins.2012.00152
| Fabrizio, KR; Di Minin, A | 2008.0 | Commercializing the laboratory: Faculty patent... | RESEARCH POLICY | 81.0 | https://dx.doi.org/10.1016/j.respol.2008.01.010
| Castellanos, FX; Di Martino, A; Craddock, RC; ... | 2013.0 | Clinical applications of the functional connec... | NEUROIMAGE | 79.0 | https://dx.doi.org/10.1016/j.neuroimage.2013.04.083
| Markman, GD; Siegel, DS; Wright, M | 2008.0 | Research and Technology Commercialization | JOURNAL OF MANAGEMENT STUDIES | 75.0 | https://dx.doi.org/10.1111/j.1467-6486.2008.00803.x
| Newman, G; Wiggins, A; Crall, A; Graham, E; Ne... | 2012.0 | The future of citizen science: emerging techno... | FRONTIERS IN ECOLOGY AND THE ENVIRONMENT | 74.0 | https://dx.doi.org/10.1890/110294
| Mello, MM; Francer, JK; Wilenzick, M; Teden, P... | 2013.0 | Preparing for Responsible Sharing of Clinical ... | NEW ENGLAND JOURNAL OF MEDICINE | 57.0 | https://dx.doi.org/10.1056/NEJMhle1309073
| Breschi, S; Catalini, C | 2010.0 | Tracing the links between science and technolo... | RESEARCH POLICY | 48.0 | https://dx.doi.org/10.1016/j.respol.2009.11.004
| Procter, R; Williams, R; Stewart, J; Poschen, ... | 2010.0 | Adoption and use of Web 2.0 in scholarly commu... | PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIET... | 44.0 | https://dx.doi.org/10.1098/rsta.2010.0155
| Mueller, ST; Piper, BJ | 2014.0 | The Psychology Experiment Building Language (P... | JOURNAL OF NEUROSCIENCE METHODS | 43.0 | https://dx.doi.org/10.1016/j.jneumeth.2013.10.024
| Mennes, M; Biswal, BB; Castellanos, FX; Milham... | 2013.0 | Making data sharing work: The FCP/INDI experience | NEUROIMAGE | 42.0 | https://dx.doi.org/10.1016/j.neuroimage.2012.10.064
Scopus, 10 most cited articles.
| Author | Year | Title | Journal | Times Cited | DOI/URL |
| --- | --- | --- | --- | --- | --- |
| Partha D., David P.A. |  1994 |  Toward a new economics of science |  Research Policy |  926.0 |  https://dx.doi.org/10.1016/0048-7333(94)01002-1 |
| Mix A.C., Bard E., Schneider R. |  2001 |  Environmental processes of the ice age: Land, ... |  Quaternary Science Reviews |  479.0 |  https://dx.doi.org/10.1016/S0277-3791(00)00145-1 |
| Balconi M., Breschi S., Lissoni F. |  2004 |  Networks of inventors and the role of academia... |  Research Policy |  216.0 |  https://dx.doi.org/10.1016/S0048-7333(03)00108-2 |
| Veugelers R., Cassiman B. |  2005 |  R&D cooperation between firms and universities... |  International Journal of Industrial Organization |  189.0 |  https://dx.doi.org/10.1016/j.ijindorg.2005.01.008 |
| Dosi G., Llerena P., Labini M.S. |  2006 |  The relationships between science, technologie... |  Research Policy |  173.0 |  https://dx.doi.org/10.1016/j.respol.2006.09.012 |
| Pordes R., Petravick D., Kramer B., Olson D., ... |  2007 |  The open science grid |  Journal of Physics: Conference Series |  112.0 |  https://dx.doi.org/10.1088/1742-6596/78/1/012057 |
| Agrawal A. |  2006 |  Engaging the inventor: Exploring licensing str... |  Strategic Management Journal |  111.0 |  https://dx.doi.org/10.1002/smj.508 |
| Markman G.D., Siegel D.S., Wright M. |  2008 |  Research and technology commercialization |  Journal of Management Studies |  110.0 |  https://dx.doi.org/10.1111/j.1467-6486.2008.00803.x |
| Nooner K.B., Colcombe S.J., Tobe R.H., Mennes ... |  2012 |  The NKI-Rockland sample: A model for accelerat... |  Frontiers in Neuroscience |  105.0 |  https://dx.doi.org/10.3389/fnins.2012.00152 |
| Fabrizio K.R., Di Minin A. |  2008 |  Commercializing the laboratory: Faculty patent... |  Research Policy |  92.0 |  https://dx.doi.org/10.1016/j.respol.2008.01.010
C. Most common journals
Depending on whether you query Scopus or Web of Science, you get different results concerning the distribution of records that are conference proceedings or journal articles. Here I have selected to include only journal articles, not conference papers, reviews or opinion pieces. The most common journals are: 
Elife
Research policy
Plos ONE
Journal of Technology Transfer
Peerj
Science
Conclusion: The dominant fields for "open science" appears to be biomedical-, multidisciplinary-, and natural sciences. 
4. Code and Data
End of explanation
#fel data , error_bad_lines=False
df = pd.read_csv('.data/WoS549recs20170121.tsv', sep="\t", encoding='utf-8') # Input: web of science tsv file, utf-8 encoding. 
df.head(3)
# Print this for explanation of WoS columns
woskeyfile = open('woskeys.txt')
woskeys = woskeyfile.read()
#print(woskeys)
dfTC = df.sort('TC', ascending=False) # Order dataframe by times cited
dfTC[['AU', 'PY', 'TI', 'SO', 'TC', 'DI']].head(10) # Ten most cited articles. 
publicationyears = sns.factorplot('PY', data=df, kind='count', size=10, aspect=2)
Explanation: A. Web of Science - Recursion 1.
Search details
Date: 
20170121, General Search. 
Search string: 
TS="open science"
Result:
544 Records. 
Important note on parsing the WoS tsv files
Delete the column BJ since it is erronneous. The separator is tab, and there is no delimiter.
End of explanation
# Read all keywords into a list. 
allkeywords = []
for keyword in df.DE:
    k = str(keyword)
    keywordsperarticle = k.split('; ')
    for word in keywordsperarticle:
        allkeywords.append(word.lower()) # make all lower case for better string matching. 
        
print("Total number of keywords: " + str(len(allkeywords)))
# Find the most common keywords
commonkeywords = Counter(allkeywords).most_common(10) # Increase if you want more results
for word in commonkeywords:
    if word[0] != "nan": # Clean out empty fields. 
        print(word[0] + "\t" + str(word[1]))
Counter(cooccurrence(df.DE)).most_common(10)
keywordsDF = pd.DataFrame(commonkeywords, columns=["keyword", "freq"])
# Plot figure while excluding "nan" values in Dataframe
keywordsWoS = sns.factorplot(x='keyword', y='freq', kind="bar", data=keywordsDF[keywordsDF.keyword.str.contains("nan") == False], size=8, aspect=2)
keywordsWoS.set_xticklabels(rotation=45)
Explanation: Keyword analysis
End of explanation
# To get only journal articles, select df.SO[df['PT'] == 'J']
for journal in Counter(frequentjournals(df.SO[df['PT'] == 'J'] )).most_common(10):
    print(journal)
Explanation: Journal analysis
End of explanation
df2 = pd.read_csv('.data/scopusRecursionOne769recs20170120.csv', encoding="utf-8")
# Print this for Scopus column names
#for header in list(df2.columns.values):
#    print(header)
#df2['Document Type']
df2TC = df2.sort('Cited by', ascending=False) # Order dataframe by times cited
df2TC.tail(3)
# NOTE: there is a cryptic character in front of the Authors column: 
# Sometimes '﻿Authors' works, sometimes 'Authors', depending on system locale settings. 
df2TC[['Authors', 'Year', 'Title', 'Source title', 'Cited by', 'DOI']].head(10) # Ten most cited articles. 
# Create a time series of the publications. Some data cleaning is needed: 
df2TCdropna = df2TC.Year.dropna() # Drop empty values in years
df2TCyears = pd.DataFrame(df2TCdropna.astype(int)) # Convert existing years to integers, make new dataframe
publicationyearsScopus = sns.factorplot('Year', data=df2TCyears, kind='count', size=8, aspect=2)
# Read all keywords into a list. 
allscopuskeywords = []
for keyword in df2['Author Keywords']:
    k = str(keyword)
    keywordsperarticle = k.split('; ')
    for word in keywordsperarticle:
        allscopuskeywords.append(word.lower()) # make all lower case for better string matching. 
        
print("Total number of keywords: " + str(len(allkeywords)))
# Find the most common keywords
commonscopuskeywords = Counter(allscopuskeywords).most_common(20) # Increase if you want more results
for word in commonscopuskeywords:
    if word[0] != "nan": # Clean out empty fields. 
        print(word[0] + "\t" + str(word[1]))
# Get co-occurrences
Counter(cooccurrence(df2['Author Keywords'])).most_common(10)
keywordsScopusDF = pd.DataFrame(commonscopuskeywords, columns=["keyword", "freq"])
# Plot figure while excluding "nan" values in Dataframe
keywordsScP = sns.factorplot(x='keyword', y='freq', kind="bar", data=keywordsScopusDF[keywordsScopusDF.keyword.str.contains("nan") == False], size=6, aspect=2)
keywordsScP.set_xticklabels(rotation=45)
keywordsScP.fig.text(0.65, 0.7, "Scopus - Recursion 1:\nSearchstring: \
'open science'\nMost frequent keywords\nN=769", ha ='left', fontsize = 15)
Explanation: Scopus - Recursive search: Iteration 1
Search date:
20170120 (TITLE-ABS-KEY"
Search string:
"open science"
Records:
769
End of explanation
# For journal articles only: df2['Source title'][df2['Document Type'] == 'Article']
for journal in Counter(frequentjournals(df2['Source title'][df2['Document Type'] == 'Article'])).most_common(10):
    print(journal)
Explanation: Journal analysis
End of explanation
df3 = pd.read_csv('.data/scopusRecursionTwo14146recs20170120.csv')
df3.tail(3) # Verify all data is there. 
# Create a time series of the publications. Some data cleaning is needed: 
df3dropna = df3.Year.dropna() # Drop empty values in years
df3years = pd.DataFrame(df3dropna.astype(int)) # Convert existing years to integers, make new dataframe
publicationyearsScopus = sns.factorplot('Year', data=df3years, kind='count', size=8, aspect=2)
publicationyearsScopus.set_xticklabels(rotation=45)
Explanation: C. Scopus Recursive search: Iteration 2
Scopus Search date:
20170120 (TITLE-ABS-KEY)
Note: Scopus will not include Author Keywords when exporting this large ammount of data. 
Search string:
Included:
"open science" OR "data sharing" OR "open data" OR "open science grid"
Excluded
* "open access" - Too broad, gets around 40k hits. 
* "open source" - Too broad.
* "reproducibility" - Too broad.
* "big data" - Too broad.
* "collaboration" - Too broad.
* ... 
Records:
14.146
End of explanation
for journal in Counter(frequentjournals(df3['Source title'][df3['Document Type'] == "Article"])).most_common(10):
    print(journal)
WoSyears = []
for year in df.PY.dropna():
    if year > 1990.0:
        WoSyears.append(int(year))
#print(sorted(WoSyears))
    
    
    
Scopusyears = []
for year in df2['Year'].dropna():
    if year > 1990.0:
        Scopusyears.append(year)
dfWoSyears = pd.DataFrame.from_dict(Counter(WoSyears), orient='index', dtype=None)
dfsorted = pd.DataFrame.sort_index(dfWoSyears)
dfsorted.head()
dfScopusyears = pd.DataFrame.from_dict(Counter(Scopusyears), orient='index', dtype=None)
dfSsorted = pd.DataFrame.sort_index(dfScopusyears)
dfSsorted.head()
def maketimeseries():
    plt.title('"Open science" - Published articles and proceedings, 1990-2016', fontsize=16)
    plt.xlabel('Year \n', fontsize=16)
    plt.ylabel('Records', fontsize=16)
    plt.ylim([0, 150])
    plt.xlim(1990,2016)
    # Line styles: http://matplotlib.org/1.3.1/examples/pylab_examples/line_styles.html
    plt.plot(dfsorted,  linestyle='--', marker='D', label="Web of Science")
    plt.plot(dfSsorted,  linestyle='-', marker='o', label="Scopus")
    # legend guide: http://matplotlib.org/1.3.1/users/legend_guide.html
    plt.legend(loc=2, borderaxespad=0., fontsize=16)
    plt.savefig(".data/fig1.png")
maketimeseries()
Explanation: Journal analysis
End of explanation
<END_TASK>