<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 prelim_month - create Reliability_Names data
2016.12.04 - work log - prelim_month - create Reliability_Names
original file name
Step1:  Setup - virtualenv jupyter kernel
Back to Table of Contents
If you are using a virtualenv, make sure that you
Step2:  Data characterization
Back to Table of Contents
Description of data, for paper.
grp_month article count = 441
Step3:  Reliability data creation - prelim_month
Back to Table of Contents
Create the data.
Initialize from file
Step4:  Example snapshot of configuration in this file
<ASSISTANT_TASK:>
Python Code:
import datetime
print( "packages imported at " + str( datetime.datetime.now() ) )
Explanation: prelim_month - create Reliability_Names data
2016.12.04 - work log - prelim_month - create Reliability_Names
original file name: 2016.12.04-work_log-prelim_month-create_Reliability_Names.ipynb
This is the notebook where the underlying name comparison data was created - one row per person per article, columns for the ways up to ten different coders captured that person from the text.
<h1>Table of Contents<span class="tocSkip"></span></h1>
<div class="toc"><ul class="toc-item"><li><span><a href="#Setup" data-toc-modified-id="Setup-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Setup</a></span><ul class="toc-item"><li><span><a href="#Setup---Imports" data-toc-modified-id="Setup---Imports-1.1"><span class="toc-item-num">1.1&nbsp;&nbsp;</span>Setup - Imports</a></span></li><li><span><a href="#Setup---virtualenv-jupyter-kernel" data-toc-modified-id="Setup---virtualenv-jupyter-kernel-1.2"><span class="toc-item-num">1.2&nbsp;&nbsp;</span>Setup - virtualenv jupyter kernel</a></span></li><li><span><a href="#Setup---Initialize-Django" data-toc-modified-id="Setup---Initialize-Django-1.3"><span class="toc-item-num">1.3&nbsp;&nbsp;</span>Setup - Initialize Django</a></span></li></ul></li><li><span><a href="#Data-characterization" data-toc-modified-id="Data-characterization-2"><span class="toc-item-num">2&nbsp;&nbsp;</span>Data characterization</a></span></li><li><span><a href="#Reliability-data-creation---prelim_month" data-toc-modified-id="Reliability-data-creation---prelim_month-3"><span class="toc-item-num">3&nbsp;&nbsp;</span>Reliability data creation - <code>prelim_month</code></a></span></li><li><span><a href="#Database-backup---sourcenet-2016.12.04.pgsql.gz" data-toc-modified-id="Database-backup---sourcenet-2016.12.04.pgsql.gz-4"><span class="toc-item-num">4&nbsp;&nbsp;</span>Database backup - <code>sourcenet-2016.12.04.pgsql.gz</code></a></span></li><li><span><a href="#Data-cleanup" data-toc-modified-id="Data-cleanup-5"><span class="toc-item-num">5&nbsp;&nbsp;</span>Data cleanup</a></span><ul class="toc-item"><li><span><a href="#Remove-single-name-reliability-data" data-toc-modified-id="Remove-single-name-reliability-data-5.1"><span class="toc-item-num">5.1&nbsp;&nbsp;</span>Remove single-name reliability data</a></span><ul class="toc-item"><li><span><a href="#Single-name-data-assessment" data-toc-modified-id="Single-name-data-assessment-5.1.1"><span class="toc-item-num">5.1.1&nbsp;&nbsp;</span>Single-name data assessment</a></span></li><li><span><a href="#Delete-selected-single-name-data" data-toc-modified-id="Delete-selected-single-name-data-5.1.2"><span class="toc-item-num">5.1.2&nbsp;&nbsp;</span>Delete selected single-name data</a></span></li></ul></li></ul></li></ul></div>
Setup
Back to Table of Contents
Setup - Imports
Back to Table of Contents
End of explanation
%pwd
%ls
%run ../django_init.py
Explanation: Setup - virtualenv jupyter kernel
Back to Table of Contents
If you are using a virtualenv, make sure that you:
have installed your virtualenv as a kernel.
choose the kernel for your virtualenv as the kernel for your notebook (Kernel --> Change kernel).
Since I use a virtualenv, need to get that activated somehow inside this notebook.  One option is to run ../dev/wsgi.py in this notebook, to configure the python environment manually as if you had activated the sourcenet virtualenv.  To do this, you'd make a code cell that contains:
%run ../dev/wsgi.py
This is sketchy, however, because of the changes it makes to your Python environment within the context of whatever your current kernel is.  I'd worry about collisions with the actual Python 3 kernel.  Better, one can install their virtualenv as a separate kernel.  Steps:
activate your virtualenv:
workon sourcenet
in your virtualenv, install the package ipykernel.
pip install ipykernel
use the ipykernel python program to install the current environment as a kernel:
python -m ipykernel install --user --name &lt;env_name&gt; --display-name "&lt;display_name&gt;"
sourcenet example:
python -m ipykernel install --user --name sourcenet --display-name "sourcenet (Python 3)"
More details: http://ipython.readthedocs.io/en/stable/install/kernel_install.html
Setup - Initialize Django
Back to Table of Contents
First, initialize my dev django project, so I can run code in this notebook that references my django models and can talk to the database using my project's settings.
End of explanation
from context_text.models import Article
# how many articles in "grp_month"?
article_qs = Article.objects.filter( tags__name__in = [ "grp_month" ] )
grp_month_count = article_qs.count()
print( "grp_month count = {}".format( grp_month_count ) )
Explanation: Data characterization
Back to Table of Contents
Description of data, for paper.
grp_month article count = 441
End of explanation
%run ../config-coder_index-prelim_month.py
Explanation: Reliability data creation - prelim_month
Back to Table of Contents
Create the data.
Initialize from file:
End of explanation
# output debug JSON to file
my_reliability_instance.debug_output_json_file_path = "/home/jonathanmorgan/" + label + ".json"
#===============================================================================
# process
#===============================================================================
# process articles
#my_reliability_instance.process_articles( tag_list )
# output to database.
#my_reliability_instance.output_reliability_data( label )
print( "reliability data created at " + str( datetime.datetime.now() ) )
Explanation: Example snapshot of configuration in this file:
'''
You must create an index-able instance and place it in my_index_instance before
    you run this code.  The index configuration in this file will be applied to
    the instance stored in "my_index_instance".
Objects you can pass in this instance:
from context_analysis.reliability.reliability_names_builder import ReliabilityNamesBuilder
from context_analysis.network.network_person_info import NetworkPersonInfo
'''
# imports
import datetime
# sourcenet imports
from context_text.shared.context_text_base import ContextTextBase
# context_analysis imports
from context_analysis.reliability.reliability_names_builder import ReliabilityNamesBuilder
from context_analysis.network.network_person_info import NetworkPersonInfo
# return reference
index_helper_OUT = None
# declare variables
tag_list = None
label = ""
# declare variables - user setup
my_info_instance = None
my_reliability_instance = None
current_coder = None
current_coder_id = -1
current_priority = -1
# declare variables - Article_Data filtering.
coder_type = ""
#===============================================================================
# configure
#===============================================================================
# list of tags of articles we want to process.
tag_list = [ "grp_month", ]
# label to associate with results, for subsequent lookup.
label = "prelim_month"
# create index instances
my_info_instance = NetworkPersonInfo()
my_reliability_instance = ReliabilityNamesBuilder()
# ! ====&gt; map coders to indices
# set it up so that...
# ...the ground truth user has highest priority (4) for index 1...
current_coder = ContextTextBase.get_ground_truth_coding_user()
current_coder_id = current_coder.id
current_index = 1
current_priority = 4
my_info_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
my_reliability_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
# ...coder ID 8 is priority 3 for index 1...
current_coder_id = 8
current_index = 1
current_priority = 3
my_info_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
my_reliability_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
# ...coder ID 9 is priority 2 for index 1...
current_coder_id = 9
current_index = 1
current_priority = 2
my_info_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
my_reliability_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
# ...coder ID 10 is priority 1 for index 1...
current_coder_id = 10
current_index = 1
current_priority = 1
my_info_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
my_reliability_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
# ...and automated coder (2) is index 2
current_coder = ContextTextBase.get_automated_coding_user()
current_coder_id = current_coder.id
current_index = 2
current_priority = 1
my_info_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
my_reliability_instance.add_coder_at_index( current_coder_id, current_index, priority_IN = current_priority )
# and only look at coding by those users.  And...
# configure so that it limits to automated coder_type of OpenCalais_REST_API_v2.
coder_type = "OpenCalais_REST_API_v2"
#my_reliability_instance.limit_to_automated_coder_type = "OpenCalais_REST_API_v2"
my_info_instance.automated_coder_type_include_list.append( coder_type )
my_reliability_instance.automated_coder_type_include_list.append( coder_type )
index_helper_OUT = my_info_instance.get_index_helper()
print( "indexing for grp_month/prelim_month initialized at " + str( datetime.datetime.now() ) )
End of explanation
<END_TASK>