<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1>Table of Contents<span class="tocSkip"></span></h1>
<div class="toc"><ul class="toc-item"><li><span><a href="#Introduction" data-toc-modified-id="Introduction-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href="#Setup" data-toc-modified-id="Setup-2"><span class="toc-item-num">2&nbsp;&nbsp;</span>Setup</a></span><ul class="toc-item"><li><span><a href="#Setup---Debug" data-toc-modified-id="Setup---Debug-2.1"><span class="toc-item-num">2.1&nbsp;&nbsp;</span>Setup - Debug</a></span></li><li><span><a href="#Setup---Imports" data-toc-modified-id="Setup---Imports-2.2"><span class="toc-item-num">2.2&nbsp;&nbsp;</span>Setup - Imports</a></span></li><li><span><a href="#Setup---working-folder-paths" data-toc-modified-id="Setup---working-folder-paths-2.3"><span class="toc-item-num">2.3&nbsp;&nbsp;</span>Setup - working folder paths</a></span></li><li><span><a href="#Setup---logging" data-toc-modified-id="Setup---logging-2.4"><span class="toc-item-num">2.4&nbsp;&nbsp;</span>Setup - logging</a></span></li><li><span><a href="#Setup---virtualenv-jupyter-kernel" data-toc-modified-id="Setup---virtualenv-jupyter-kernel-2.5"><span class="toc-item-num">2.5&nbsp;&nbsp;</span>Setup - virtualenv jupyter kernel</a></span></li><li><span><a href="#Setup---Initialize-Django" data-toc-modified-id="Setup---Initialize-Django-2.6"><span class="toc-item-num">2.6&nbsp;&nbsp;</span>Setup - Initialize Django</a></span></li><li><span><a href="#Setup---Initialize-LoggingHelper" data-toc-modified-id="Setup---Initialize-LoggingHelper-2.7"><span class="toc-item-num">2.7&nbsp;&nbsp;</span>Setup - Initialize LoggingHelper</a></span></li><li><span><a href="#Setup---initialize-ProquestHNPNewspaper" data-toc-modified-id="Setup---initialize-ProquestHNPNewspaper-2.8"><span class="toc-item-num">2.8&nbsp;&nbsp;</span>Setup - initialize ProquestHNPNewspaper</a></span><ul class="toc-item"><li><span><a href="#load-from-database" data-toc-modified-id="load-from-database-2.8.1"><span class="toc-item-num">2.8.1&nbsp;&nbsp;</span>load from database</a></span></li><li><span><a href="#set-up-manually" data-toc-modified-id="set-up-manually-2.8.2"><span class="toc-item-num">2.8.2&nbsp;&nbsp;</span>set up manually</a></span></li></ul></li></ul></li><li><span><a href="#Find-articles-to-be-loaded" data-toc-modified-id="Find-articles-to-be-loaded-3"><span class="toc-item-num">3&nbsp;&nbsp;</span>Find articles to be loaded</a></span><ul class="toc-item"><li><span><a href="#Uncompress-files" data-toc-modified-id="Uncompress-files-3.1"><span class="toc-item-num">3.1&nbsp;&nbsp;</span>Uncompress files</a></span></li><li><span><a href="#Work-with-uncompressed-files" data-toc-modified-id="Work-with-uncompressed-files-3.2"><span class="toc-item-num">3.2&nbsp;&nbsp;</span>Work with uncompressed files</a></span></li><li><span><a href="#parse-and-load-XML-files" data-toc-modified-id="parse-and-load-XML-files-3.3"><span class="toc-item-num">3.3&nbsp;&nbsp;</span>parse and load XML files</a></span></li><li><span><a href="#build-list-of-all-ObjectTypes" data-toc-modified-id="build-list-of-all-ObjectTypes-3.4"><span class="toc-item-num">3.4&nbsp;&nbsp;</span>build list of all ObjectTypes</a></span></li><li><span><a href="#map-files-to-types" data-toc-modified-id="map-files-to-types-3.5"><span class="toc-item-num">3.5&nbsp;&nbsp;</span>map files to types</a></span><ul class="toc-item"><li><span><a href="#explore-all-known-object-types" data-toc-modified-id="explore-all-known-object-types-3.5.1"><span class="toc-item-num">3.5.1&nbsp;&nbsp;</span>explore all known object types</a></span></li><li><span><a href="#files-in-archive-Newsday_20171006231925_00050---1989" data-toc-modified-id="files-in-archive-Newsday_20171006231925_00050---1989-3.5.2"><span class="toc-item-num">3.5.2&nbsp;&nbsp;</span>files in archive Newsday_20171006231925_00050 - 1989</a></span></li></ul></li></ul></li><li><span><a href="#XML-Analysis" data-toc-modified-id="XML-Analysis-4"><span class="toc-item-num">4&nbsp;&nbsp;</span>XML Analysis</a></span></li><li><span><a href="#TODO" data-toc-modified-id="TODO-5"><span class="toc-item-num">5&nbsp;&nbsp;</span>TODO</a></span></li></ul></div>
Introduction
Back to Table of Contents
This is a notebook that expands on the OpenCalais code in the file article_coding.py, also in this folder.  It includes more sections on selecting publications you want to submit to OpenCalais as an example.  It is intended to be copied and re-used.
Setup
Back to Table of Contents
Setup - Debug
Back to Table of Contents
Step1:  Setup - Imports
Back to Table of Contents
Step2:  Setup - working folder paths
Back to Table of Contents
What data are we looking at?
Step3:  Setup - logging
Back to Table of Contents
configure logging for this notebook's kernel (If you do not run this cell, you'll get the django application's logging configuration.
Step4:  Setup - virtualenv jupyter kernel
Back to Table of Contents
If you are using a virtualenv, make sure that you
Step5:  Setup - Initialize LoggingHelper
Back to Table of Contents
Create a LoggingHelper instance to use to log debug and also print at the same time.
Preconditions
Step6:  Setup - initialize ProquestHNPNewspaper
Back to Table of Contents
Create an initialize an instance of ProquestHNPNewspaper for this paper.
load from database
Back to Table of Contents
Step7:  set up manually
Back to Table of Contents
Step8:  If desired, add to database.
Step9:  Find articles to be loaded
Back to Table of Contents
Specify which folder of XML files should be loaded into system, then process all files within the folder.
The compressed archives from proquest_hnp just contain publication XML files, no containing folder.
To process
Step10:  For each *.zip file in the paper's source folder
Step11:  Work with uncompressed files
Back to Table of Contents
Change working directories to the uncompressed paper path.
Step12:  parse and load XML files
Back to Table of Contents
Load one of the files into memory and see what we can do with it.  Beautiful Soup?
Looks like the root element is "Record", then the high-level type of the article is "ObjectType".
ObjectType values
Step13:  Processing 5752 files in /mnt/hgfs/projects/phd/proquest_hnp/uncompressed/BostonGlobe/BG_20171002210239_00001
----&gt; XML file count
Step14:  Example output
Step15:  explore all known object types
Back to Table of Contents
Look at all known object types to see which contain actual news content.
Step16:  files in archive Newsday_20171006231925_00050 - 1989
Back to Table of Contents
Archive details
<ASSISTANT_TASK:>
Python Code:
debug_flag = False
Explanation: <h1>Table of Contents<span class="tocSkip"></span></h1>
<div class="toc"><ul class="toc-item"><li><span><a href="#Introduction" data-toc-modified-id="Introduction-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href="#Setup" data-toc-modified-id="Setup-2"><span class="toc-item-num">2&nbsp;&nbsp;</span>Setup</a></span><ul class="toc-item"><li><span><a href="#Setup---Debug" data-toc-modified-id="Setup---Debug-2.1"><span class="toc-item-num">2.1&nbsp;&nbsp;</span>Setup - Debug</a></span></li><li><span><a href="#Setup---Imports" data-toc-modified-id="Setup---Imports-2.2"><span class="toc-item-num">2.2&nbsp;&nbsp;</span>Setup - Imports</a></span></li><li><span><a href="#Setup---working-folder-paths" data-toc-modified-id="Setup---working-folder-paths-2.3"><span class="toc-item-num">2.3&nbsp;&nbsp;</span>Setup - working folder paths</a></span></li><li><span><a href="#Setup---logging" data-toc-modified-id="Setup---logging-2.4"><span class="toc-item-num">2.4&nbsp;&nbsp;</span>Setup - logging</a></span></li><li><span><a href="#Setup---virtualenv-jupyter-kernel" data-toc-modified-id="Setup---virtualenv-jupyter-kernel-2.5"><span class="toc-item-num">2.5&nbsp;&nbsp;</span>Setup - virtualenv jupyter kernel</a></span></li><li><span><a href="#Setup---Initialize-Django" data-toc-modified-id="Setup---Initialize-Django-2.6"><span class="toc-item-num">2.6&nbsp;&nbsp;</span>Setup - Initialize Django</a></span></li><li><span><a href="#Setup---Initialize-LoggingHelper" data-toc-modified-id="Setup---Initialize-LoggingHelper-2.7"><span class="toc-item-num">2.7&nbsp;&nbsp;</span>Setup - Initialize LoggingHelper</a></span></li><li><span><a href="#Setup---initialize-ProquestHNPNewspaper" data-toc-modified-id="Setup---initialize-ProquestHNPNewspaper-2.8"><span class="toc-item-num">2.8&nbsp;&nbsp;</span>Setup - initialize ProquestHNPNewspaper</a></span><ul class="toc-item"><li><span><a href="#load-from-database" data-toc-modified-id="load-from-database-2.8.1"><span class="toc-item-num">2.8.1&nbsp;&nbsp;</span>load from database</a></span></li><li><span><a href="#set-up-manually" data-toc-modified-id="set-up-manually-2.8.2"><span class="toc-item-num">2.8.2&nbsp;&nbsp;</span>set up manually</a></span></li></ul></li></ul></li><li><span><a href="#Find-articles-to-be-loaded" data-toc-modified-id="Find-articles-to-be-loaded-3"><span class="toc-item-num">3&nbsp;&nbsp;</span>Find articles to be loaded</a></span><ul class="toc-item"><li><span><a href="#Uncompress-files" data-toc-modified-id="Uncompress-files-3.1"><span class="toc-item-num">3.1&nbsp;&nbsp;</span>Uncompress files</a></span></li><li><span><a href="#Work-with-uncompressed-files" data-toc-modified-id="Work-with-uncompressed-files-3.2"><span class="toc-item-num">3.2&nbsp;&nbsp;</span>Work with uncompressed files</a></span></li><li><span><a href="#parse-and-load-XML-files" data-toc-modified-id="parse-and-load-XML-files-3.3"><span class="toc-item-num">3.3&nbsp;&nbsp;</span>parse and load XML files</a></span></li><li><span><a href="#build-list-of-all-ObjectTypes" data-toc-modified-id="build-list-of-all-ObjectTypes-3.4"><span class="toc-item-num">3.4&nbsp;&nbsp;</span>build list of all ObjectTypes</a></span></li><li><span><a href="#map-files-to-types" data-toc-modified-id="map-files-to-types-3.5"><span class="toc-item-num">3.5&nbsp;&nbsp;</span>map files to types</a></span><ul class="toc-item"><li><span><a href="#explore-all-known-object-types" data-toc-modified-id="explore-all-known-object-types-3.5.1"><span class="toc-item-num">3.5.1&nbsp;&nbsp;</span>explore all known object types</a></span></li><li><span><a href="#files-in-archive-Newsday_20171006231925_00050---1989" data-toc-modified-id="files-in-archive-Newsday_20171006231925_00050---1989-3.5.2"><span class="toc-item-num">3.5.2&nbsp;&nbsp;</span>files in archive Newsday_20171006231925_00050 - 1989</a></span></li></ul></li></ul></li><li><span><a href="#XML-Analysis" data-toc-modified-id="XML-Analysis-4"><span class="toc-item-num">4&nbsp;&nbsp;</span>XML Analysis</a></span></li><li><span><a href="#TODO" data-toc-modified-id="TODO-5"><span class="toc-item-num">5&nbsp;&nbsp;</span>TODO</a></span></li></ul></div>
Introduction
Back to Table of Contents
This is a notebook that expands on the OpenCalais code in the file article_coding.py, also in this folder.  It includes more sections on selecting publications you want to submit to OpenCalais as an example.  It is intended to be copied and re-used.
Setup
Back to Table of Contents
Setup - Debug
Back to Table of Contents
End of explanation
import datetime
import glob
import logging
import lxml
import os
import six
import xml
import xmltodict
import zipfile
Explanation: Setup - Imports
Back to Table of Contents
End of explanation
# paper identifier
paper_identifier = "Newsday"
archive_identifier = "Newsday_20171006230211_00001"
# source
source_paper_folder = "/mnt/hgfs/projects/phd/proquest_hnp/proquest_hnp/data"
source_paper_path = "{}/{}".format( source_paper_folder, paper_identifier )
# uncompressed
uncompressed_paper_folder = "/mnt/hgfs/projects/phd/proquest_hnp/uncompressed"
uncompressed_paper_path = "{}/{}".format( uncompressed_paper_folder, paper_identifier )
# make sure an identifier is set before you make a path here.
if ( ( archive_identifier is not None ) and ( archive_identifier != "" ) ):
    
    # identifier is set.
    source_archive_file = "{}.zip".format( archive_identifier )
    source_archive_path = "{}/{}".format( source_paper_path, source_archive_file )
    uncompressed_archive_path = "{}/{}".format( uncompressed_paper_path, archive_identifier )
#-- END check to see if archive_identifier present. --#
%pwd
# current working folder
current_working_folder = "/home/jonathanmorgan/work/django/research/work/phd_work/data/article_loading/proquest_hnp/{}".format( paper_identifier )
current_datetime = datetime.datetime.now()
current_date_string = current_datetime.strftime( "%Y-%m-%d-%H-%M-%S" )
Explanation: Setup - working folder paths
Back to Table of Contents
What data are we looking at?
End of explanation
logging_file_name = "{}/research-data_load-{}-{}.log.txt".format( current_working_folder, paper_identifier, current_date_string )
logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
    filename = logging_file_name,
    filemode = 'w' # set to 'a' if you want to append, rather than overwrite each time.
)
Explanation: Setup - logging
Back to Table of Contents
configure logging for this notebook's kernel (If you do not run this cell, you'll get the django application's logging configuration.
End of explanation
# init django
django_init_folder = "/home/jonathanmorgan/work/django/research/work/phd_work"
django_init_path = "django_init.py"
if( ( django_init_folder is not None ) and ( django_init_folder != "" ) ):
    
    # add folder to front of path.
    django_init_path = "{}/{}".format( django_init_folder, django_init_path )
    
#-- END check to see if django_init folder. --#
%run $django_init_path
# context_text imports
from context_text.article_coding.article_coding import ArticleCoder
from context_text.article_coding.article_coding import ArticleCoding
from context_text.article_coding.open_calais_v2.open_calais_v2_article_coder import OpenCalaisV2ArticleCoder
from context_text.collectors.newsbank.newspapers.GRPB import GRPB
from context_text.collectors.newsbank.newspapers.DTNB import DTNB
from context_text.models import Article
from context_text.models import Article_Subject
from context_text.models import Newspaper
from context_text.shared.context_text_base import ContextTextBase
# context_text_proquest_hnp
from context_text_proquest_hnp.proquest_hnp_newspaper_helper import ProquestHNPNewspaperHelper
Explanation: Setup - virtualenv jupyter kernel
Back to Table of Contents
If you are using a virtualenv, make sure that you:
have installed your virtualenv as a kernel.
choose the kernel for your virtualenv as the kernel for your notebook (Kernel --> Change kernel).
Since I use a virtualenv, need to get that activated somehow inside this notebook.  One option is to run ../dev/wsgi.py in this notebook, to configure the python environment manually as if you had activated the sourcenet virtualenv.  To do this, you'd make a code cell that contains:
%run ../dev/wsgi.py
This is sketchy, however, because of the changes it makes to your Python environment within the context of whatever your current kernel is.  I'd worry about collisions with the actual Python 3 kernel.  Better, one can install their virtualenv as a separate kernel.  Steps:
activate your virtualenv:
workon research
in your virtualenv, install the package ipykernel.
pip install ipykernel
use the ipykernel python program to install the current environment as a kernel:
python -m ipykernel install --user --name &lt;env_name&gt; --display-name "&lt;display_name&gt;"
sourcenet example:
python -m ipykernel install --user --name sourcenet --display-name "research (Python 3)"
More details: http://ipython.readthedocs.io/en/stable/install/kernel_install.html
Setup - Initialize Django
Back to Table of Contents
First, initialize my dev django project, so I can run code in this notebook that references my django models and can talk to the database using my project's settings.
End of explanation
# python_utilities
from python_utilities.logging.logging_helper import LoggingHelper
# init
my_logging_helper = LoggingHelper()
my_logging_helper.set_logger_name( "proquest_hnp-article-loading-{}".format( paper_identifier ) )
log_message = None
Explanation: Setup - Initialize LoggingHelper
Back to Table of Contents
Create a LoggingHelper instance to use to log debug and also print at the same time.
Preconditions: Must be run after Django is initialized, since python_utilities is in the django path.
End of explanation
my_paper = ProquestHNPNewspaperHelper()
paper_instance = my_paper.initialize_from_database( paper_identifier )
my_paper.source_all_papers_folder = source_paper_folder
my_paper.destination_all_papers_folder = uncompressed_paper_folder
print( my_paper )
print( paper_instance )
Explanation: Setup - initialize ProquestHNPNewspaper
Back to Table of Contents
Create an initialize an instance of ProquestHNPNewspaper for this paper.
load from database
Back to Table of Contents
End of explanation
my_paper = ProquestHNPNewspaperHelper()
my_paper.paper_identifier = paper_identifier
my_paper.source_all_papers_folder = source_paper_folder
my_paper.source_paper_path = source_paper_path
my_paper.destination_all_papers_folder = uncompressed_paper_folder
my_paper.destination_paper_path = uncompressed_paper_path
my_paper.paper_start_year = 1940
my_paper.paper_end_year = 1989
my_newspaper = Newspaper.objects.get( id = 7 )
my_paper.newspaper = my_newspaper
Explanation: set up manually
Back to Table of Contents
End of explanation
phnp_newspaper_instance = my_paper.create_PHNP_newspaper()
print( phnp_newspaper_instance )
Explanation: If desired, add to database.
End of explanation
# create folder to hold the results of decompressing paper's zip files.
did_uncomp_paper_folder_exist = my_paper.make_dest_paper_folder()
Explanation: Find articles to be loaded
Back to Table of Contents
Specify which folder of XML files should be loaded into system, then process all files within the folder.
The compressed archives from proquest_hnp just contain publication XML files, no containing folder.
To process:
uncompresed paper folder ( &lt;paper_folder&gt; ) - make a folder in /mnt/hgfs/projects/phd/proquest_hnp/uncompressed for the paper whose data you are working with, named the same as the paper's folder in /mnt/hgfs/projects/phd/proquest_hnp/proquest_hnp/data.
for example, for the Boston Globe, name it "BostonGlobe".
uncompressed archive folder ( &lt;archive_folder&gt; ) - inside a given paper's folder in uncompressed, for each archive file, create a folder named the same as the archive file, but with no ".zip" at the end.
For example, for the file "BG_20171002210239_00001.zip", make a folder named "BG_20171002210239_00001".
path should be "&lt;paper_folder&gt;/&lt;archive_name_no_zip&gt;.
unzip the archive into this folder:
unzip &lt;path_to_zip&gt; -d &lt;archive_folder&gt;
Uncompress files
Back to Table of Contents
See if the uncompressed paper folder exists.  If not, set flag and create it.
End of explanation
# decompress the files
my_paper.uncompress_paper_zip_files()
Explanation: For each *.zip file in the paper's source folder:
parse file name from path returned by glob.
parse the part before ".zip" from the file name.  This is referred to subsequently as the "archive identifier".
check if folder named the same as the "archive identifier" is present.
If no:
create it.
then, uncompress the archive into it.
If yes:
output a message.  Don't want to uncompress if it was already uncompressed once.
End of explanation
%cd $uncompressed_paper_path
%ls
Explanation: Work with uncompressed files
Back to Table of Contents
Change working directories to the uncompressed paper path.
End of explanation
# loop over files in the current archive folder path.
object_type_to_count_map = my_paper.process_archive_object_types( uncompressed_archive_path )
Explanation: parse and load XML files
Back to Table of Contents
Load one of the files into memory and see what we can do with it.  Beautiful Soup?
Looks like the root element is "Record", then the high-level type of the article is "ObjectType".
ObjectType values:
Advertisement
...
Good options for XML parser:
lxml.etree - https://stackoverflow.com/questions/12290091/reading-xml-file-and-fetching-its-attributes-value-in-python
xmltodict - https://docs.python-guide.org/scenarios/xml/
beautifulsoup using lxml
End of explanation
xml_folder_list = glob.glob( "{}/*".format( uncompressed_paper_path ) )
print( "folder_list: {}".format( xml_folder_list ) )
# build map of all object types for a paper to the overall counts of each
paper_object_type_to_count_map = my_paper.process_paper_object_types()
Explanation: Processing 5752 files in /mnt/hgfs/projects/phd/proquest_hnp/uncompressed/BostonGlobe/BG_20171002210239_00001
----&gt; XML file count: 5752
Counters:
- Processed 5752 files
- No Record: 0
- No ObjectType: 0
- No ObjectType value: 0
ObjectType values and occurrence counts:
- A|d|v|e|r|t|i|s|e|m|e|n|t: 1902
- Article|Feature: 1792
- N|e|w|s: 53
- Commentary|Editorial: 36
- G|e|n|e|r|a|l| |I|n|f|o|r|m|a|t|i|o|n: 488
- S|t|o|c|k| |Q|u|o|t|e: 185
- Advertisement|Classified Advertisement: 413
- E|d|i|t|o|r|i|a|l| |C|a|r|t|o|o|n|/|C|o|m|i|c: 31
- Correspondence|Letter to the Editor: 119
- Front Matter|Table of Contents: 193
- O|b|i|t|u|a|r|y: 72
- F|r|o|n|t| |P|a|g|e|/|C|o|v|e|r| |S|t|o|r|y: 107
- I|m|a|g|e|/|P|h|o|t|o|g|r|a|p|h: 84
- Marriage Announcement|News: 6
- I|l|l|u|s|t|r|a|t|i|o|n: 91
- R|e|v|i|e|w: 133
- C|r|e|d|i|t|/|A|c|k|n|o|w|l|e|d|g|e|m|e|n|t: 30
- News|Legal Notice: 17
build list of all ObjectTypes
Back to Table of Contents
Loop over all folders in the paper path.  For each folder, grab all files in the folder.  For each file, parse XML, then get the ObjectType value and if it isn't already in map of obect types to counts, add it.  Increment count.
From command line, in the uncompressed BostonGlobe folder:
find . -type f -iname "*.xml" | wc -l
resulted in 11,374,500 articles.  That is quite a few.
End of explanation
news_object_type_list = []
news_object_type_list.append( 'Article|Feature' )
news_object_type_list.append( 'Feature|Article' )
news_object_type_list.append( 'F|r|o|n|t| |P|a|g|e|/|C|o|v|e|r| |S|t|o|r|y' )
Explanation: Example output:
XML file count: 5752
Counters:
- Processed 5752 files
- No Record: 0
- No ObjectType: 0
- No ObjectType value: 0
ObjectType values and occurrence counts:
- A|d|v|e|r|t|i|s|e|m|e|n|t: 2114224
- Feature|Article: 5271887
- I|m|a|g|e|/|P|h|o|t|o|g|r|a|p|h: 249942
- O|b|i|t|u|a|r|y: 625143
- G|e|n|e|r|a|l| |I|n|f|o|r|m|a|t|i|o|n: 1083164
- S|t|o|c|k| |Q|u|o|t|e: 202776
- N|e|w|s: 140274
- I|l|l|u|s|t|r|a|t|i|o|n: 106925
- F|r|o|n|t| |P|a|g|e|/|C|o|v|e|r| |S|t|o|r|y: 386421
- E|d|i|t|o|r|i|a|l| |C|a|r|t|o|o|n|/|C|o|m|i|c: 78993
- Editorial|Commentary: 156342
- C|r|e|d|i|t|/|A|c|k|n|o|w|l|e|d|g|e|m|e|n|t: 68356
- Classified Advertisement|Advertisement: 291533
- R|e|v|i|e|w: 86889
- Table of Contents|Front Matter: 69798
- Letter to the Editor|Correspondence: 202071
- News|Legal Notice: 24053
- News|Marriage Announcement: 41314
- B|i|r|t|h| |N|o|t|i|c|e: 926
- News|Military/War News: 3
- U|n|d|e|f|i|n|e|d: 5
- Article|Feature: 137526
- Front Matter|Table of Contents: 11195
- Commentary|Editorial: 3386
- Marriage Announcement|News: 683
- Correspondence|Letter to the Editor: 7479
- Legal Notice|News: 1029
- Advertisement|Classified Advertisement: 12163
map files to types
Back to Table of Contents
Choose a directory, then loop over the files in the directory to build a map of types to lists of file names.
End of explanation
# get list of all object types
master_object_type_list = my_paper.get_all_object_types()
print( "Object Types: {}".format( master_object_type_list ) )
# directory to work in.
uncompressed_archive_folder = "Newsday_20171006231925_00050"
uncompressed_archive_path = "{}/{}".format( uncompressed_paper_path, uncompressed_archive_folder )
print( 'Uncompressed archive folder: {}'.format( uncompressed_archive_path ) )
# build map of file types to lists of files of that type in specified folder.
object_type_to_file_path_map = my_paper.map_archive_folder_files_to_types( uncompressed_archive_path )
# which types do we want to preview?
#types_to_output = news_object_type_list
types_to_output = [ "Advertisement|Classified Advertisement" ]
types_to_output = [ "A|d|v|e|r|t|i|s|e|m|e|n|t" ]
types_to_output = [ 'Advertisement|Classified Advertisement' ]
types_to_output = [ 'Article|Feature' ]
types_to_output = [ 'B|i|r|t|h| |N|o|t|i|c|e' ]
types_to_output = [ 'Classified Advertisement|Advertisement' ]
types_to_output = [ 'Commentary|Editorial' ]
types_to_output = [ 'Correspondence|Letter to the Editor' ]
types_to_output = [ 'C|r|e|d|i|t|/|A|c|k|n|o|w|l|e|d|g|e|m|e|n|t' ]
types_to_output = [ 'E|d|i|t|o|r|i|a|l| |C|a|r|t|o|o|n|/|C|o|m|i|c' ]
types_to_output = [ 'Editorial|Commentary' ]
types_to_output = [ 'Feature|Article' ]
types_to_output = [ 'Front Matter|Table of Contents' ]
types_to_output = [ 'F|r|o|n|t| |P|a|g|e|/|C|o|v|e|r| |S|t|o|r|y' ]
types_to_output = [ 'G|e|n|e|r|a|l| |I|n|f|o|r|m|a|t|i|o|n' ]
types_to_output = [ 'I|l|l|u|s|t|r|a|t|i|o|n' ]
types_to_output = [ 'I|m|a|g|e|/|P|h|o|t|o|g|r|a|p|h' ]
types_to_output = [ 'Legal Notice|News' ]
types_to_output = [ 'Letter to the Editor|Correspondence' ]
types_to_output = [ 'Marriage Announcement|News' ]
types_to_output = [ 'N|e|w|s' ]
types_to_output = [ 'News|Legal Notice' ]
types_to_output = [ 'News|Marriage Announcement' ]
types_to_output = [ 'News|Military/War News' ]
types_to_output = [ 'O|b|i|t|u|a|r|y' ]
types_to_output = [ 'R|e|v|i|e|w' ]
types_to_output = [ 'S|t|o|c|k| |Q|u|o|t|e' ]
types_to_output = [ 'Table of Contents|Front Matter' ]
types_to_output = [ 'Table Of Contents|Front Matter' ]
types_to_output = [ 'U|n|d|e|f|i|n|e|d' ]
# declare variables
xml_file_path_list = None
xml_file_path_count = None
xml_file_path_example_list = None
xml_file_path = None
xml_file = None
xml_dict = None
xml_string = None
# loop over types
for object_type in types_to_output:
    
    # print type and count
    xml_file_path_list = object_type_to_file_path_map.get( object_type, [] )
    xml_file_path_count = len( xml_file_path_list )
    xml_file_path_example_list = xml_file_path_list[ : 10 ]
    print( "\n- {} - {} files:".format( object_type, xml_file_path_count ) )
    for xml_file_path in xml_file_path_example_list:
        
        print( "----> {}".format( xml_file_path ) )
        # try to parse the file
        with open( xml_file_path ) as xml_file:
            # parse XML
            xml_dict = xmltodict.parse( xml_file.read() )
            
        #-- END with open( xml_file_path ) as xml_file: --#
            
        # pretty-print
        xml_string = xmltodict.unparse( xml_dict, pretty = True )
        # output
        print( xml_string )
        
    #-- END loop over example file paths. --#
    
#-- END loop over object types. --#
Explanation: explore all known object types
Back to Table of Contents
Look at all known object types to see which contain actual news content.
End of explanation
# directory to work in.
uncompressed_archive_folder = "Newsday_20171006231925_00050"
uncompressed_archive_path = "{}/{}".format( uncompressed_paper_path, uncompressed_archive_folder )
print( 'Uncompressed archive folder: {}'.format( uncompressed_archive_path ) )
# build map of file types to lists of files of that type in specified folder.
object_type_to_file_path_map = my_paper.map_archive_folder_files_to_types( uncompressed_archive_path )
# which types do we want to preview?
types_to_output = news_object_type_list
# declare variables
xml_file_path_list = None
xml_file_path_count = None
xml_file_path_example_list = None
xml_file_path = None
xml_file = None
xml_dict = None
xml_string = None
# loop over types
for object_type in types_to_output:
    
    # print type and count
    xml_file_path_list = object_type_to_file_path_map.get( object_type, [] )
    xml_file_path_count = len( xml_file_path_list )
    xml_file_path_example_list = xml_file_path_list[ : 10 ]
    print( "\n- {} - {} files:".format( object_type, xml_file_path_count ) )
    for xml_file_path in xml_file_path_example_list:
        
        print( "----> {}".format( xml_file_path ) )
        # try to parse the file
        with open( xml_file_path ) as xml_file:
            # parse XML
            xml_dict = xmltodict.parse( xml_file.read() )
            
        #-- END with open( xml_file_path ) as xml_file: --#
            
        # pretty-print
        xml_string = xmltodict.unparse( xml_dict, pretty = True )
        # output
        print( xml_string )
        
    #-- END loop over example file paths. --#
    
#-- END loop over object types. --#
Explanation: files in archive Newsday_20171006231925_00050 - 1989
Back to Table of Contents
Archive details:
ID: 496
Newspaper: 2 - Newsday - Newsday
archive_identifier: Newsday_20171006231925_00050
min_date: 1941-05-06
max_date: 1989-12-30
path: /mnt/hgfs/projects/phd/proquest_hnp/uncompressed/Newsday/Newsday_20171006231925_00050
End of explanation
<END_TASK>