<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 MAZ176 statistics
Author
Step1:  Number of Connectors
Step2:  Number of Connectors by Connector Type
Step3:  Number of Connectors by Connector Token
Step4:  Number of Connectors by Connector POS
Step5:  Total number of connectives
Step6:  Number of RST relations by type
Step7:  RST relations by type (incl. generic 'span' relation)
Step8:  RST relations by type (excl. generic 'span' relation)
Step9:  Total number of RST relations (incl. generic 'span' relation)
Step10:  Total number of RST relations (excl. generic 'span' relation)
Step11:  Coreference counts
Step12:  Number of binary coreference relations
Step13:  Total number of binary coreference relations
Step14:  Number of binary coreference relations by subtype
Step15:  Number of coreference chains
Please note that the number of coreference chains is somewhat skewed.
MMAX allows markables to point to more than one markable/target/antecedent.
This features was used when MAZ176 was annotated.
As an example, consider a coreference chain in which A points to B and B points
to both C and D (i.e. A -&gt; B -&gt; {C, D}). The algorithm used here will interpret this as two separate
coreference chains (A-B-C and A-B-D).
cf. https
Step16:  Number of connected coreference components
To avoid the abovementioned problem, we can calculate the
number of connected components from a graph constructed using all
binary coreference relations. In this setting, A -&gt; B -&gt; {C, D}
will be counted as one connected component.
Step17:  Reproducability information
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import os
from collections import Counter
from operator import itemgetter
import pandas as pd
from networkx import Graph
from networkx.algorithms.components.connected import connected_components
from discoursegraphs import select_edges_by, get_pointing_chains
from discoursegraphs.readwrite import ConanoDocumentGraph, MMAXDocumentGraph, RSTGraph, TigerDocumentGraph
MAZ_ROOTDIR = os.path.expanduser('~/repos/pcc-annis-merged/maz176/')
Explanation: MAZ176 statistics
Author: Arne Neumann
Date: 2014-10-13
End of explanation
CONANO_DIR = os.path.join(MAZ_ROOTDIR, 'connectors')
conano_files = !ls $CONANO_DIR/*.xml
Explanation: Number of Connectors
End of explanation
all_connectors = Counter()
for conano_file in conano_files:
    cdg = ConanoDocumentGraph(conano_file)
    for token_id in cdg.tokens:
        if 'conano:connective' in cdg.node[token_id]['layers']:
            all_connectors[cdg.node[token_id]['relation']] += 1
connector_counts = sorted(all_connectors.iteritems(), key=itemgetter(1), reverse=True)
connector_df = pd.DataFrame(connector_counts, columns=['connectors', 'counts'])
connector_df
connector_df.plot(kind='bar', x='connectors', y='counts')
Explanation: Number of Connectors by Connector Type
End of explanation
connector_tokens = Counter()
for conano_file in conano_files:
    cdg = ConanoDocumentGraph(conano_file)
    for token_id in cdg.tokens:
        if 'conano:connective' in cdg.node[token_id]['layers']:
            # count tokens normalized to lower case
            connector_tokens[cdg.get_token(token_id).lower()] += 1
connector_token_counts = sorted(connector_tokens.iteritems(), key=itemgetter(1), reverse=True)
connector_token_df = pd.DataFrame(connector_token_counts, columns=['connectors', 'counts'])
connector_token_df
import matplotlib.pylab as pylab
pylab.rcParams['figure.figsize'] = 15, 5
connector_token_df[connector_token_df.counts > 4].plot(kind='bar', x='connectors', title='Connector tokens occurring five times or more')
Explanation: Number of Connectors by Connector Token
End of explanation
TIGER_DIR = os.path.join(MAZ_ROOTDIR, 'syntax')
connector_pos = Counter()
for conano_file in conano_files:
    cdg = ConanoDocumentGraph(conano_file)
    maz_id = os.path.basename(conano_file).split('.')[0]
    tdg = TigerDocumentGraph(os.path.join(TIGER_DIR, maz_id+'.xml'))
    cdg.merge_graphs(tdg)
    for token_id in cdg.tokens:
        if 'conano:connective' in cdg.node[token_id]['layers']:
            connector_pos[cdg.node[token_id]['tiger:pos']] += 1
connector_pos_counts = sorted(connector_pos.iteritems(), key=itemgetter(1), reverse=True)
connector_pos_df = pd.DataFrame(connector_pos_counts, columns=['connector-pos', 'counts'])
connector_pos_df
pylab.rcParams['figure.figsize'] = 10, 5
connector_pos_df.plot(kind='bar', x='connector-pos', y='counts')
Explanation: Number of Connectors by Connector POS
End of explanation
sum(connector_df['counts'])
Explanation: Total number of connectives
End of explanation
RST_DIR = os.path.join(MAZ_ROOTDIR, 'rst')
rst_files = !ls $RST_DIR/*.rs3
all_rst_relations = Counter()
for rst_file in rst_files:
    rdg = RSTGraph(rst_file)
    for source, target, edge_attribs in select_edges_by(rdg, layer='rst:relation', data=True):
        all_rst_relations[edge_attribs['rst:relname']] += 1
rst_relations_count = sorted(all_rst_relations.iteritems(), key=itemgetter(1), reverse=True)
rst_relations_df = pd.DataFrame(rst_relations_count, columns=['RST relations', 'counts'])
rst_relations_df
Explanation: Number of RST relations by type
End of explanation
pylab.rcParams['figure.figsize'] = 15, 5
rst_relations_df.plot(kind='bar', x='RST relations', y='counts')
Explanation: RST relations by type (incl. generic 'span' relation)
End of explanation
rst_relations_df[1:].plot(kind='bar', x='RST relations', y='counts')
Explanation: RST relations by type (excl. generic 'span' relation)
End of explanation
sum(rst_relations_df['counts'])
Explanation: Total number of RST relations (incl. generic 'span' relation)
End of explanation
sum(rst_relations_df['counts'][1:])
Explanation: Total number of RST relations (excl. generic 'span' relation)
End of explanation
COREF_DIR = os.path.join(MAZ_ROOTDIR, 'coreference')
coref_files = !ls $COREF_DIR/*.mmax
Explanation: Coreference counts
End of explanation
binary_coref_relations = 0
binary_coref_relation_types = Counter()
for coref_file in coref_files:
    mdg = MMAXDocumentGraph(coref_file)
    for source, target, edge_attribs in select_edges_by(mdg, edge_type='points_to', data=True):
        binary_coref_relations += 1
        binary_coref_relation_types[edge_attribs['label']] += 1
Explanation: Number of binary coreference relations
End of explanation
binary_coref_relations
Explanation: Total number of binary coreference relations
End of explanation
sorted(binary_coref_relation_types.iteritems(), key=itemgetter(1), reverse=True)
Explanation: Number of binary coreference relations by subtype
End of explanation
coref_chains = 0
for coref_file in coref_files:
    mdg = MMAXDocumentGraph(coref_file)
    coref_chains += len(get_pointing_chains(mdg))
print coref_chains
Explanation: Number of coreference chains
Please note that the number of coreference chains is somewhat skewed.
MMAX allows markables to point to more than one markable/target/antecedent.
This features was used when MAZ176 was annotated.
As an example, consider a coreference chain in which A points to B and B points
to both C and D (i.e. A -&gt; B -&gt; {C, D}). The algorithm used here will interpret this as two separate
coreference chains (A-B-C and A-B-D).
cf. https://github.com/arne-cl/discoursegraphs/issues/40
End of explanation
from networkx import Graph
from networkx.algorithms.components.connected import connected_components
coref_connected_components = 0
for coref_file in coref_files:
    coref_digraph = Graph()
    mdg = MMAXDocumentGraph(coref_file)
    for source, target, edge_attribs in select_edges_by(mdg, edge_type='points_to', data=True):
        coref_digraph.add_edge(source, target, edge_attribs)
    coref_connected_components += len(list(connected_components(coref_digraph)))
    
print coref_connected_components
Explanation: Number of connected coreference components
To avoid the abovementioned problem, we can calculate the
number of connected components from a graph constructed using all
binary coreference relations. In this setting, A -&gt; B -&gt; {C, D}
will be counted as one connected component.
End of explanation
%load_ext version_information
%version_information networkx, pandas
Explanation: Reproducability information
End of explanation
<END_TASK>