<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step3:  Brand New Concordance
A quick and dirty way of building a concordance
Step4:  Testing and learning stuff
Step5:  A result in a resultset refers to a recordStore
(which in the case of the chapter-idx are chapters)
Step6:  The occurences value is wrong, from what I can gather.
Step7:  The values in proxInfo are not sorted
Step8:  From the result one can get an actual result
which we can use to get_xml (as a string)
or get_dom (an XML tree).
Step9:  Types of XPATH queries needed for the concordance
Step10:  If one has the eid and the character ofset (which
is also part of the w elements!) one can the actual
word that was a hit, but one can also go a bit further
and get the sentence and paragraphs id's.
Step11:  Three ways to get the book, chapter and sentence id
Step12:  If one creates a smaller tree, the computations 
are more efficient.
Step16:  Searching for fog
Step21:  Specs / Questions / Todos
what if the search terms are more complex? how do you then do their len()?
handle quotes etc.
doing pagination
adding query builder
highlighting in a form if a word is frequent and it will thus take some time
searches for more than a word (either a phrase or an or search)
a transformer somewhere?
In quotes
<ASSISTANT_TASK:>
Python Code:
# coding: utf-8
import os
from cheshire3.baseObjects import Session
from cheshire3.document import StringDocument
from cheshire3.internal import cheshire3Root
from cheshire3.server import SimpleServer   
session = Session()
session.database = 'db_dickens'
serv = SimpleServer(session, os.path.join(cheshire3Root, 'configs', 'serverConfig.xml'))
db = serv.get_object(session, session.database)
qf = db.get_object(session, 'defaultQueryFactory')
resultSetStore = db.get_object(session, 'resultSetStore')
idxStore = db.get_object(session, 'indexStore')
def build_concordance(term, context):
    
    query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/proxinfo c3.chapter-idx all/proxinfo "{}" ).format(term))
    result_set = db.search(session, query)
    concordance = []
        
    for result in result_set:
        record = result.fetch_record(session)
        tree = record.get_dom(session)
        text_nodes = tree.xpath('//txt/text()')
        text_only = ' '.join(sentence for sentence in text_nodes)
            
        for hit in result.proxInfo:
            word_id = hit[0][1]  
            char_location = hit[0][2]
            concordance_line = text_only[char_location - context : char_location + len(term) + context]
            #NOTE in these cases record.process_xpath(session, xpath) is not faster
            #TODO check there is only one result
            #sentence = tree.xpath('/div/descendant::w[%i]/ancestor-or-self::s/@id' % int(word_id + 1))
            #paragraph = tree.xpath('/div/descendant::w[%i]/ancestor-or-self::p/@id' % int(word_id + 1))
            #concordance.append((concordance_line, sentence[0], paragraph[0]))
            concordance.append((hit, concordance_line, text_only, tree))
    
    return concordance
#%timeit -n6 concordance = build_concordance("fog", 25)
concordance = build_concordance("fog", 25)
print len(concordance)
concordance[55][:2]
text = concordance[55][2]
print text[15292:19500]
# is the fourth element the number of w nodes?
from lxml import etree
xmlstring = etree.tostring(concordance[55][3])
snippet = w o="94">home</w><n>.</n></toks></s><s sid="194" id="GE.c15.s194" eid="456"><txt>Beyond town, we found a heavy mist out, and it fell wet and thick.</txt><toks><w o="0">Beyond</w><n> </n><w o="7">town</w><n>,</n><n> </n><w o="13">we</w><n> </n><w o="16">found</w><n> </n><w o="22">a</w><n> </n><w o="24">heavy</w><n> </n><w o="30">mist</w><n> </n><w o="35">out</w><n>,</n><n> </n><w o="40">and</w><n> </n><w o="44">it</w><n> </n><w o="47">fell</w><n> </n><w o="52">wet</w><n> </n><w o="56">and</w><n> </n><w o="60">thick</w><n>.</n></toks></s><s sid="195" id="GE.c15.s195" eid="457"><txt>The turnpike lamp was a blur, quite out of the lamp\'s usual place apparently, and its rays looked solid substance on the fog.</txt><toks><w o="0">The</w><n> </n><w o="4">turnpike</w><n> </n><w o="13">lamp</w><n> </n><w o="18">was</w><n> </n><w o="22">a</w><n> </n><w o="24">blur</w><n>,</n><n> </n><w o="30">quite</w><n> </n><w o="36">out</w><n> </n><w o="40">of</w><n> </n><w o="43">the</w><n> </n><w o="47">lamp\'s</w><n> </n><w o="54">usual</w><n> </n><w o="60">place</w><n> </n><w o="66">apparently</w><n>,</n><n> </n><w o="78">and</w><n> </n><w o="82">its</w><n> </n><w o="86">rays</w><n> </n><w o="91">looked</w><n> </n><w o="98">solid</w><n> </n><w o="104">substance</w><n> </n><w o="114">on</w><n> </n><w o="117">the</w><n> </n><w o="121">fog</w><n>.</n></toks></s><s sid="196" id="GE.c15.s196" eid="458"><txt>We were noticing this, and saying how that the mist rose with a change of wind from a certain quarter of our marshes, when we came upon a man, slouching under the lee of the turnpike house.</txt><toks><w o="0">We</w><n> </n><w o="3">were</w><n> </n><w o="8">noticing</w><n> </n><w o="17">this</w><n>,</n><n> </n><w o="23">and</w><n> </n><w o="27">saying</w><n> </n><w o="34">how</w><n> </n><w o="38">that</w><n> </n><w o="43">the</w><n> </n><w o="47">mist</w><n> </n><w o="52">rose</w><n> </n><w o="57">with</w><n> </n><w o="62">a</w><n> </n><w o="64">change</w><n> </n><w o="71">of</w><n> </n><w o="74">wind</w><n> </n><w o="79">from</w><n> </n><w o="84">a</w><n> </n><w o="86">certain</w><n> </n><w o="94">quarter</w><n> </n><w o="102">of</w><n> </n><w o="105">our</w><n> </n><w o="109">marshes</w><n>,</n><n> </n><w o="118">when</w><n> </n><w o="123">we</w><n> </n><w o="126">c
print snippet
# idea: use a tokenmerger?
xmlstring
# concordance = build_concordance("lamp", 25)
print len(concordance)
for line in concordance[55:56]:
    print concordance.index(line), line[0], line[1], "########", line[2][:100]
for line in concordance[0:1000]:
    print concordance.index(line)
def build_concordance_with_locations(term, context, max_hits):
    
    query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.chapter-idx = "{}" ).format(term))
    result_set = db.search(session, query)
    concordance = []
    
    count = 0
    
    for result in result_set:
    
        if count < max_hits:
            record = result.fetch_record(session)
            tree = record.get_dom(session)
            text_nodes = tree.xpath('//txt/text()')
            text_only = ' '.join(sentence for sentence in text_nodes)
            for hit in result.proxInfo:
                if count < max_hits:
                    count +=1 
                
                    word_id = hit[0][1]  
                    char_location = hit[0][2]
                    concordance_line = text_only[char_location - context : char_location + len(term) + context]
                    #NOTE in these cases record.process_xpath(session, xpath) is not faster
                    #TODO check there is only one result
                    sentence = tree.xpath('/div/descendant::w[%i]/ancestor-or-self::s/@id' % int(word_id + 1))
                    paragraph = tree.xpath('/div/descendant::w[%i]/ancestor-or-self::p/@id' % int(word_id + 1))
                    concordance.append((concordance_line, sentence[0], paragraph[0]))
    
    return concordance
%timeit build_concordance_with_locations("fog", 25, 100)
%timeit build_concordance_with_locations("the", 25, 100)
%timeit build_concordance_with_locations("the", 25, 1000)
%prun build_concordance_with_locations("the", 25)
%timeit build_concordance_with_locations("the", 25, 10000)
%timeit build_concordance_with_locations("the", 25, 100000)
%timeit -n1 concordance = build_concordance_with_locations("the", 25, 1000000)
concordance = build_concordance_with_locations("dense fog", 25, 1000)
len(concordance)
for line in concordance:
    print line
Explanation: Brand New Concordance
A quick and dirty way of building a concordance
End of explanation
# coding: utf-8
import os
from cheshire3.baseObjects import Session
from cheshire3.document import StringDocument
from cheshire3.internal import cheshire3Root
from cheshire3.server import SimpleServer   
session = Session()
session.database = 'db_dickens'
serv = SimpleServer(session, os.path.join(cheshire3Root, 'configs', 'serverConfig.xml'))
db = serv.get_object(session, session.database)
qf = db.get_object(session, 'defaultQueryFactory')
resultSetStore = db.get_object(session, 'resultSetStore')
idxStore = db.get_object(session, 'indexStore')
query = qf.get_query(session, '(c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.chapter-idx any "fog")')
result_set = db.search(session, query)
Explanation: Testing and learning stuff
End of explanation
result = result_set[0]
print result
Explanation: A result in a resultset refers to a recordStore
(which in the case of the chapter-idx are chapters)
End of explanation
result.occurences
Explanation: The occurences value is wrong, from what I can gather.
End of explanation
result.proxInfo[:15]
Explanation: The values in proxInfo are not sorted
End of explanation
record = result.fetch_record(session)
record.byteCount
?record.fetch_proxVector(session)
record.get_xml(session)
tree = record.get_dom(session)
Explanation: From the result one can get an actual result
which we can use to get_xml (as a string)
or get_dom (an XML tree).
End of explanation
%timeit tree.xpath("//txt/text()")
one = tree.xpath("//txt/text()")
len(one)
%timeit tree.xpath("/div/p/s/txt/text()")
two = tree.xpath("/div/p/s/txt/text()")
len(two)
Explanation: Types of XPATH queries needed for the concordance
End of explanation
tree.xpath("/div/p/s[@eid=183]/toks/w[@o=215]/text()")
tree.xpath("/div/p/s[@eid=183]/toks")
tree.xpath("/div/p/s[@eid=183]/toks/w/text()")
tree.xpath("/div/p/self::s[@eid=183]/toks/w[@o=215]")
tree.xpath("/div/p/s/toks/w[@o=215]")
Explanation: If one has the eid and the character ofset (which
is also part of the w elements!) one can the actual
word that was a hit, but one can also go a bit further
and get the sentence and paragraphs id's.
End of explanation
%timeit tree.xpath("//p/s[@eid=183]/@id")
tree.xpath("//p/s[@eid=183]/@id")
%timeit tree.xpath("//*[@eid=183]/@id")
# see how slow this is!
tree.xpath("//*[@eid=183]/@id")
%timeit tree.xpath("//s[@eid=183]/@id")
tree.xpath("//s[@eid=183]/@id")
Explanation: Three ways to get the book, chapter and sentence id:
End of explanation
s = tree.xpath("//p/s[@eid=%i]" % 183)
s
print s[0]
s[0].xpath("self::s/@id")
s[0].xpath("attribute::id")
%timeit tree.xpath("//s[@eid=183]/ancestor::p/@id")
tree.xpath("//s[@eid=183]/ancestor::p/@id")
%timeit s[0].xpath("ancestor::p/@id")
s[0].xpath("ancestor::p/@id")
sentence = tree.xpath("//p/s[@eid=%i]" % 1)    # format: ['BH.c1.s93']
sentence
Explanation: If one creates a smaller tree, the computations 
are more efficient.
End of explanation
query = qf.get_query(session, '(c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.chapter-idx any "fog")')
result_set = db.search(session, query)
result = result_set[0]
proxinfo = result.proxInfo[:15]
from pprint import pprint  # to "prettry print" a nested list nicely
pprint(proxinfo)
record.get_xml(session)[6409:8000]
tree = record.get_dom(session)
prox1 = proxinfo[0][0][0]
prox2 = proxinfo[0][0][1]
prox3 = proxinfo[0][0][2]
prox4 = proxinfo[0][0][3]
for prox in (prox1, prox2, prox3, prox4): print prox
# output is a boolean
tree.xpath('/div/p/s/toks/w/@o=%i' % prox1)
w = tree.xpath('/div/p/s/toks/w[@o=%i]/text()' % prox1)
print w
# o = 0 in this case; this is interesting to see
# quickly what words sentences start with. 
w = tree.xpath('/div/descendant::w[%i]' % prox2)
print w
w = tree.xpath('/div/descendant::w[%i]/text()' % int(prox2 + 1))
print w
tree.xpath('/div/descendant::w[%i]/text()' % int(prox2 + 1))
tree.xpath('/div/descendant::w[%i]/ancestor-or-self::s/@id' % int(prox2 + 1))
tree.xpath('/div/descendant::w[%i]/ancestor-or-self::p/@id' % int(prox2 + 1))
#TODO time
record.process_xpath(session, '//*[@eid="%d"]/following::w[%d+1]/ancestor-or-self::s' % (prox1, prox2))
def build_concordance_with_location(term, context):
    
    query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.chapter-idx any "{}" ).format(term))
    result_set = db.search(session, query)
    concordance = []
        
    for result in result_set:
        record = result.fetch_record(session)
        tree = record.get_dom(session)
        text_nodes = tree.xpath('//txt/text()')
        text_only = ' '.join(sentence for sentence in text_nodes)
            
        for hit in result.proxInfo:
            word_id = hit[0][1]  
            char_location = hit[0][2]
            concordance_line = text_only[char_location - context : char_location + len(term) + context]
            #NOTE in these cases record.process_xpath(session, xpath) is not faster
            #TODO check there is only one result
            sentence = tree.xpath('/div/descendant::w[%i]/ancestor-or-self::s/@id' % int(word_id + 1))
            paragraph = tree.xpath('/div/descendant::w[%i]/ancestor-or-self::p/@id' % int(word_id + 1))
            concordance.append((concordance_line, sentence[0], paragraph[0]))
    
    return concordance
%timeit -n1 concordance = build_concordance_with_locations("the", 25, 1000000)
# coding: utf-8
import os
from cheshire3.baseObjects import Session
from cheshire3.document import StringDocument
from cheshire3.internal import cheshire3Root
from cheshire3.server import SimpleServer   
session = Session()
session.database = 'db_dickens'
serv = SimpleServer(session, os.path.join(cheshire3Root, 'configs', 'serverConfig.xml'))
db = serv.get_object(session, session.database)
qf = db.get_object(session, 'defaultQueryFactory')
resultSetStore = db.get_object(session, 'resultSetStore')
idxStore = db.get_object(session, 'indexStore')
%timeit build_concordance_with_locations("the", 25, 100000)
term = 'fog'
(c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.chapter-idx any "{}" ).format(term)
query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.chapter-idx any "{}" ).format(term))
result_set = db.search(session, query)
len(result_set)
#build_concordance(result_set, 'we', 25)
# %timeit build_concordance(result_set, 'the', 25)
# option 1: get chapter as string
# option 2: get sentence as string with location info as well.
#TODO implement eid for quotes and suspensions
#TODO implement start and end result for pagination
the_location_dickens = []
def test():
    for result in result_set:
        proxinfo = result.proxInfo
        record = result.fetch_record(session)
        tree = record.get_dom(session)
        for hit in proxinfo:
            word_location = hit[0][1]
            char_location = hit[0][2]
            word_location = tree.xpath("/div/p/s/toks/w[@o=215]")
            # word_location = tree.xpath('//w[%i]' % word_location)
            the_location_dickens.append(word_location)
            #location = tree.xpath('//w[word_location]')
            #the_in_dickens.append([concordance_line, location])
#%timeit test()
test()
len(the_location_dickens)
# option 1: merge all txt together and then do xpath to get the location
# option 2: get txt with location info as well.
for result in result_set:
    proxinfo = result.proxInfo
    record = result.fetch_record(session)
    tree = record.get_dom(session)
    tree.xpath('//w[1]')
the_in_dickens = []
for result in result_set:
    proxinfo = result.proxInfo
    record = result.fetch_record(session)
    tree = record.get_dom(session)
    raw_tree = tree.xpath('//txt/text()')
    raw_str = ' '.join(sentence for sentence in raw_tree)
    for hit in proxinfo:
        char_location = hit[0][2]
        concordance_line = raw_str[char_location-20:char_location+len("the")+20]
        the_in_dickens.append(concordance_line)
for result in result_set:
    proxinfo = result.proxInfo
    record = result.fetch_record(session)
    tree = record.get_dom(session)
    raw_tree = tree.xpath('//txt/text()')
    raw_str = ' '.join(sentence for sentence in raw_tree)
    for hit in proxinfo:
        char_location = hit[0][2]
        print raw_str[char_location-20:char_location+len("the")+20]
        
for hit in proxinfo:
    char_location = hit[0][2]
    print raw_str[char_location-20:char_location+len("the")+20]        
Explanation: Searching for fog
End of explanation
def build_concordance_with_locations(term, idx, context, max_hits):
    
    query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.{} any/proxinfo "{}" ).format(idx, term))
    result_set = db.search(session, query)
    concordance = []
    
    count = 0
    
    for result in result_set:
    
        if count < max_hits:
            record = result.fetch_record(session)
            tree = record.get_dom(session)
            text_nodes = tree.xpath('//txt/text()')
            text_only = ' '.join(sentence for sentence in text_nodes)
            for hit in result.proxInfo:
                if count < max_hits:
                    count +=1 
                
                    element_id = hit[0][0]
                    word_id = hit[0][1]  
                    char_location = hit[0][2]
                    
                    ## the eid only differs from 0 for indexes other than the chapter index
                    if element_id:
                        el = tree.xpath('//*[@eid="{}"]'.format(element_id))[0]
                        el_char_offset = el.get('offset')
                        word_id = el.get('wordOffset')
                        char_location = int(el_char_offset) + char_location - 1
                    
                    concordance_line = text_only[char_location - context : char_location + len(term) + context]
                    #NOTE in these cases record.process_xpath(session, xpath) is not faster
                    #TODO check there is only one result
                    sentence = tree.xpath('/div/descendant::w[{}]/ancestor-or-self::s/@id'.format(int(word_id) + 1))
                    paragraph = tree.xpath('/div/descendant::w[{}]/ancestor-or-self::p/@id'.format(int(word_id) + 1))
                    concordance.append((concordance_line, sentence[0], paragraph[0]))
    
    return concordance
concordance = build_concordance_with_locations("fog", "quote-idx", 25, 100)
for line in concordance:
    print line[0]
build_concordance_with_locations("fog", "quote-idx", 25, 100)
build_concordance_with_locations("fog", "non-quote-idx", 25, 100)
query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.{} any "{}" ).format("quote-idx", "fog"))
result_set = db.search(session, query)
for result in result_set:
    print result.proxInfo
one_result = result_set[0]
one_fetched_result = one_result.fetch_record(session)
dom = one_fetched_result.get_dom(session)
qs = dom.xpath('//*[@eid="1"]')
qs[0].attrib
dom_text_nodes = dom.xpath('//txt/text()')
dom_text_only = ' '.join(sentence for sentence in dom_text_nodes)
dom_text_only[30639+2:30700]
second_result = result_set[1]
print second_result.proxInfo
second_fetched_result = second_result.fetch_record(session)
second_dom = second_fetched_result.get_dom(session)
second_qs = second_dom.xpath('//qs[@eid="94"]')[0]
second_qs.attrib
second_dom_text_nodes = second_dom.xpath('//txt/text()')
second_dom_text_only = ' '.join(sentence for sentence in second_dom_text_nodes)
second_dom_text_only[3259+4:3400]
query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.{} any "{}" ).format("chapter-idx", "fog"))
result_set = db.search(session, query)
for result in result_set:
    print result.proxInfo
query = qf.get_query(session, (c3.subcorpus-idx all "dickens" and/cql.proxinfo c3.{} = "{}" ).format("chapter-idx", "substance on the fog"))
result_set = db.search(session, query)
print len(result_set)
for result in result_set:
    print result.proxInfo
# record.get_xml(session)
result = result_set[0]
result = result.fetch_record(session)
record = result.get_xml(session)
record.find('fog')
record[143708:143800]
type(record)
dom = result.get_dom(session)
dom.xpath('//w[@o="19257"]')
xml_to_clean_txt(dom)[19257:19500]
def xml_to_clean_txt(xmltree):
    text_nodes = xmltree.xpath('//txt/text()')
    text_only = ' '.join(sentence for sentence in text_nodes)
    return text_only  
xml_to_clean_txt(dom)[19000
                      :19500]
Explanation: Specs / Questions / Todos
what if the search terms are more complex? how do you then do their len()?
handle quotes etc.
doing pagination
adding query builder
highlighting in a form if a word is frequent and it will thus take some time
searches for more than a word (either a phrase or an or search)
a transformer somewhere?
In quotes
End of explanation
<END_TASK>