<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data
Both datasets are text collections from this site. 
TCP-ECCO (170mb uncompressed) can be downloaded here
Lincoln (700kb uncompressed) can be downloaded here
Step1:  Intialize swhoosh index
Step2:  Add documents to swhoosh index
Step3:  Save and close, then reload swoosh index
Note that the index MUST be saved -- it does not autosave!
Step4:  Wrap Index in an IndexReader and get statistics needed for BM25
Step5:  Get term info and postings for 'lincoln'
Step6:  Run a BM25 search with Whoosh API
Step7:  Compare results to Whoosh
Step8:  Note
Step9:  And repeating with multiprocessing enabled
Step10:  Comparing to whoosh default
Step11:  And to whoosh with multiprocessing enabled
Step12:  Again, the matcher results are the same
Step14:  Query Performance (BM25)
To benchmark this, we extrac token's from one of lincoln's speechs (not in TCP-ECCO), and select queries at random from the resulting ~5000 tokens.
Step15:  3 word queries
Step16:  6 word queries
Step17:  30 word queries
<ASSISTANT_TASK:>
Python Code:
def get_lincoln():
    for filepath in sorted(glob.glob('Lincoln/*.txt')):
        with open(filepath, 'r', encoding='latin') as f:
            doc = f.read()
        yield {'filepath': filepath, 'doc': doc}
        
def get_TCP():
    for filepath in sorted(glob.glob('TCP-ECCO/*.txt')):
        with open(filepath, 'r', encoding='latin') as f:
            doc = f.read()
        yield {'filepath': filepath, 'doc': doc}
Explanation: Data
Both datasets are text collections from this site. 
TCP-ECCO (170mb uncompressed) can be downloaded here
Lincoln (700kb uncompressed) can be downloaded here
End of explanation
s = swhoosh.Index('randomIdx/randomIdx', simple_schema(), reset=True)
s.load()
Explanation: Intialize swhoosh index
End of explanation
t = time.time()
s.add_documents(get_lincoln())
print("TIME:", time.time() - t)
Explanation: Add documents to swhoosh index
End of explanation
s.save_and_close()
with open('randomIdx/randomIdx.manager', 'rb') as f:
    s = pickle.load(f)
    s.load()
Explanation: Save and close, then reload swoosh index
Note that the index MUST be saved -- it does not autosave!
End of explanation
r = s.reader()
print(r.doc_count())
print(r.doc_frequency('doc',b'lincoln'))
print(r.doc_field_length(21, 'doc')) 
print(r.avg_field_length('doc'))
Explanation: Wrap Index in an IndexReader and get statistics needed for BM25
End of explanation
# returns (overall frequency, num docs, start loc in postings file, postings size)
s._idx['doc'].terminfo(b'lincoln')
[swhoosh.postings.load2(a[2], a[1]) for a in s._idx['doc']._postings(b'lincoln')]
# what the postings look like: (docId, frequency, positions)
s._idx['doc']._postings(b'lincoln')
Explanation: Get term info and postings for 'lincoln'
End of explanation
qp = QueryParser("doc", schema=s._schema)
q = qp.parse("lincoln")
with s.searcher() as searcher:
    results = searcher.search(q)
    print(results)
    for hit in results:
        print('{:f}'.format(hit.score), ' | ', hit['filepath'])
Explanation: Run a BM25 search with Whoosh API
End of explanation
def make_clean_index(ix_dirname, paths, procs=1):
    ix = whoosh.index.create_in(ix_dirname, schema=simple_schema())
    writer = ix.writer(procs=procs)
    for filepath in paths:
        add_doc(writer, filepath)
    writer.commit()
    
    return ix
def add_doc(writer, filepath):
    with open(filepath, 'rb') as f:
        text = f.read().decode('latin')
    writer.add_document(doc=text, filepath=filepath)
t = time.time()
ix = make_clean_index('wind', sorted(glob.glob('Lincoln/*.txt')))
print("TIME:", time.time() - t)
with ix.searcher() as searcher:
    results = searcher.search(q)
    print(results)
    for hit in results:
        print('{:f}'.format(hit.score), ' | ', hit['filepath'])
Explanation: Compare results to Whoosh
End of explanation
s = swhoosh.Index('randomIdx2/randomIdx2', simple_schema(), reset=True)
s.load()
t = time.time()
s.add_documents(get_TCP())
print("TIME:", time.time() - t)
Explanation: Note: the BM25 scores returned by whoosh's default settings are a tiny bit smaller because the default whoosh reader adds 1 to the current document length for some reason (I don't think this is correct). 
Indexing a bigger collection
End of explanation
s = swhoosh.Index('randomIdx2/randomIdx2', simple_schema(), reset=True)
s.load()
t = time.time()
s.add_documents_multiprocessing(get_TCP(), num_procs=4)
print("TIME:", time.time() - t)
s.save()
Explanation: And repeating with multiprocessing enabled:
End of explanation
t = time.time()
ix = make_clean_index('wind2', sorted(glob.glob('TCP-ECCO/*.txt')))
print("TIME:", time.time() - t)
Explanation: Comparing to whoosh default:
End of explanation
t = time.time()
ix = make_clean_index('wind2', sorted(glob.glob('TCP-ECCO/*.txt')), procs=4)
print("TIME:", time.time() - t)
Explanation: And to whoosh with multiprocessing enabled:
End of explanation
with s.searcher() as searcher:
    results = searcher.search(q)
    print(results)
    for hit in results:
        print('{:f}'.format(hit.score), ' | ', hit['filepath'])
print('')
with ix.searcher() as searcher:
    results = searcher.search(q)
    print(results)
    for hit in results:
        print('{:f}'.format(hit.score), ' | ', hit['filepath'])
Explanation: Again, the matcher results are the same:
End of explanation
with open('randomIdx2/randomIdx2.manager', 'rb') as f:
    s = pickle.load(f)
    s.load()
ix = whoosh.index.open_dir('wind2')
import numpy as np
s1 = s.searcher()
s2 = ix.searcher()
qp = QueryParser("doc", schema=s._schema)
with open('Lincoln/24-speech-1856.txt', 'r', encoding='latin') as f:
    data = f.read()
query_vocab = [t.text for t in s._schema['doc'].analyzer(data)]
print('Length of query vocab:',len(query_vocab))
def random_n_query(n):
    Generates a random query of length n
    return ' '.join(np.random.choice(query_vocab, size=n))
def benchmark_n_query(n, trials):
    t_swhoosh, t_whoosh = 0, 0 
    for i in range(trials):
        q = qp.parse(random_n_query(n))
        t = time.time()
        results = s1.search(q)
        t_swhoosh += time.time() - t
        t = time.time()
        results = s2.search(q) 
        t_whoosh += time.time() - t
    print('- Swhoosh time per query:', "{:.2f}".format(t_swhoosh / trials * 1000), "ms")
    print('- Whoosh time per query:', "{:.2f}".format(t_whoosh / trials * 1000), "ms")
    return t_swhoosh/trials, t_whoosh/trials
Explanation: Query Performance (BM25)
To benchmark this, we extrac token's from one of lincoln's speechs (not in TCP-ECCO), and select queries at random from the resulting ~5000 tokens.
End of explanation
x, y = benchmark_n_query(3, 100)
print('\nSwhoosh was', "{0:.0f}%".format(100*(y-x)/y), 'percent faster.')
Explanation: 3 word queries
End of explanation
x, y = benchmark_n_query(6, 100)
print('\nSwhoosh was', "{0:.0f}%".format(100*(y-x)/y), 'percent faster.')
Explanation: 6 word queries
End of explanation
x, y = benchmark_n_query(30, 100)
print('\nSwhoosh was', "{0:.0f}%".format(100*(y-x)/y), 'percent faster.')
Explanation: 30 word queries
End of explanation
<END_TASK>