<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Clustering experiments
I hope that by interrogating various ways of looking at the newspaper text placement and the way it is aligned on a page, that some sort of grouping might surface. From the selection of poetry, it seems that a poem is likely to have an aligned left edge to the text, but a more wildly varying left edge.
'clustering.py' can create a database of vectors for a given date range slice of the (readable) Burney newspaper archive. This vector can then be used to investigate various coorelations to see if, in fact, it is possible to cluster the text columns in such a way that poetry is very likely to be found near each other.
Further to this, one we have a means of creating interesting clusters of text, we can ask it about other data and find out which cluster it would put the new data. If we find a cluster that is by majority poetry, then if it puts new data into this cluster, we can have a level of confidence that the new data is also like these and a poem.
Plan
Step1:  What do these 'vectors' look like? What do the columns refer to?
Step2:  Going from a vector back to the metadata reference
Step3:  Initial data woes
There was a considerable discrepancy between the x1 average indent and the column "box" left edge. Looking at the data, the presence of a few outliers can really affect this value. Omitting the 2 smallest and largest x values might be enough to avoid this biasing the sample too badly.
Also, the initial 'drift correction' (adjustments made to correct warped or curved columns) seemed to add more issues than it solved, so the dataset was remade without it.
Step4:  Seems the elbow is quite wide and not sharply defined, based on just the line variences. Let's see what it looks like in general.
Step5:  x1 vs x2 varience?
What is the rough shape of this data? The varience of x1 and x2 are equivalent to the left and right alignment of the text varies in a given block of text.
Step6:  Attempting K-Means
What sort of clustering algorithm to employ is actually a good question. K-means can give fairly meaningless responses if the data is of a given sort. Generally, it can be useful but cannot be used blindly. 
Given the data above, it might be a good start however.
Step7:  Interesting!
The lack of really well defined clusters bolstered the "elbow" test above. K-means is likely not put to good use here, with just these two variables.
The left edge of the scatterplot is a region that contains those blocks of text with lines aligned to the left edge of the paper's column, but have some considerable variation to the length of the line.
For example, I'd expect text looking like the following
Step8:  How about the area density? In other words, what does it look like if the total area of the block is compared to the area taken up by just the words themselves?
Step9:  More outliers skewing the results. This time for blocks with nearly zero varience at either end, but a huge amount of letter area attributed to it by the ocr, but sweeping out a very small overall area. Perhaps mask out the columns which aren't actually columns but dividers mistaken for text? ie skip all blocks that are narrow under 100px perhaps. Another way might be to ignore blocks which are under approximately 40 words (40 words * 5 characters)
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import mpld3
mpld3.enable_notebook()
# Get the dataset:
from clustering import create_cluster_dataset, NewspaperArchive
DBFILE = "1749_1750_no_drift.db"
n = NewspaperArchive()
ds = create_cluster_dataset(n, daterange = [1749, 1750], dbfile = DBFILE)
Explanation: Clustering experiments
I hope that by interrogating various ways of looking at the newspaper text placement and the way it is aligned on a page, that some sort of grouping might surface. From the selection of poetry, it seems that a poem is likely to have an aligned left edge to the text, but a more wildly varying left edge.
'clustering.py' can create a database of vectors for a given date range slice of the (readable) Burney newspaper archive. This vector can then be used to investigate various coorelations to see if, in fact, it is possible to cluster the text columns in such a way that poetry is very likely to be found near each other.
Further to this, one we have a means of creating interesting clusters of text, we can ask it about other data and find out which cluster it would put the new data. If we find a cluster that is by majority poetry, then if it puts new data into this cluster, we can have a level of confidence that the new data is also like these and a poem.
Plan:
Iterate through the following steps:
Pull or derive a set of interesting types of numbers from the dataset. Each block of text will have a set of these numbers (a 'vector').
Create a suitable number of clusters using two (though hopefully more) of these types to test.
Check to see if these clusters are sensible and are not arbitrary in nature subjectively.
Given the set of found poems, see into which clusters the poems get assigned.
If a high % of the poems get assigned to a single cluster -> Success! Focus on this!
Otherwise, try again from the top.
End of explanation
data, transform, id_list = ds
print(data.toarray())
print(transform.get_feature_names())
Explanation: What do these 'vectors' look like? What do the columns refer to?
End of explanation
from clustering import ClusterDB
db = ClusterDB(DBFILE)
print(dict(db.vecidtoitem(id_list[-1])))
print(data.toarray()[-1])
from burney_data import BurneyDB
bdb = BurneyDB("burney.db")
bdb.get_title_row(titleAbbreviation="B0574REMEMBRA")
Explanation: Going from a vector back to the metadata reference:
By keeping an 'id_list', we can look up the identifier for any vector in the list from the database we've made for this clustering attemp. This lets us look up what the reference for that is, and where we can find it:
End of explanation
from scipy import cluster
from matplotlib import pyplot as plt
import numpy as np
# Where is the K-means 'elbow'?
# Try between 1 and 10
# use only the x1 and x2 variences
vset = [cluster.vq.kmeans(data.toarray()[:, [3,6]], i) for i in range(1,10)]
plt.plot([v for (c,v) in vset])
plt.show()
Explanation: Initial data woes
There was a considerable discrepancy between the x1 average indent and the column "box" left edge. Looking at the data, the presence of a few outliers can really affect this value. Omitting the 2 smallest and largest x values might be enough to avoid this biasing the sample too badly.
Also, the initial 'drift correction' (adjustments made to correct warped or curved columns) seemed to add more issues than it solved, so the dataset was remade without it.
End of explanation
# Mask off leaving just the front and end variance columns
npdata = data.toarray()
mask = np.ones((8), dtype=bool)
mask[[0,1,2,4,5,7]] = False
marray = npdata[:,mask]
Explanation: Seems the elbow is quite wide and not sharply defined, based on just the line variences. Let's see what it looks like in general.
End of explanation
plt.scatter(marray[:,0], marray[:,1])
plt.show()
Explanation: x1 vs x2 varience?
What is the rough shape of this data? The varience of x1 and x2 are equivalent to the left and right alignment of the text varies in a given block of text.
End of explanation
#trying a different KMeans
from sklearn.cluster import KMeans
estimators = {'k_means_3': KMeans(n_clusters=3),
              'k_means_5': KMeans(n_clusters=5),
              'k_means_8': KMeans(n_clusters=8),}
fignum = 1
for name, est in estimators.items():
    fig = plt.figure(fignum, figsize=(8, 8))
    plt.clf()
    plt.cla()
    est.fit(marray)
    labels = est.labels_
    
    plt.scatter(marray[:,0], marray[:,1], c=labels.astype(np.float))
    fignum = fignum + 1
plt.show()
Explanation: Attempting K-Means
What sort of clustering algorithm to employ is actually a good question. K-means can give fairly meaningless responses if the data is of a given sort. Generally, it can be useful but cannot be used blindly. 
Given the data above, it might be a good start however.
End of explanation
mpld3.disable_notebook()  # switch off the interactive graph functionality which doesn't work well with the 3D library
from mpl_toolkits.mplot3d import Axes3D
X = npdata[:, [3,5,6]]
fignum = 1
for name, est in estimators.items():
    fig = plt.figure(fignum, figsize=(8, 8))
    plt.clf()
    ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=5, azim=30)
    plt.cla()
    est.fit(X)
    labels = est.labels_
    
    ax.scatter(X[:,0], X[:,2], X[:,1], c=labels.astype(np.float))
    
    ax.set_xlabel('x1 varience')
    ax.set_ylabel('x2 varience')
    ax.set_zlabel('Average indent')
    
    fignum = fignum + 1
plt.show()
Explanation: Interesting!
The lack of really well defined clusters bolstered the "elbow" test above. K-means is likely not put to good use here, with just these two variables.
The left edge of the scatterplot is a region that contains those blocks of text with lines aligned to the left edge of the paper's column, but have some considerable variation to the length of the line.
For example, I'd expect text looking like the following:
Qui quis at ex voluptatibus cupiditate quod quia. 
Quas fuga quasi sit mollitia quos atque. Saepe atque officia sed dolorem. 
Numquam quas aperiam eaque nam sunt itaque est. Sed expedita 
maxime fugiat mollitia error necessitatibus quam soluta. Amet laborum eius
sequi quae sit sit.
This is promising (as long as the data is realistic and there isn't a bug in generating that...)
Now, I wonder if including the "margin" (x1ave-ledge: average x1 coordinate minus the leftmost edge) might help find or distinguish these further?
End of explanation
X = npdata[:, [3,0,6]]
fignum = 1
for name, est in estimators.items():
    fig = plt.figure(fignum, figsize=(8, 8))
    plt.clf()
    ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=25, azim=40)
    plt.cla()
    est.fit(X)
    labels = est.labels_
    
    ax.scatter(X[:,0], X[:,2], X[:,1], c=labels.astype(np.float))
    
    ax.set_xlabel('x1 varience')
    ax.set_ylabel('x2 varience')
    ax.set_zlabel('Density')
    
    fignum = fignum + 1
plt.show()
Explanation: How about the area density? In other words, what does it look like if the total area of the block is compared to the area taken up by just the words themselves?
End of explanation
mask = npdata[:,1] > 40 * 5  # mask based on the ltcount value
print(mask)
print("Amount of vectors: {0}, Vectors with ltcount < 50: {1}".format(len(npdata), sum([1 for item in mask if item == False])))
m_npdata = npdata[mask, :]
X = m_npdata[:, [3,0,6]]
# Let's just plot one graph to see:
est = estimators['k_means_8']
fig = plt.figure(fignum, figsize=(8, 8))
plt.clf()
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=25, azim=40)
plt.cla()
est.fit(X)
labels = est.labels_
    
ax.scatter(X[:,0], X[:,2], X[:,1], c=labels.astype(np.float))
    
ax.set_xlabel('x1 varience')
ax.set_ylabel('x2 varience')
ax.set_zlabel('Density')
    
plt.show()
Explanation: More outliers skewing the results. This time for blocks with nearly zero varience at either end, but a huge amount of letter area attributed to it by the ocr, but sweeping out a very small overall area. Perhaps mask out the columns which aren't actually columns but dividers mistaken for text? ie skip all blocks that are narrow under 100px perhaps. Another way might be to ignore blocks which are under approximately 40 words (40 words * 5 characters)
End of explanation
<END_TASK>