<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Initialization
Step1:  Algorithm
Step2:  Save coefs (X) and Y, along with tests (to know what each row refers to) to work with it later
Step3:  What without NEs
Step4:  SGD
Step5:  Simple trial
Step6:  Run cross validation for each model
Step7:  Scores of each model
Step8:  Final results
Step9:  The one not normalised gets less error due to the fact that bigger absolute values make sigmoid closer to 1 or 0, thus reducing the error. Anyhow, the desired value is the one normalised (values must sum 1)
Step10:  Notice how with not-ne-what What becomes slightly more important (maybe because it doesn't lose accuracy due to NE mistreatment?). Even if the resulting accuracy is the same but more stable, the other system makes What be more easily computed in terms of performance, so it would still be the selected approach, because the algorithm is slow enough already. In next steps, when the algorithm is optimised, working with not-ne-what would be a good idea.
Finally, by checking what would be the values with just Who and Where
<ASSISTANT_TASK:>
Python Code:
folder = os.path.join('..', 'data')
newsbreaker.init(os.path.join(folder, 'topic_model'), 'topic_model.pkl', 'vocab.txt')
entries = load_entries(folder)
entries_dict = defaultdict(list)
for entry in entries:
    entries_dict[entry.feed].append(entry)
client = MongoClient()
db = client.newstagger
Explanation: Initialization
End of explanation
def get_entry(s):
    feedname, index = s.split('|')
    try:
        index = int(index)
    except ValueError:
        raise KeyError('Malformed entry %s' % s)
    
    for feed, l in entries_dict.items():
        if feed.name == feedname:
            for entry in l:
                if entry.index == index:
                    return entry
            else:
                break
    
    raise KeyError('Entry %s not found' % s)
coefs = []
Y = []
tests = list(db.pairs.find())
for test in tests:
    base = get_entry(test['base'])
    e1 = get_entry(test['e1'])
    e2 = get_entry(test['e2'])
    
    coefs.append(
        [
            [
                base.what_distance(e1),
                base.who_distance(e1),
                base.where_distance(e1)
            ],
            [
                base.what_distance(e2),
                base.who_distance(e2),
                base.where_distance(e2)
            ]
        ]
    )
    
    Y.append(float(test['res']))
Explanation: Algorithm
End of explanation
with open('X.txt', 'w') as f:
    f.write('\n'.join(str(x) for x in coefs))
with open('Y.txt', 'w') as f:
    f.write('\n'.join(str(x) for x in Y))
import json
with open('tests.json', 'w') as f:
    f.write(
        json.dumps(
            [
                { k: v for k, v in d.items() if k != '_id' }
                for d in tests
            ], 
            indent=2
        )
    )
Explanation: Save coefs (X) and Y, along with tests (to know what each row refers to) to work with it later
End of explanation
from collections import Counter
def what_without_ne(entry):
    entry.doc(tag=True, parse=False, entity=True)
    
    avoid_ent_cats = set(entry.who_ne_cats)
    avoid_ent_cats.update(entry.where_ne_cats)
    
    avoid_ents = [
        (ent.start, ent.end)
        for ent in entry.doc.ents
        if ent.label_ in avoid_ent_cats
    ]
    
    words = []
    doc_words = list(entry.doc)
    
    while doc_words and avoid_ents:
        i = doc_words[0].i
            
        low, high = avoid_ents[0]
        
        if i < low:
            words.append(doc_words.pop(0))
        elif low <= i and i < high:
            doc_words.pop(0) # but don't save it, since is part of NE
        else: # low < high <= i
            avoid_ents.pop(0) # delete ent, since we overpassed it
            
    words += doc_words # no more ents to filter with
    
    counter = Counter(
        word.lower_
        for word in words
    )
    
    entry._what = entry.topic_model.model.transform(
        np.array([ counter[word] for word in entry.topic_model.vocab ])
    )
not_ne_what_coefs = []
for test in tests:
    base = get_entry(test['base'])
    what_without_ne(base)
    
    e1 = get_entry(test['e1'])
    what_without_ne(e1)
    
    e2 = get_entry(test['e2'])
    what_without_ne(e2)
    
    not_ne_what_coefs.append(
        [
            base.what_distance(e1),
            base.what_distance(e2)
        ]
    )
Explanation: What without NEs
End of explanation
with open('X.txt') as f:
    coefs = [eval(x) for x in f.read().split('\n')]
with open('Y.txt') as f:
    Y = [float(x) for x in f.read().split('\n')]
with open('tests.json') as f:
    tests = json.loads(f.read())
X_copy = list(coefs); Y_copy = list(Y)
X = np.array(
    [
        [
            v1[i] - v2[i]
            for i in range(3)
        ]
        
        for v1, v2 in coefs
    ]
)
Y = np.array(Y)
X_not_ne_what = np.array(
    [
        [
            not_ne_what_coefs[n][0] - not_ne_what_coefs[n][1],
            row[1], row[2]
        ]
        
        for n, row in enumerate(X)
    ]
)
def sigmoid(x, gamma=1.):
    return 1.0 / (1.0 + np.exp(-gamma * x))
def cost(theta, X=None, Y=None): # theta is np.array    
    return np.sum(
        (sigmoid(np.dot(X, np.abs(theta))) - Y) ** 2
    ) / len(X)
grad_cost = grad(cost)
class SGD:
    
    def __init__(self, learning=0.5, max_iters=10**5, prec=10**-3):
        self.learning = learning
        self.max_iters = max_iters
        self.prec = prec
        
        self.theta = None
        
        self._iters = None
        self._costs = None
        
        
    def get_params(self, deep=True):
        return {
            'learning': self.learning,
            'max_iters': self.max_iters,
            'prec': self.prec
        }
    
    
    @property
    def iters(self):
        if self._iters is None:
            raise Exception('SGD must be fitted to access iters')
        
        return self._iters
    
    @iters.setter
    def iters(self, value): self._iters = value
        
        
    @property
    def costs(self):
        if self._costs is None:
            raise Exception('SGD must be fitted to access costs')
        
        return self._costs
    
    @costs.setter
    def costs(self, value): self._costs = value
        
    
    def fit(self, X, Y):
        self.iters = 0
        self.costs = []
        theta = np.random.random(3)
    
        while self.iters < self.max_iters: 
            self.iters += 1
            self.costs.append(cost(theta, X=X, Y=Y))
            
            prev_theta = theta.copy()
            theta -= self.learning * grad_cost(theta, X=X, Y=Y)
            if np.linalg.norm(theta - prev_theta) < self.prec:
                break
        
        self.costs.append(cost(theta, X=X, Y=Y))
        self.theta = theta
        return self
    
    
    def score(self, X, Y):
        return sum(
            (not ((pred > 0.) ^ (cls > 0.))) if pred != 0. else 0.
            for pred, cls in zip(np.dot(X, self.theta), Y)
        ) / len(Y)
class WhatSGD(SGD):
    
    def fit(self, X, Y):
        self.theta = np.array([1., 0., 0.])
        return self
Explanation: SGD
End of explanation
threshold = int(len(X) * 0.9)
X_train, X_test = X[:threshold], X[threshold:]
Y_train, Y_test = Y[:threshold], Y[threshold:]
trained_sgd = SGD()
trained_sgd.fit(X_train, Y_train)
pd.Series(trained_sgd.costs).plot() # error on each iteration
Explanation: Simple trial
End of explanation
X_not_what = X.copy()
for row in X_not_what:
    row[0] = 0
sgd = SGD()
what_sgd = WhatSGD()
sgd_not_what = SGD()
rows = []
for i in range(2, 20 + 1):
    rows.append(
        [
            cross_validation.cross_val_score(
                sgd, X, Y, cv=i
            ), 
            cross_validation.cross_val_score(
                sgd, X_not_ne_what, Y, cv=i
            ), 
            cross_validation.cross_val_score(
                what_sgd, X, Y, cv=i
            ),
            cross_validation.cross_val_score(
                what_sgd, X_not_ne_what, Y, cv=i
            )
        ]
    )
    
for n, i in enumerate(range(2, 20 + 1)):
    rows[n].append(
        cross_validation.cross_val_score(
            sgd_not_what, X_not_what, Y, cv=i
        )
    )
df = pd.DataFrame(
    [[s.mean() for s in row] for row in rows],
    columns=['sgd, what with NE', 'sgd, what without NE', 'what with NE', 'what without NE', 'sgd without what'],
    index=[100 - 100 // i for i in range(2, 20 + 1)]
)
df.plot(ylim=(0, 1))
df.plot()
df.mean()
df[df.index > 75].mean()
df[df.index > 90].mean()
Explanation: Run cross validation for each model
End of explanation
scores = cross_validation.cross_val_score(
    sgd, X, Y, cv=10
)
scores, scores.mean()
scores = cross_validation.cross_val_score(
    sgd, X_not_ne_what, Y, cv=10
)
scores, scores.mean()
scores = cross_validation.cross_val_score(
    what_sgd, X, Y, cv=10
)
scores, scores.mean()
scores = cross_validation.cross_val_score(
    what_sgd, X_not_ne_what, Y, cv=10
)
scores, scores.mean()
sgd_not_what = SGD()
scores = cross_validation.cross_val_score(
    sgd_not_what, X_not_what, Y, cv=10
)
scores, scores.mean()
Explanation: Scores of each model
End of explanation
sgd = SGD()
sgd.fit(X, Y)
cost(sgd.theta, X, Y), cost(sgd.theta / sgd.theta.sum(), X, Y)
Explanation: Final results
End of explanation
sgd.theta, sgd.theta / sgd.theta.sum()
sgd = SGD()
sgd.fit(X_not_ne_what, Y)
sgd.theta, sgd.theta / sgd.theta.sum()
Explanation: The one not normalised gets less error due to the fact that bigger absolute values make sigmoid closer to 1 or 0, thus reducing the error. Anyhow, the desired value is the one normalised (values must sum 1)
End of explanation
sgd_not_what = SGD()
sgd_not_what.fit(X_not_what, Y)
(sgd_not_what.theta - np.array([sgd_not_what.theta[0], 0., 0.])) / sgd_not_what.theta[1:].sum()
Explanation: Notice how with not-ne-what What becomes slightly more important (maybe because it doesn't lose accuracy due to NE mistreatment?). Even if the resulting accuracy is the same but more stable, the other system makes What be more easily computed in terms of performance, so it would still be the selected approach, because the algorithm is slow enough already. In next steps, when the algorithm is optimised, working with not-ne-what would be a good idea.
Finally, by checking what would be the values with just Who and Where:
End of explanation
<END_TASK>