Witten Bell Discounting explanation
<p>In the book by jurafsky and martin, equation (6.16) states that we can consider training corpus as a series of events - one for a token and one for a new N gram type. But then the way the probability is derived, I guess its assumed that the events are disjoint. But if the next word is new then its an event corresponding to a token as well a new N gram - contradicting the assumption of disjoint events. Where am I going wrong in understanding it ?</p>