<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href='http
Step1:  Doc.sents is a generator
It is important to note that doc.sents is a generator. That is, a Doc is not segmented until doc.sents is called. This means that, where you could print the second Doc token with print(doc[1]), you can't call the "second Doc sentence" with print(doc.sents[1])
Step2:  However, you can build a sentence collection by running doc.sents and saving the result to a list
Step3:  <font color=green>NOTE
Step4:  sents are Spans
At first glance it looks like each sent contains text from the original Doc object. In fact they're just Spans with start and end token pointers.
Step5:  Adding Rules
spaCy's built-in sentencizer relies on the dependency parse and end-of-sentence punctuation to determine segmentation rules. We can add rules of our own, but they have to be added before the creation of the Doc object, as that is where the parsing of segment start tokens happens
Step6:  <font color=green>Notice we haven't run doc2.sents, and yet token.is_sent_start was set to True on two tokens in the Doc.</font>
Let's add a semicolon to our existing segmentation rules. That is, whenever the sentencizer encounters a semicolon, the next token should start a new segment.
Step7:  <font color=green>The new rule has to run before the document is parsed. Here we can either pass the argument before='parser' or first=True.
Step8:  Why not change the token directly?
Why not simply set the .is_sent_start value to True on existing tokens?
Step9:  <font color=green>spaCy refuses to change the tag after the document is parsed to prevent inconsistencies in the data.</font>
Changing the Rules
In some cases we want to replace spaCy's default sentencizer with our own set of rules. In this section we'll see how the default sentencizer breaks on periods. We'll then replace this behavior with a sentencizer that breaks on linebreaks.
Step10:  <font color=green>While the function split_on_newlines can be named anything we want, it's important to use the name sbd for the SentenceSegmenter.</font>
<ASSISTANT_TASK:>
Python Code:
# Perform standard imports
import spacy
nlp = spacy.load('en_core_web_sm')
# From Spacy Basics:
doc = nlp(u'This is the first sentence. This is another sentence. This is the last sentence.')
for sent in doc.sents:
    print(sent)
Explanation: <a href='http://www.pieriandata.com'> <img src='../Pierian_Data_Logo.png' /></a>
Sentence Segmentation
In spaCy Basics we saw briefly how Doc objects are divided into sentences. In this section we'll learn how sentence segmentation works, and how to set our own segmentation rules.
End of explanation
print(doc[1])
print(doc.sents[1])
Explanation: Doc.sents is a generator
It is important to note that doc.sents is a generator. That is, a Doc is not segmented until doc.sents is called. This means that, where you could print the second Doc token with print(doc[1]), you can't call the "second Doc sentence" with print(doc.sents[1]):
End of explanation
doc_sents = [sent for sent in doc.sents]
doc_sents
Explanation: However, you can build a sentence collection by running doc.sents and saving the result to a list:
End of explanation
# Now you can access individual sentences:
print(doc_sents[1])
Explanation: <font color=green>NOTE: list(doc.sents) also works. We show a list comprehension as it allows you to pass in conditionals.</font>
End of explanation
type(doc_sents[1])
print(doc_sents[1].start, doc_sents[1].end)
Explanation: sents are Spans
At first glance it looks like each sent contains text from the original Doc object. In fact they're just Spans with start and end token pointers.
End of explanation
# Parsing the segmentation start tokens happens during the nlp pipeline
doc2 = nlp(u'This is a sentence. This is a sentence. This is a sentence.')
for token in doc2:
    print(token.is_sent_start, ' '+token.text)
Explanation: Adding Rules
spaCy's built-in sentencizer relies on the dependency parse and end-of-sentence punctuation to determine segmentation rules. We can add rules of our own, but they have to be added before the creation of the Doc object, as that is where the parsing of segment start tokens happens:
End of explanation
# SPACY'S DEFAULT BEHAVIOR
doc3 = nlp(u'"Management is doing things right; leadership is doing the right things." -Peter Drucker')
for sent in doc3.sents:
    print(sent)
# ADD A NEW RULE TO THE PIPELINE
def set_custom_boundaries(doc):
    for token in doc[:-1]:
        if token.text == ';':
            doc[token.i+1].is_sent_start = True
    return doc
nlp.add_pipe(set_custom_boundaries, before='parser')
nlp.pipe_names
Explanation: <font color=green>Notice we haven't run doc2.sents, and yet token.is_sent_start was set to True on two tokens in the Doc.</font>
Let's add a semicolon to our existing segmentation rules. That is, whenever the sentencizer encounters a semicolon, the next token should start a new segment.
End of explanation
# Re-run the Doc object creation:
doc4 = nlp(u'"Management is doing things right; leadership is doing the right things." -Peter Drucker')
for sent in doc4.sents:
    print(sent)
# And yet the new rule doesn't apply to the older Doc object:
for sent in doc3.sents:
    print(sent)
Explanation: <font color=green>The new rule has to run before the document is parsed. Here we can either pass the argument before='parser' or first=True.
End of explanation
# Find the token we want to change:
doc3[7]
# Try to change the .is_sent_start attribute:
doc3[7].is_sent_start = True
Explanation: Why not change the token directly?
Why not simply set the .is_sent_start value to True on existing tokens?
End of explanation
nlp = spacy.load('en_core_web_sm')  # reset to the original
mystring = u"This is a sentence. This is another.\n\nThis is a \nthird sentence."
# SPACY DEFAULT BEHAVIOR:
doc = nlp(mystring)
for sent in doc.sents:
    print([token.text for token in sent])
# CHANGING THE RULES
from spacy.pipeline import SentenceSegmenter
def split_on_newlines(doc):
    start = 0
    seen_newline = False
    for word in doc:
        if seen_newline:
            yield doc[start:word.i]
            start = word.i
            seen_newline = False
        elif word.text.startswith('\n'): # handles multiple occurrences
            seen_newline = True
    yield doc[start:]      # handles the last group of tokens
sbd = SentenceSegmenter(nlp.vocab, strategy=split_on_newlines)
nlp.add_pipe(sbd)
Explanation: <font color=green>spaCy refuses to change the tag after the document is parsed to prevent inconsistencies in the data.</font>
Changing the Rules
In some cases we want to replace spaCy's default sentencizer with our own set of rules. In this section we'll see how the default sentencizer breaks on periods. We'll then replace this behavior with a sentencizer that breaks on linebreaks.
End of explanation
doc = nlp(mystring)
for sent in doc.sents:
    print([token.text for token in sent])
Explanation: <font color=green>While the function split_on_newlines can be named anything we want, it's important to use the name sbd for the SentenceSegmenter.</font>
End of explanation
<END_TASK>