<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 I'm learning  NLTK + Stanford NLP package
Pattern matching of a sentence
Step2:  Another way to match a sentence is to construct a regular grammar
Step4:  Now, here is a sentence which shows ambiguity since it can be parsed into two different trees.
(http
Step5:  Stanford NLP + Python NLTK
Stanford NLP
* http
Step6:  II. 命名實體識別
Step7:  III. 詞性標註
Step8:  IV. 語法解析
Step9:  V. 依存語法解析
Step10:  稍微了解一下Stanford NLP套件後，我們再回到Python NLTK：
http
Step11:  先以 gutenberg語料庫中的 Leaves of Grass by Walt Whitman為例。接下來的一些範例可以讓我們了解以下四個方法的意義：
nltk.corpus.gutenberg.fileids()
nltk.corpus.gutenberg.raw()
nltk.corpus.gutenberg.sents()
nltk.corpus.gutenberg.words()
Step12:  印出該文前100字
Step13:  印出該文前兩句
Step14:  印出文本尺寸，文本總字數，平均詞長，句長等
Step15:  nltk.FreqDist() can be used to calculate the frequency of words showing in the text.
Step16:  the method {str}.startswith()
Step17:  use the method nltk.ConditionalFreqDist() to calculate the conditional freq. distribution
Step18:  yet another example of the use of the method nltk.ConditionalFreqDist()
from http
<ASSISTANT_TASK:>
Python Code:
alist=[7,1,3,4,5]
print(min(alist))
import re
string="One person found xxx helpful"
if ("One" in string):
    print(True)
else:
    print(False)
pattern="((the|a) (cat|dog|fish)) (barked|slept|swam) ((and|or) (barked|slept|swam))*"
m=re.match(pattern, "the cat barked or swam")
print m.groups()
print m.group(),"\n"
print m.span()
print m.start()
print m.end(),"\n"
print m.group()[0],"\n",m.group()[21],"\n",m.group()[0:22]
Explanation: I'm learning  NLTK + Stanford NLP package
Pattern matching of a sentence:
Regular Expression
e.g. a sentence: "the cat barked or swam" can be matched via the Regular Expression:
((the|a)(cat|dog|fish))(barked|slept|swam)((and|or)(barked|slept|swam))*
End of explanation
from IPython.core.display import display
grammar = nltk.CFG.fromstring(
S -> 'the' S1
S1 -> 'cat' VP
VP -> 'barked' VP1
VP1 ->  C VP
C  -> 'or' | 'and'
VP -> 'swam'
)
sent = ['the', 'cat', 'barked', 'or', 'swam']
parser = nltk.ChartParser(grammar)
for tree in parser.parse(sent):
    print tree
    display(tree)
Explanation: Another way to match a sentence is to construct a regular grammar:
Regular Grammar
End of explanation
from IPython.core.display import display
groucho_grammar = nltk.CFG.fromstring(
S -> NP VP
PP -> P NP
NP -> Det N | Det N PP | 'I'
VP -> V NP | VP PP
Det -> 'an' | 'my'
N -> 'elephant' | 'pajamas'
V -> 'shot'
P -> 'in'
)
sent = ['I', 'shot', 'an', 'elephant', 'in', 'my', 'pajamas']
parser = nltk.ChartParser(groucho_grammar)
for tree in parser.parse(sent):
    print tree
    display(tree)
Explanation: Now, here is a sentence which shows ambiguity since it can be parsed into two different trees.
(http://www.nltk.org/book/ch08.html)
A well-known example of ambiguity is shown in (2), from the Groucho Marx movie, Animal Crackers (1930):
(2)        While hunting in Africa, I shot an elephant in my pajamas. How he got into my pajamas, I don't know.
This grammar permits the sentence to be analyzed in two ways, depending on whether the prepositional phrase in my pajamas describes the elephant or the shooting event.
End of explanation
import nltk
import os
# 提供環境變數
os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home" 
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"
from nltk.tokenize import StanfordTokenizer
tokenizer = StanfordTokenizer()
sent = "Good muffins cost $3.88\nin New York.  Please buy me\ntwo of them.\nThanks."
print(tokenizer.tokenize(sent))
Explanation: Stanford NLP + Python NLTK
Stanford NLP
* http://nlp.stanford.edu/software/
Installation guides
* https://github.com/nltk/nltk/wiki/Installing-Third-Party-Software
* http://www.cnblogs.com/baiboy/p/nltk1.html
I. 分詞
End of explanation
import nltk
import os
os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home" 
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"
from nltk.tag import StanfordNERTagger
eng_tagger = StanfordNERTagger(model_filename=r'english.all.3class.distsim.crf.ser.gz')
print(eng_tagger.tag(['New York',]))
print(eng_tagger.tag('Rami Eid is studying at Stony Brook University in New York'.split()))
Explanation: II. 命名實體識別
End of explanation
from nltk.tag import StanfordPOSTagger
os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home" 
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"
eng_tagger = StanfordPOSTagger(model_filename=r'english-bidirectional-distsim.tagger')
print(eng_tagger.tag('What is the airspeed of an unladen swallow ?'.split()))
Explanation: III. 詞性標註
End of explanation
from nltk.parse.stanford import StanfordParser
from IPython.core.display import display
os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home" 
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"
eng_parser = StanfordParser()
structure=eng_parser.parse("the quick brown fox jumps over the lazy dog".split())
for tree in structure:
    print(tree)
    display(tree)
#print(list(eng_parser.parse("the quick brown fox jumps over the lazy dog".split())))
Explanation: IV. 語法解析
End of explanation
from nltk.parse.stanford import StanfordDependencyParser
os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home" 
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"
eng_parser = StanfordDependencyParser()
res = list(eng_parser.parse("the quick brown fox jumps over the lazy dog".split()))
for row in res[0].triples():
    print(row)
Explanation: V. 依存語法解析
End of explanation
import nltk
print(nltk.corpus.gutenberg.fileids())
Explanation: 稍微了解一下Stanford NLP套件後，我們再回到Python NLTK：
http://www.cnblogs.com/baiboy/p/nltk4.html
End of explanation
print(nltk.corpus.gutenberg.raw("whitman-leaves.txt")[:500])
Explanation: 先以 gutenberg語料庫中的 Leaves of Grass by Walt Whitman為例。接下來的一些範例可以讓我們了解以下四個方法的意義：
nltk.corpus.gutenberg.fileids()
nltk.corpus.gutenberg.raw()
nltk.corpus.gutenberg.sents()
nltk.corpus.gutenberg.words()
End of explanation
print(nltk.corpus.gutenberg.words("whitman-leaves.txt")[:100])
Explanation: 印出該文前100字
End of explanation
for j in range(2):
    print( nltk.corpus.gutenberg.sents("whitman-leaves.txt")[j])
Explanation: 印出該文前兩句
End of explanation
num_char = len(nltk.corpus.gutenberg.raw("whitman-leaves.txt"))    # 原始文本的长度，包括空格、符号等
num_words = len(nltk.corpus.gutenberg.words("whitman-leaves.txt")) # 词的数量
num_sents = len(nltk.corpus.gutenberg.sents("whitman-leaves.txt")) # 句子的数量
num_vocab = len(set([w.lower() for w in nltk.corpus.gutenberg.words("whitman-leaves.txt")])) #文本的尺寸
# 印出文本尺寸，以及文本總字數
print(len(set([w.lower() for w in nltk.corpus.gutenberg.words("whitman-leaves.txt")])),len([w.lower() for w in nltk.corpus.gutenberg.words("whitman-leaves.txt")]))
# 印出平均詞長（包括一个空白符号）、平均句子長度、和文本中每个詞出现的平均次数
print(int(num_char/num_words),int(num_words/num_sents),int(num_words/num_vocab),"whitman-leaves.txt")
Explanation: 印出文本尺寸，文本總字數，平均詞長，句長等
End of explanation
import nltk
from nltk.corpus import brown
print(brown.categories())
new_texts=brown.words(categories='news')
print(brown.raw(categories='news')[:300]+"\n")
wfreq=nltk.FreqDist([w.lower() for w in new_texts])
modals=['can','could','may','might','must','will']
for m in modals:
    print(m + ':',wfreq[m])
len([w.lower() for w in new_texts])
Explanation: nltk.FreqDist() can be used to calculate the frequency of words showing in the text.
End of explanation
a='123 as2'
print(a.startswith('123 111'))
print(a.startswith('123 a'))
print(inaugural.words("1793-Washington.txt")[:200])
Explanation: the method {str}.startswith()
End of explanation
from nltk.corpus import inaugural
print(len(inaugural.fileids()))
print([fileid for fileid in inaugural.fileids()])
import nltk
cfd=nltk.ConditionalFreqDist((target,fileid[:4]) for fileid in inaugural.fileids() for w in inaugural.words(fileid) for target in ['america','citizen'] if w.lower().startswith(target))
cfd.plot()
cfd.tabulate()
[(w.lower()) for fileid in inaugural.fileids() for w in inaugural.words(fileid) for target in ['america','citizen'] if w.lower().startswith(target)][:20]
Explanation: use the method nltk.ConditionalFreqDist() to calculate the conditional freq. distribution:
here's a built-in corpus which records the inaugural speeches of presidents of the U.S.
End of explanation
import nltk
from nltk.tokenize import word_tokenize
sent = "the the the dog dog some other words that we do not care about"
print(word_tokenize(sent))
cfdist = nltk.ConditionalFreqDist((len(word), word) for word in word_tokenize(sent))
cfdist.plot()
print(cfdist.tabulate())
Explanation: yet another example of the use of the method nltk.ConditionalFreqDist()
from http://www.nltk.org/_modules/nltk/probability.html
End of explanation
<END_TASK>