<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Spark version of wordcount examples
Prepare the pyspark environment.
Step1:  Make sure your HDFS is still on and the input files (the three books) are still in the input folder.
Create the input RDD from the files on the HDFS (hdfs
Step2:  Simple Word Count
Perform the counting, by flatMap, map, and reduceByKey.
Step3:  Take the top 10 frequently used words
Step4:  Pattern Matching WordCount
Read the pattern file into a set. (file
Step5:  Perform the counting, by flatMap, filter, map, and reduceByKey.
Step6:  Collect and show the results.
<ASSISTANT_TASK:>
Python Code:
import findspark
import os
findspark.init('/home/ubuntu/shortcourse/spark-1.5.1-bin-hadoop2.6')
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("test").setMaster("local[2]")
sc = SparkContext(conf=conf)
Explanation: Spark version of wordcount examples
Prepare the pyspark environment.
End of explanation
lines = sc.textFile('hdfs://localhost:54310/user/ubuntu/input')
lines.count()
Explanation: Make sure your HDFS is still on and the input files (the three books) are still in the input folder.
Create the input RDD from the files on the HDFS (hdfs://localhost:54310/user/ubuntu/input).
End of explanation
from operator import add
counts = lines.flatMap(lambda x: x.split()).map(lambda x: (x, 1)).reduceByKey(add)
Explanation: Simple Word Count
Perform the counting, by flatMap, map, and reduceByKey.
End of explanation
counts.takeOrdered(10, lambda x: -x[1])
Explanation: Take the top 10 frequently used words
End of explanation
pattern = set()
f = open('/home/ubuntu/shortcourse/notes/scripts/wordcount2/wc2-pattern.txt')
for line in f:
    words = line.split()
    for word in words:
        pattern.add(word)
Explanation: Pattern Matching WordCount
Read the pattern file into a set. (file: /home/ubuntu/shortcourse/notes/scripts/wordcount2/wc2-pattern.txt)
End of explanation
result = lines.flatMap(lambda x: x.split()).filter(lambda x: x in pattern).map(lambda x: (x, 1)).reduceByKey(add)
Explanation: Perform the counting, by flatMap, filter, map, and reduceByKey.
End of explanation
result.collect()
# stop the spark context
sc.stop()
Explanation: Collect and show the results.
End of explanation
<END_TASK>