<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Challenge of processing large amounts of data
<ul>
<li>How to process is quickly?
<li>So how do we go about making the problem map so that it can be distributed computation?
<li> Distributed/Parrallel Programming is hard
</ul>
Mapreduce addresses all the challengs
<ul>
<li>Google's computational/data manipulation model
<li> Elegant way to work with big data
</ul>
Map Reduce and the New Software Stack
<ul>
<li> Covered in Chapter 2 in the Ullman book
<li> Covered in Chapter 24 of the Joel Grus book
<li> Python support packages https
Step1:  If there are more parameters then each could be an array, and they are applied together one element at a time
Step2:  Python Reduce Function
For an array passed in, compute a single result.
So a reduce function always has two paramters to carry the result foward with the next element in the sequence.
The operation starts by using the first two values then passes the result with the next element to the function...
Step3:  If there is only one value in a sequence then that element is returned; if the sequence is empty, an exception is raised.
Word Count using python in a map reduce manner
Using the Canterbury Corpus Test file from http
Step4:  MapReduce using MRJOB
Find documentation for MRJOB at https
Step5:  MRJOB Word count function
Step6:  Another way to implement this is to use SPARK on your local box
<ASSISTANT_TASK:>
Python Code:
def cube(x): return x*x*x
map(cube,range(1,11))
Explanation: Challenge of processing large amounts of data
<ul>
<li>How to process is quickly?
<li>So how do we go about making the problem map so that it can be distributed computation?
<li> Distributed/Parrallel Programming is hard
</ul>
Mapreduce addresses all the challengs
<ul>
<li>Google's computational/data manipulation model
<li> Elegant way to work with big data
</ul>
Map Reduce and the New Software Stack
<ul>
<li> Covered in Chapter 2 in the Ullman book
<li> Covered in Chapter 24 of the Joel Grus book
<li> Python support packages https://pypi.python.org/pypi/mrjob
<li> map() and reduce() are basic built in functions in python https://docs.python.org/2/library/functions.html
</ul>
Built-in Python Functional Programming Tools
https://docs.python.org/2/tutorial/datastructures.html
Python Map Function
For each value in a sequence, process each one and output a new result for each element.
End of explanation
seq = range(8)
def add(x,y): return x+y
map(add, seq,seq)
Explanation: If there are more parameters then each could be an array, and they are applied together one element at a time
End of explanation
result = map(add, seq,seq)
reduce(add, result)  # adding each element of the result together
Explanation: Python Reduce Function
For an array passed in, compute a single result.
So a reduce function always has two paramters to carry the result foward with the next element in the sequence.
The operation starts by using the first two values then passes the result with the next element to the function...
End of explanation
import re
import pandas as pd
import numpy as np
aliceFile = open('data/canterbury/alice29.txt','r')
map1=[]
WORD_RE = re.compile(r"[\w']+")
# Create the map of words with prelminary counts
for line in aliceFile:
    for w in WORD_RE.findall(line):
        map1.append([w,1])
#sort the map            
map2 = sorted(map1)
#Separate the map into groups by the key values
df = pd.DataFrame(map2)
uniquewords = df[0].unique()
DataFrameDict = {elem : pd.DataFrame for elem in uniquewords}
for key in DataFrameDict.keys():
    DataFrameDict[key] = df[:][df[0] == key]
def wordcount(x,y):
    x[1] = x[1] + y[1]
    return x
#Add up the counts using reduce
for uw in uniquewords:
    uarray = np.array(DataFrameDict[uw])
    print reduce(wordcount,uarray)
Explanation: If there is only one value in a sequence then that element is returned; if the sequence is empty, an exception is raised.
Word Count using python in a map reduce manner
Using the Canterbury Corpus Test file from http://compression.ca/act/files/canterbury.zip,
We will attempt to count the each unique word.
End of explanation
! pip install mrjob
# %load code/MRWordFrequencyCount.py
# Do not run this cell it is just displaying the content of the code
from mrjob.job import MRJob
class MRWordFrequencyCount(MRJob):
    
    def mapper(self, _ , line):
        yield "chars", len(line)
        yield "words", len(line.split())
        yield "lines", 1
        
    def reducer(self, key, values):
        yield key, sum(values)
        
if __name__ == '__main__':
    MRWordFrequencyCount.run()
%run -G code/MRWordFrequencyCount.py  ./.mrjob.conf data/canterbury/alice29.txt
Explanation: MapReduce using MRJOB
Find documentation for MRJOB at https://pythonhosted.org/mrjob/
A framework that allows you to do mapreduce jobs without HADOOP but will run the same jobs in an hadoop environment.
( DUMBO and Pydoop give you lower level access to HADOOP)
Before using it for the first time install this package using : <code> pip install mrjob </code> 
If that does not work use the alternatives in provided on https://pythonhosted.org/mrjob/guides/quickstart.html#installation
%%writefile myfile.py
write/save cell contents into myfile.py (use -a to append). Another alias: %%file myfile.py
%run myfile.py
run myfile.py and output results in the current cell
%load myfile.py
load "import" myfile.py into the current cell
End of explanation
# %load code\MRWordFreqCount.py
# Do not run this block of code just a display of the stored program
from mrjob.job import MRJob
import re
WORD_RE = re.compile(r"[\w']+")
class MRWordFreqCount(MRJob):
    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield word.lower(), 1
    def combiner(self, word, counts):
        yield word, sum(counts)
    def reducer(self, word, counts):
        yield word, sum(counts)
if __name__ == '__main__':
    MRWordFreqCount.run()
%run code/MRWordFreqCount.py data/canterbury/alice29.txt
!pip install pyspark
Explanation: MRJOB Word count function
End of explanation
# %load code\pysparkcount.py
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#https://github.com/apache/spark/blob/master/examples/src/main/python/wordcount.py
from __future__ import print_function
import sys
from operator import add
from pyspark import SparkContext
if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: wordcount <file>", file=sys.stderr)
        exit(-1)
    sc = SparkContext(appName="PythonWordCount")
    lines = sc.textFile(sys.argv[1], 1)
    counts = lines.flatMap(lambda x: x.split(' ')) \
                  .map(lambda x: (x, 1)) \
                  .reduceByKey(add)
    output = counts.collect()
    for (word, count) in output:
        print("%s: %i" % (word, count))
    sc.stop()
Explanation: Another way to implement this is to use SPARK on your local box: 
Need to load spark and pyspark to get all this working
http://spark.apache.org/downloads.html
pyspark:  pip install pyspark
On windows you need to follow:
 https://medium.com/@GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c
Test code From:
https://github.com/eBay/Spark/blob/master/examples/src/main/python/wordcount.py
End of explanation
<END_TASK>