<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Let's read the data
Step1:  Let's check what's inside these files...
Step2:  Only proper rows with posts
Step3:  Let's parse this mess...
Step4:  Better
Step5:  Let's compute tag counts!
Step6:  Taking long? go to
Step7:  Shout if you're the first one here! Congrats!
Puzzles
<ASSISTANT_TASK:>
Python Code:
! gsutil ls gs://pyspark-workshop/so-posts
lines = sc.textFile("gs://pyspark-workshop/so-posts/*")
# or a smaller piece of them
lines = sc.textFile("gs://pyspark-workshop/so-posts/Posts.xml-*a")
Explanation: Let's read the data
End of explanation
lines.take(5)
Explanation: Let's check what's inside these files...
End of explanation
rows = lines.filter(lambda x: x.lstrip().startswith('<row'))
Explanation: Only proper rows with posts
End of explanation
import xml.etree.ElementTree as ET
parsed = lines.map(lambda x: x.lstrip()).filter(lambda x: x.startswith('<row')).map(lambda x: ET.fromstring(x))
from pprint import pprint
pprint(parsed.take(2))
Explanation: Let's parse this mess...
End of explanation
pprint(parsed.map(lambda x: x.attrib).take(3))
Explanation: Better:
End of explanation
def parse_tags(x):
    return x[1:-1].split("><")
tags = parsed.map(lambda x: parse_tags(x.attrib['Tags']) if 'Tags' in x.attrib else [])
tags.take(5)
counts = tags.flatMap(lambda x: x).groupBy(lambda x: x).map(lambda x: (x[0], len(x[1])))
Explanation: Let's compute tag counts!
End of explanation
counts.sortBy(lambda x: x[1], ascending=False).take(10)
Explanation: Taking long? go to: http://cluster-1-m:8088 and explore it (if you're using default cluster name).
Did you know flatMap?? If yes, rewrite the statement before to use flatMap.
End of explanation
# if you hate xml (you do), then save it as json on hdfs!
import json
parsed.map(lambda x: json.dumps(x.attrib)).saveAsTextFile("posts.jsons")
Explanation: Shout if you're the first one here! Congrats!
Puzzles:
Can you compute how many times someone asked about Python this month (you can compute posts with python tag only)?
Can you measure Pythons monthly popularity over last year? Can you plot it?
Can you do the same but only for main posts (questions)?
(*) Can you find the question that has the most posts attached?? 
Do the same but use ranking by total score of subposts.
End of explanation
<END_TASK>