<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Getting Started with Data
Example code blocks showing how to load various data types into Python
Designed for Python 3.5+
Step1:  .csv — Comma-Separated Values
Data type with the least overhead (aka file size) and very common
Std Python
Conceptually, the basics of a CSV are just like splitting a string by commas
Step2:  But there are more than just strings or rows with a single data type. We can use the csv library to handle edge cases and memory management for us.
Step3:  Pandas
Because CSV matches the structure of pandas.DataFrame, it's really easy to import the data.
Step4:  .xls .xlsx — Excel Spreadsheet
Excel spreadsheets are more complicated than CSV (ex they can contain multiple datasets), but they still follow the 2D array format of a CSV and DataFrames
Std Python
Step5:  Pandas
Step6:  .json — JavaScript Object Notation
JSON is a common web and API communication format and easy to read. It also happens to be copy/paste compatible as native Python code (with some restrictions the other way around)
Std Python
Step7:  Pandas
Step8:  .xml — eXtensible Markup Language
XML looks and acts just like HTML. It is a popular web format but is being superseded by JSON
Std Python
Step9:  Pandas
There is no native way to read or write XML with pandas. There are plenty of code snippets out there if you absolutely need to
.yml .yaml — Yet Another Markup Language
YAML has less overhead than JSON and can reference other objects within the same file to save space
Std Python
Step10:  Pandas
There's not a native way to read in YAML data to Pandas, but you could use it's JSON lib to do so
Step11:  .p .pkl — Pickle
Pickle files allow us to save and restore binary copies of native language objects (int, list, DataFrame, custom class, etc)
Std Python
Step12:  Pandas
Step13:  Bonus
Here are some scripts we used to create the different datasets
<ASSISTANT_TASK:>
Python Code:
#pandas is commonly imported as pd
import pandas as pd
#We'll import the other libraries as needed
Explanation: Getting Started with Data
Example code blocks showing how to load various data types into Python
Designed for Python 3.5+
End of explanation
print("Split on comma as strings")
csv_row = '1,2.0,Three point five,True'
print(csv_row.split(','))
print("\nSplit on comma and converted to ints")
csv_row = '1,2,3,4,5,6,7,8,9'
print([int(c) for c in csv_row.split(',')])
Explanation: .csv — Comma-Separated Values
Data type with the least overhead (aka file size) and very common
Std Python
Conceptually, the basics of a CSV are just like splitting a string by commas:
End of explanation
import csv
with open('data/volcanoes.csv') as fin:
    csv_data = csv.reader(fin)
    #csv.reader is a generator
    #for row in csv_data: do stuff
    print(csv_data.__next__())
    print()
    print(csv_data.__next__())
with open('data/volcanoes.csv') as fin:
    csv_dict_data = csv.DictReader(fin)
    for row in csv_dict_data:
        print(row)
        print()
        print('Name:', row['Name'], '\nType:', row['Type'])
        break
Explanation: But there are more than just strings or rows with a single data type. We can use the csv library to handle edge cases and memory management for us.
End of explanation
csv_df = pd.read_csv(open('data/volcanoes.csv'))
csv_df.head()
Explanation: Pandas
Because CSV matches the structure of pandas.DataFrame, it's really easy to import the data.
End of explanation
import openpyxl
# You can also use xlrd, xlsxwriter, and a host of others.
# See http://www.python-excel.org/ for all your options.
wb = openpyxl.Workbook()
ws = wb.create_sheet("NewSheet")
ws['A1'] = "Name"
ws['A2'] = "Michael"
ws['B1'] = "fav_color"
ws['B2'] = "Purple"
wb.save("data/ExcelData.xlsx")
excel_row = 'data/ExcelData.xlsx'
data = openpyxl.load_workbook(excel_row)
for sheet in data:
    for row in sheet:
        print(row)
        print(*[cell.value for cell in row])
Explanation: .xls .xlsx — Excel Spreadsheet
Excel spreadsheets are more complicated than CSV (ex they can contain multiple datasets), but they still follow the 2D array format of a CSV and DataFrames
Std Python
End of explanation
file = "data/volcanoes.xlsx"
# Pandas excel loader is built off of several other excel readers,
# such as openXLRD and xlsxwriter
# this is reflected in how many ways there are to read in an excel file.
basicLoad = pd.read_excel(file)
alternateLoad = pd.ExcelFile(file)
basicLoad.head()
# read_excel defaults to read the first sheet in an excel book
# For a comprehensive list of parameters for read_excel, see: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html
# you can specify the sheets you want by name
sheetByName = pd.read_excel(file, sheetname="volcanoes")
# by index
sheetsByIndex = pd.read_excel(file, sheetname=[0])
# if you don't know which sheets you want, you can specify header=None and
# all sheets will be loaded in a nested structure:
allSheetsByHeader = pd.read_excel(file, header=None)
allSheetsBySheets = pd.read_excel(file, sheetname=0)
# You can skip rows or columns
cols = ['Number', 'Name', 'Country', 'Region', 'Type', 'Activity Evidence',
        'Last Known Eruption', 'Latitude', 'Longitude', 'Elevation (Meters)',
        'Dominant Rock Type', 'Tectonic Setting']
subset = pd.read_excel(file, skip_footer=5, skiprows=2, names=cols)
Explanation: Pandas
End of explanation
import json
color_data = [
    {"name": "Michael", "fav_color": "purple"},
    {"name": "Casey", "fav_color": "turquoise"}
]
#load, loads, dump, and dumps are common library APIs
#load/dump tkae file objects while loads/dumps take strings
print(json.dumps(color_data))
json_data = json.load(open('data/volcanoes.json'))
print('Rows:', len(json_data))
Explanation: .json — JavaScript Object Notation
JSON is a common web and API communication format and easy to read. It also happens to be copy/paste compatible as native Python code (with some restrictions the other way around)
Std Python
End of explanation
json_df = pd.read_json('data/volcanoes.json')
json_df.head()
Explanation: Pandas
End of explanation
import xml.etree.ElementTree as ET
tree = ET.parse('data/volcano.xml')
#ET.fromstring(some_xml_string)
root = tree.getroot()
for item in root:
    print(item.tag, ' -- ', item.text)
Explanation: .xml — eXtensible Markup Language
XML looks and acts just like HTML. It is a popular web format but is being superseded by JSON
Std Python
End of explanation
import yaml
yaml_data = yaml.load(open('data/volcanoes.yml'))
print('Rows:', len(yaml_data))
Explanation: Pandas
There is no native way to read or write XML with pandas. There are plenty of code snippets out there if you absolutely need to
.yml .yaml — Yet Another Markup Language
YAML has less overhead than JSON and can reference other objects within the same file to save space
Std Python
End of explanation
yaml_df = pd.io.json.json_normalize(yaml.load(open('data/volcanoes.yml')))
yaml_df.head()
Explanation: Pandas
There's not a native way to read in YAML data to Pandas, but you could use it's JSON lib to do so
End of explanation
import pickle
print('Raw hex format:\n', pickle.dumps(color_data))
pickle.dump(color_data, open('data/colors.pkl', 'wb'))
new_data = pickle.load(open('data/colors.pkl', 'rb'))
print("\nSame? :", color_data == new_data)
Explanation: .p .pkl — Pickle
Pickle files allow us to save and restore binary copies of native language objects (int, list, DataFrame, custom class, etc)
Std Python
End of explanation
color_df = pd.DataFrame(pd.read_pickle('data/colors.pkl'))
color_df
color_df.to_pickle('data/color_df.pkl')
Explanation: Pandas
End of explanation
#Create full JSON from CSV
with open('data/volcanoes.csv') as fin:
    data = [dict(row) for row in csv.DictReader(fin)]
    json.dump(data, open('data/volcanoes.json', 'w'))
#Create single element XML from the JSON data
#This library is not included
from dicttoxml import dicttoxml
print(dicttoxml(json_data[0]), file=open('data/volcano.xml'))
#Create full YAML from the JSON data
yaml.dump(json_data, open('data/volcanoes.yml', 'w'))
Explanation: Bonus
Here are some scripts we used to create the different datasets
End of explanation
<END_TASK>