<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Linguistics 110
Step1:  Exploring TIMIT Data <a id='timit'></a>
We will start off by exploring TIMIT data taken from 8 different regions. These measurements are taken at the midpoint of vowels, where vowel boundaries were determined automatically using forced alignment.
Uploading the data
Prior to being able to work with the data, we have to upload our dataset. The following two lines of code will read in our data and create a dataframe. The last line of code prints the timit dataframe, but instead of printing the whole dataframe, by using the method .head, it only prints the first 5 rows.
Step2:  Look at the dataframe you created and try to figure out what each column measures. Each column represents a different attribute, see the following table for more information.
|Column Name|Details|
|---|---|
|speaker|unique speaker ID|
|gender|Speaker’s self-reported gender|
|region|Speaker dialect region number|
|word|Lexical item (from sentence prompt)|
|vowel|Vowel ID|
|duration|Vowel duration (seconds)|
|F1/F2/F3/f0|f0 and F1-F3 in BPM (Hz)|
Sometimes data is encoded with with an identifier, or key, to save space and simplify calculations. Each of those keys corresponds to a specific value. If you look at the region column, you will notice that all of the values are numbers. Each of those numbers corresponds to a region, for example, in our first row the speaker, cjf0, is from region 1. That corresponds to New England. Below is a table with all of the keys for region.
|Key|Region|
|---|---|
|1|New England|
|2|Northern|
|3|North Midland|
|4|South Midland|
|5|Southern|
|6|New York City|
|7|Western|
|8|Army Brat|
Transformations
When inspecting data, you may realize that there are changes to be made -- possibly due to the representation to the data or errors in the recording. Before jumping into analysis, it is important to clean the data. 
One thing to notice about timit is that the column vowel contains ARPABET identifiers for the vowels. We want to convert the vowel column to be IPA characters, and will do so in the cell below.
Step3:  Most of the speakers will say the same vowel multiple times, so we are going to average those values together. The end result will be a dataframe where each row represents the average values for each vowel for each speaker.
Step4:  Splitting on Gender
Using the same dataframe from above, timit_avg, we are going to split into dataframes grouped by gender. To identify the possible values of gender in the gender column, we can use the method .unique on the column.
Step5:  You could see that for this specific dataset there are only "female" and "male" values in the column. Given that information, we'll create two subsets based off of gender. 
We'll split timit_avg into two separate dataframes, one for females, timit_female, and one for males, timit_male. Creating these subset dataframes does not affect the original timit_avg dataframe.
Step6:  Distribution of Formants
We want to inspect the distributions of F1, F2, and F3 for those that self-report as male and those that self-report as female to identify possible trends or relationships. Having our two split dataframes, timit_female and timit_male, eases the plotting process. 
Run the cell below to see the distribution of F1.
Step7:  Does there seem to be a notable difference between male and female distributions of F1?
Next, we plot F2.
Step8:  Finally, we create the same visualization, but for F3.
Step9:  Do you see a more pronounced difference across the the different F values? Are they the same throughout? Can we make any meaningful assumptions from these visualizations?
An additional question
Step10:  The ID column contains a unique value for each individual. Each individual has a row for each of the different vowels they measured.
Step11:  Splitting on Gender
As we did with the TIMIT data, we are going to split class_data based on self-reported gender. We need to figure out what the possible responses for the column were.
Step12:  Notice that there are three possible values for the column. We do not have a large enough sample size to responsibly come to conclusions for Prefer not to answer, so for now we'll compare Male and Female. We'll call our new split dataframes class_female and class_male.
Step13:  Comparing Distributions
The following visualizations compare the the distribution of formants for males and females, like we did for the TIMIT data.
First, we'll start with F1.
Step14:  Next is F2.
Step15:  And finally F3.
Step16:  Do the spread of values appear to be the same for females and males? Do the same patterns that occur in the TIMIT data appear in the class's data?
Vowel Spaces <a id='vs'></a>
Run the cell below to define some functions that we will be using.
Step17:  We are going to be recreating the following graphic from this website.
Before we can get to creating, we need to get a singular value for each column for each of the vowels (so we can create coordinate pairs). To do this, we are going to find the average formant values for each of the vowels in our dataframes. We'll do this for both timit and class_data.
Step18:  Each of these new tables has a row for each vowel, which comprisises of the averaged values across all speakers.
Plotting the Vowel Space
Run the cell below to construct a vowel space for the class's data, in which we plot F1 on F2.
Note that both axes are descending.
Step19:  Using Logarithmic Axes
In our visualization above, we use linear axes in order to construct our vowel space. The chart we are trying to recreate has logged axes (though the picture does not indicate it). Below we log-transform all of the values in our dataframes.
Step20:  Below we plot the vowel space using these new values.
Step21:  What effect does using the logged values have, if any? What advantages does using these values have? Are there any negatives? This paper might give some ideas.
Overlaying a Vowel Space Chart
Finally, we are going to overlay a blank vowel space chart outline to see how close our data reflects the theoretical vowel chart.
Step22:  How well does it match the original?
Below we generate the same graph, except using the information from the TIMIT dataset.
Step23:  How does the TIMIT vowel space compare to the vowel space from our class data? What may be the cause for any differences between our vowel space and the one constructed using the TIMIT data? Do you notice any outliers or do any points that seem off?
Variation in Vowel Spaces <a id='vvs'></a>
In the following visualizations, we are going to show each individual vowel from each person in the F2 and F1 dimensions (logged). Each color corresponds to a different vowel -- see the legend for the exact pairs.
Step24:  In the following visualization, we replace the colors with the IPA characters and attempt to clump the vowels together.
Step25:  Formants vs Height <a id='fvh'></a>
We are going to compare each of the formants and height to see if there is a relationship between the two. To help visualize that, we are going to plot a regression line, which is also referred to as the line of best fit.
We are going to use the maximum of each formant to compare to height. So for each speaker, we will calculate their greatest F1, F2, and F3 across all vowels, then compare one of those to their height. We create the necessary dataframe in the cell below using the class's data.
Step26:  First we will plot Max F1 against Height.
Note
Step27:  Is there a general trend for the data that you notice? What do you notice about the different color dots?
Next, we plot Max F2 on Height.
Step28:  Finally, Max F3 vs Height.
Step29:  Do you notice a difference between the trends for the three formants?
Now we are going to plot two lines of best fit -- one for males, one for females. Before we plotted one line for all of the values, but now we are separating by gender to see if gender explains some of the difference in formants values. 
For now, we're going deal with just Max F1.
Step30:  Is there a noticeable difference between the two? Did you expect this result?
We're going to repeat the above graph, plotting a different regression line for males and females, but this time, using timit -- having a larger sample size may help expose patterns. Before we do that, we have to repeat the process of calulating the maximum value for each formants for each speaker. Run the cell below to do that and generate the plot. The blue dots are females, the orange dots are males, and the green line is the regression line for all speakers.
<ASSISTANT_TASK:>
Python Code:
# DON'T FORGET TO RUN THIS CELL
import math
import numpy as np
import pandas as pd
import seaborn as sns
import datascience as ds
import matplotlib.pyplot as plt
sns.set_style('darkgrid')
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
Explanation: Linguistics 110: Vowel Formants
Professor Susan Lin
In this notebook, we use both data from an outside source and that the class generated to explore the relationships between formants, gender, and height.
Table of Contents
1 - Exploring TIMIT Data
2 - Using the Class's Data
3 - Vowel Spaces
4 - Variation in Vowel Spaces
5 - Formants vs Height
Remember that to run a cell, you can either click the play button in the toolbar, or you can press shift and enter on your keyboard. To get a quick review of Jupyter notebooks, you can look at the VOT Notebook. Make sure to run the following cell before you get started.
End of explanation
timit = pd.read_csv('data/timitvowels.csv')
timit.head()
Explanation: Exploring TIMIT Data <a id='timit'></a>
We will start off by exploring TIMIT data taken from 8 different regions. These measurements are taken at the midpoint of vowels, where vowel boundaries were determined automatically using forced alignment.
Uploading the data
Prior to being able to work with the data, we have to upload our dataset. The following two lines of code will read in our data and create a dataframe. The last line of code prints the timit dataframe, but instead of printing the whole dataframe, by using the method .head, it only prints the first 5 rows.
End of explanation
IPAdict = {"AO" : "ɔ", "AA" : "ɑ", "IY" : "i", "UW" : "u", "EH" : "ɛ", "IH" : "ɪ", "UH":"ʊ", "AH": "ʌ", "AX" : "ə", "AE":"æ", "EY" :"eɪ", "AY": "aɪ", "OW":"oʊ", "AW":"aʊ", "OY" :"ɔɪ", "ER":"ɚ"}
timit['vowel'] = [IPAdict[x] for x in timit['vowel']]
timit.head()
Explanation: Look at the dataframe you created and try to figure out what each column measures. Each column represents a different attribute, see the following table for more information.
|Column Name|Details|
|---|---|
|speaker|unique speaker ID|
|gender|Speaker’s self-reported gender|
|region|Speaker dialect region number|
|word|Lexical item (from sentence prompt)|
|vowel|Vowel ID|
|duration|Vowel duration (seconds)|
|F1/F2/F3/f0|f0 and F1-F3 in BPM (Hz)|
Sometimes data is encoded with with an identifier, or key, to save space and simplify calculations. Each of those keys corresponds to a specific value. If you look at the region column, you will notice that all of the values are numbers. Each of those numbers corresponds to a region, for example, in our first row the speaker, cjf0, is from region 1. That corresponds to New England. Below is a table with all of the keys for region.
|Key|Region|
|---|---|
|1|New England|
|2|Northern|
|3|North Midland|
|4|South Midland|
|5|Southern|
|6|New York City|
|7|Western|
|8|Army Brat|
Transformations
When inspecting data, you may realize that there are changes to be made -- possibly due to the representation to the data or errors in the recording. Before jumping into analysis, it is important to clean the data. 
One thing to notice about timit is that the column vowel contains ARPABET identifiers for the vowels. We want to convert the vowel column to be IPA characters, and will do so in the cell below.
End of explanation
timit_avg = timit.groupby(['speaker', 'vowel', 'gender', 'region']).mean().reset_index()
timit_avg.head()
Explanation: Most of the speakers will say the same vowel multiple times, so we are going to average those values together. The end result will be a dataframe where each row represents the average values for each vowel for each speaker.
End of explanation
timit_avg.gender.unique()
Explanation: Splitting on Gender
Using the same dataframe from above, timit_avg, we are going to split into dataframes grouped by gender. To identify the possible values of gender in the gender column, we can use the method .unique on the column.
End of explanation
timit_female = timit_avg[timit_avg['gender'] == 'female']
timit_male = timit_avg[timit_avg['gender'] == 'male']
Explanation: You could see that for this specific dataset there are only "female" and "male" values in the column. Given that information, we'll create two subsets based off of gender. 
We'll split timit_avg into two separate dataframes, one for females, timit_female, and one for males, timit_male. Creating these subset dataframes does not affect the original timit_avg dataframe.
End of explanation
sns.distplot(timit_female['F1'], kde_kws={"label": "female"})
sns.distplot(timit_male['F1'], kde_kws={"label": "male"})
plt.title('F1')
plt.xlabel("Hz")
plt.ylabel('Proportion per Hz');
Explanation: Distribution of Formants
We want to inspect the distributions of F1, F2, and F3 for those that self-report as male and those that self-report as female to identify possible trends or relationships. Having our two split dataframes, timit_female and timit_male, eases the plotting process. 
Run the cell below to see the distribution of F1.
End of explanation
sns.distplot(timit_female['F2'], kde_kws={"label": "female"})
sns.distplot(timit_male['F2'], kde_kws={"label": "male"})
plt.title('F2')
plt.xlabel("Hz")
plt.ylabel('Proportion per Hz');
Explanation: Does there seem to be a notable difference between male and female distributions of F1?
Next, we plot F2.
End of explanation
sns.distplot(timit_female['F3'], kde_kws={"label": "female"})
sns.distplot(timit_male['F3'], kde_kws={"label": "male"})
plt.title('F3')
plt.xlabel("Hz")
plt.ylabel('Proportion per Hz');
Explanation: Finally, we create the same visualization, but for F3.
End of explanation
# reading in the data
class_data = pd.read_csv('data/110_formants.csv')
class_data.head()
Explanation: Do you see a more pronounced difference across the the different F values? Are they the same throughout? Can we make any meaningful assumptions from these visualizations?
An additional question: How do you think the fact that we average each vowel together first for each individual affects the shape of the histograms?
Using the Class's Data <a id='cls'></a>
This portion of the notebook will rely on the data that was submit for HW5. Just like we did for the TIMIT data, we are going to read it into a dataframe and modify the column vowel to reflect the corresponding IPA translation. We will name the dataframe class_data.
End of explanation
# translating the vowel column
class_data['vowel'] = [IPAdict[x] for x in class_data['vowel']]
class_data.head()
Explanation: The ID column contains a unique value for each individual. Each individual has a row for each of the different vowels they measured.
End of explanation
class_data['Gender'].unique()
Explanation: Splitting on Gender
As we did with the TIMIT data, we are going to split class_data based on self-reported gender. We need to figure out what the possible responses for the column were.
End of explanation
class_female = class_data[class_data['Gender'] == 'Female']
class_male = class_data[class_data['Gender'] == 'Male']
Explanation: Notice that there are three possible values for the column. We do not have a large enough sample size to responsibly come to conclusions for Prefer not to answer, so for now we'll compare Male and Female. We'll call our new split dataframes class_female and class_male.
End of explanation
sns.distplot(class_female['F1'], kde_kws={"label": "female"})
sns.distplot(class_male['F1'], kde_kws={"label": "male"})
plt.title('F1')
plt.xlabel("Hz")
plt.ylabel('Proportion per Hz');
Explanation: Comparing Distributions
The following visualizations compare the the distribution of formants for males and females, like we did for the TIMIT data.
First, we'll start with F1.
End of explanation
sns.distplot(class_female['F2'], kde_kws={"label": "female"})
sns.distplot(class_male['F2'], kde_kws={"label": "male"})
plt.title('F2')
plt.xlabel("Hz")
plt.ylabel('Proportion per Hz');
Explanation: Next is F2.
End of explanation
sns.distplot(class_female['F3'], kde_kws={"label": "female"})
sns.distplot(class_male['F3'], kde_kws={"label": "male"})
plt.title('F3')
plt.xlabel("Hz")
plt.ylabel('Proportion per Hz');
Explanation: And finally F3.
End of explanation
def plot_blank_vowel_chart():
    im = plt.imread('images/blankvowel.png')
    plt.imshow(im, extent=(plt.xlim()[0], plt.xlim()[1], plt.ylim()[0], plt.ylim()[1]))
def plot_vowel_space(avgs_df):
    plt.figure(figsize=(10, 8))
    plt.gca().invert_yaxis()
    plt.gca().invert_xaxis()
    
    vowels = ['eɪ', 'i', 'oʊ', 'u', 'æ', 'ɑ', 'ɚ', 'ɛ', 'ɪ', 'ʊ', 'ʌ'] + ['ɔ']
    
    for i in range(len(avgs_df)):
        plt.scatter(avgs_df.loc[vowels[i]]['F2'], avgs_df.loc[vowels[i]]['F1'], marker=r"$ {} $".format(vowels[i]), s=1000)
        
    plt.ylabel('F1')
    plt.xlabel('F2')
Explanation: Do the spread of values appear to be the same for females and males? Do the same patterns that occur in the TIMIT data appear in the class's data?
Vowel Spaces <a id='vs'></a>
Run the cell below to define some functions that we will be using.
End of explanation
class_vowel_avgs = class_data.drop('ID', axis=1).groupby('vowel').mean()
class_vowel_avgs.head()
timit_vowel_avgs = timit.groupby('vowel').mean()
timit_vowel_avgs.head()
Explanation: We are going to be recreating the following graphic from this website.
Before we can get to creating, we need to get a singular value for each column for each of the vowels (so we can create coordinate pairs). To do this, we are going to find the average formant values for each of the vowels in our dataframes. We'll do this for both timit and class_data.
End of explanation
plot_vowel_space(class_vowel_avgs)
plt.xlabel('F2 (Hz)')
plt.ylabel('F1 (Hz)');
Explanation: Each of these new tables has a row for each vowel, which comprisises of the averaged values across all speakers.
Plotting the Vowel Space
Run the cell below to construct a vowel space for the class's data, in which we plot F1 on F2.
Note that both axes are descending.
End of explanation
log_timit_vowels = timit_vowel_avgs.apply(np.log)
log_class_vowels = class_vowel_avgs.apply(np.log)
class_data['log(F1)'] = np.log(class_data['F1'])
class_data['log(F2)'] = np.log(class_data['F2'])
log_class_vowels.head()
Explanation: Using Logarithmic Axes
In our visualization above, we use linear axes in order to construct our vowel space. The chart we are trying to recreate has logged axes (though the picture does not indicate it). Below we log-transform all of the values in our dataframes.
End of explanation
plot_vowel_space(log_class_vowels)
plt.xlabel('log(F2) (Hz)')
plt.ylabel('log(F1) (Hz)');
Explanation: Below we plot the vowel space using these new values.
End of explanation
plot_vowel_space(log_class_vowels)
plot_blank_vowel_chart()
plt.xlabel('log(F2) (Hz)')
plt.ylabel('log(F1) (Hz)');
Explanation: What effect does using the logged values have, if any? What advantages does using these values have? Are there any negatives? This paper might give some ideas.
Overlaying a Vowel Space Chart
Finally, we are going to overlay a blank vowel space chart outline to see how close our data reflects the theoretical vowel chart.
End of explanation
plot_vowel_space(log_timit_vowels)
plot_blank_vowel_chart()
plt.xlabel('log(F2) (Hz)')
plt.ylabel('log(F1) (Hz)');
Explanation: How well does it match the original?
Below we generate the same graph, except using the information from the TIMIT dataset.
End of explanation
sns.lmplot('log(F2)', 'log(F1)', hue='vowel', data=class_data, fit_reg=False, size=8, scatter_kws={'s':30})
plt.xlim(8.2, 6.7)
plt.ylim(7.0, 5.7);
Explanation: How does the TIMIT vowel space compare to the vowel space from our class data? What may be the cause for any differences between our vowel space and the one constructed using the TIMIT data? Do you notice any outliers or do any points that seem off?
Variation in Vowel Spaces <a id='vvs'></a>
In the following visualizations, we are going to show each individual vowel from each person in the F2 and F1 dimensions (logged). Each color corresponds to a different vowel -- see the legend for the exact pairs.
End of explanation
plt.figure(figsize=(10, 12))
pick_vowel = lambda v: class_data[class_data['vowel'] == v]
colors = ['Greys_r', 'Purples_r', 'Blues_r', 'Greens_r', 'Oranges_r', \
          'Reds_r', 'GnBu_r', 'PuRd_r', 'winter_r', 'YlOrBr_r', 'pink_r', 'copper_r']
for vowel, color in list(zip(class_data.vowel.unique(), colors)):
    vowel_subset = pick_vowel(vowel)
    sns.kdeplot(vowel_subset['log(F2)'], vowel_subset['log(F1)'], n_levels=1, cmap=color, shade=False, shade_lowest=False)
for i in range(1, len(class_data)+1):
    plt.scatter(class_data['log(F2)'][i], class_data['log(F1)'][i], color='black', linewidths=.5, marker=r"$ {} $".format(class_data['vowel'][i]), s=40)
plt.xlim(8.2, 6.7)
plt.ylim(7.0, 5.7);
Explanation: In the following visualization, we replace the colors with the IPA characters and attempt to clump the vowels together.
End of explanation
genders = class_data['Gender']
plotting_data = class_data.drop('vowel', axis=1)[np.logical_or(genders == 'Male', genders == 'Female')]
maxes = plotting_data.groupby(['ID', 'Gender']).max().reset_index()[plotting_data.columns[:-2]]
maxes.columns = ['ID', 'Language', 'Gender', 'Height', 'Max F1', 'Max F2', 'Max F3']
maxes_female = maxes[maxes['Gender'] == 'Female']
maxes_male = maxes[maxes['Gender'] == 'Male']
maxes.head()
Explanation: Formants vs Height <a id='fvh'></a>
We are going to compare each of the formants and height to see if there is a relationship between the two. To help visualize that, we are going to plot a regression line, which is also referred to as the line of best fit.
We are going to use the maximum of each formant to compare to height. So for each speaker, we will calculate their greatest F1, F2, and F3 across all vowels, then compare one of those to their height. We create the necessary dataframe in the cell below using the class's data.
End of explanation
sns.regplot('Height', 'Max F1', data=maxes)
sns.regplot('Height', 'Max F1', data=maxes_male, fit_reg=False)
sns.regplot('Height', 'Max F1', data=maxes_female, fit_reg=False)
plt.xlabel('Height (cm)')
plt.ylabel('Max F1 (Hz)')
print('female: green')
print('male: orange')
Explanation: First we will plot Max F1 against Height.
Note: Each gender has a different color dot, but the line represents the line of best fit for ALL points.
End of explanation
sns.regplot('Height', 'Max F2', data=maxes)
sns.regplot('Height', 'Max F2', data=maxes_male, fit_reg=False)
sns.regplot('Height', 'Max F2', data=maxes_female, fit_reg=False)
plt.xlabel('Height (cm)')
plt.ylabel('Max F2 (Hz)')
print('female: green')
print('male: orange')
Explanation: Is there a general trend for the data that you notice? What do you notice about the different color dots?
Next, we plot Max F2 on Height.
End of explanation
sns.regplot('Height', 'Max F3', data=maxes)
sns.regplot('Height', 'Max F3', data=maxes_male, fit_reg=False)
sns.regplot('Height', 'Max F3', data=maxes_female, fit_reg=False)
plt.xlabel('Height (cm)')
plt.ylabel('Max F3 (Hz)')
print('female: green')
print('male: orange')
Explanation: Finally, Max F3 vs Height.
End of explanation
sns.lmplot('Height', 'Max F1', data=maxes, hue='Gender')
plt.xlabel('Height (cm)')
plt.ylabel('Max F1 (Hz)');
Explanation: Do you notice a difference between the trends for the three formants?
Now we are going to plot two lines of best fit -- one for males, one for females. Before we plotted one line for all of the values, but now we are separating by gender to see if gender explains some of the difference in formants values. 
For now, we're going deal with just Max F1.
End of explanation
timit_maxes = timit.groupby(['speaker', 'gender']).max().reset_index()
timit_maxes.columns = ['speaker', 'gender', 'region', 'height', 'word', 'vowel', 'Max duration', 'Max F1', 'Max F2', 'Max F3', 'Max f0']
plt.xlim(140, 210)
plt.ylim(500, 1400)
sns.regplot('height', 'Max F1', data=timit_maxes[timit_maxes['gender'] == 'female'], scatter_kws={'alpha':0.3})
sns.regplot('height', 'Max F1', data=timit_maxes[timit_maxes['gender'] == 'male'], scatter_kws={'alpha':0.3})
sns.regplot('height', 'Max F1', data=timit_maxes, scatter=False)
plt.xlabel('Height (cm)')
plt.ylabel('Max F1 (Hz)');
Explanation: Is there a noticeable difference between the two? Did you expect this result?
We're going to repeat the above graph, plotting a different regression line for males and females, but this time, using timit -- having a larger sample size may help expose patterns. Before we do that, we have to repeat the process of calulating the maximum value for each formants for each speaker. Run the cell below to do that and generate the plot. The blue dots are females, the orange dots are males, and the green line is the regression line for all speakers.
End of explanation
<END_TASK>