<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 A Focus on European Wine
By
Step1:  Code to remove the columns “Class”, “By”, “TA”, “JL”, “PW” (i.e. irrelevant and less relevant data to do with the project)
Step2:  Code to clean the data (remove unwanted symbols)
Step3:  Code to normalize data (normalize ratings from 0 to 100)
Step4:  Print data
Step5:  Finding Trends in the Dataset
Note
Step6:  1)  How consistent are wine ratings?
Step7:  While the figure above shows that the standard deviations of wine ratings is slightly skewed to the left, the figure appears to be approximately a normal distribution. 
The mean of standard deviations (approximately 23.0) and median of standard deviations (approximately 24.4) are quite high, suggesting that there is a high deviation among the wine ratings (which are normalized to be out of 100). 
The data thus suggest that wine ratings are not very consistent among different critics for the same wines. 
2) Do ratings actually justify the price at all price points?
Step8:  The figure suggests that there may be a possible trend that higher-priced wines have higher ratings. In general, low-priced wines seem to achieve more lower scores than high-priced wines (it should be noted, however, that there are also more low-priced wines than high-priced wines in the data). Also, it is interesting that above a price point of around 400 euros, a median rating below 50 (out of a scale of 100) does not appear.
However, the figure also shows that many lower priced wines also achieve high ratings. In fact, some low-priced wines are shown to achieve ratings just as high as those of higher-priced wines. Even if there is a trend, the figure suggests a weak correlation, which is also supported by the correlation calculation between price and median rating of approximately 0.429. Furthermore, the data appear to be heteroscedastic; there is a greater variation in median rating among cheaper wines than more highly-priced wines.
As a result, we can cannot conclude whether ratings justify the price of wines at all price points, as there does not appear to be a statistically significant trend in the dataset.
3) Do ratings justify the price of luxury wines?
Step9:  To be categorized as a luxury wine, the price of the wine has been categorized as within the top decile of wine prices. 
Once again, the figure produced suggests that there is at most a very weak trend between price and median ratings of expensive wines. In fact, the correlation between price and median rating among luxury wines is lower than that between price and median rating among all wines. 
Overall, this suggests that higher or lower ratings do not seem to affect the price of wines. In addition, the data suggest that higher-priced wines do not necessarily yield higher critic ratings.
4)  What years yielded the highest rated wines?
Step10:  Given the dataset of wines from 2004 to 2016, the figure above suggests that 2015 and 2016 yielded wines with higher ratings. Yearly ratings were based on median ratings. From a numerical standpoint, the median rating of 2015 vintage wines is approximately 66.7 (on a scale of 100)
Step11:  By looking at the median ratings of wines by region ("AOC" in the dataset), Sauternes appears to yield the highest rated wines (numerically, Sauternes's median rating is approximately 69.811 out of a scale of 100). The difference in median rating between Sauternes and Pauillac is quite high (approximately 7.55), suggesting that Sauternes produces wines that could potentially be considered better than the other locations.
Following Sauternes, Pauillac and St-Julien take the top three regions for producing highly-rated wine according to median wine ratings.
6)  Which regions yield more expensive wines?
It should be noted that the price of wines are right-skewed because of a few extremely expensive vintage wines. As such, both mean and median price appear (the mean is considered to take into account the effect of outliers)
Step12:  Based on median prices, the figure identifies that Pomerol produces the most expensive wines (with a median wine price of 44 euros). Following Pomerol are St-Julien and Sauternes (with median wine prices of 42.5 and 39.0, respectively).
Step13:  Based on mean prices, Pauillac produces the most expensive wines (with a mean price of approximately 101.74 euros). The change in region can be explained by the fact that some wines are much more highly priced than others and thus skew the data.
Overall, Pomerol appears to be a top region to produce highly-priced wines since it appeared in both regions producing expensive wines based on mean and median prices. Pauillac was also placed fourth for producing highly-priced wines based on median prices. 
7)  What characteristics of wines tend to make wines appreciate over time?
In order to determine what characteristics affect how wine appreciates over time, critic ratings, year, and place of origin are considered
Step14:  In terms of critic ratings, the figure above suggests that median ratings and wine appreciation amounts have at most a very weak trend. In fact, there is a correlation of only approximately 0.279, which suggests that critic ratings likely are unrelated to whether a wine will appreciate and by how much.
Step15:  In terms of year, the figure above shows that wines produced in 2005, 2009, 2010, 2014, and 2015 appreciated. The wines of 2009 seem to have appreciated the most (by a mean appreciation of approximately 68.13%). 2005 wines appreciated a similar amount at approximately 66.43%. 
It should be noted that there is a lack of 2004 and 2016 appreciation data. Thus, there appears to be 0% appreciation for both years.
<ASSISTANT_TASK:>
Python Code:
import pandas
file = open("./data.html")
data = pandas.io.html.read_html(file, encoding='utf-8')[0]
Explanation: A Focus on European Wine
By: Anabelle Liao al4622
Project Background
This project’s purpose is to determine the worthiness of wines at different price points and to draw conclusions from general trends in the data. The project, for instance, explores whether there is justification for price premiums on particular wines. 
The subset of data that is used may be found at: http://www.bordoverview.com/?year=all&bank=both
The data subset covers the data of 5181 vintage wines between 2004 and 2016. 
About The Dataset
Attributes of the Dataset
Wine: The wine name 
Year: The year the grapes were harvested and wine was produced (min year: 2004; max year: 2016)
AOC: The wine’s place of origin, stands for "Appellation d'origine contrôlée"
Size: Size of the property involved in producing the wine (in hectares)
RP: Ratings by Robert Parker, a wine critic for The Wine Advocate 
NM: Ratings by Neal Martin, a wine critic who contributes to Wine-Journal
JR: Ratings by Jancis Robinson, an English wine critic
BD: Ratings by Michel Bettane and Thierry Desseauve, French wine critics for TAST
JS: Ratings by James Suckling, American wine critic for 
De: Ratings by Decanter, an English wine magazine
RVF: Ratings from La Revue du Vin de France, a French wine magazine
RG: Ratings by Rene Gabriel, a Swiss wine critic for WeinWisser
Price: the initial price of the wine in the year it was produced
+/-: the percentage price change of the wine since its produced year (until 2017)
Accessing, Cleaning, and Normalizing the Dataset
Code to access the subset of the data as a CSV file after downloading dataset as an HTML file (by parsing the HTML code):
End of explanation
data.drop(["Class.", "By", "TA", "JL", "PW"], axis=1, inplace=True)
Explanation: Code to remove the columns “Class”, “By”, “TA”, “JL”, “PW” (i.e. irrelevant and less relevant data to do with the project):
End of explanation
import re
def cleanEntry(str):
    match = re.search(r"(-?(?:\d*\.)?\d+)(?:-(-?(?:\d*\.)?\d+))?", str)
    if match is None:
        return None
    else:
        groups = match.groups()
        if groups[1] is None:
            return float(groups[0])
        else:
            return (float(groups[0]) + float(groups[1])) / 2
        
data.rename(columns={"+/–": "+/-"}, inplace=True)
data.replace(["·", "\?", "½", "%", "€ ", "\+", "–"],
             ["", "", ".5", "", "", "", "-"],
             inplace=True, regex=True)
colsToClean = ["RP", "NM", "JR", "BD", "JS", "De", "RVF", "JA", "RG", "Price", "+/-"]
data[colsToClean] = data[colsToClean].applymap(cleanEntry)
Explanation: Code to clean the data (remove unwanted symbols):
End of explanation
ratingColNames = ["RP", "NM", "JR", "BD", "JS", "De", "RVF", "JA", "RG"]
for col in ratingColNames:
    data[col] -= data[col].min()
    data[col] /= data[col].max()
    data[col] *= 100
Explanation: Code to normalize data (normalize ratings from 0 to 100):
End of explanation
print(data)
Explanation: Print data:
End of explanation
%matplotlib inline
import matplotlib.pyplot as plt
Explanation: Finding Trends in the Dataset
Note: the median is mainly used to determine trends in the data, as median is generally a better indicator in case of any skewdness in the dataset (such as among prices, etc.). However, for appreciation amount, the mean is used to take into account the effect of outliers.
End of explanation
stddevs = data[ratingColNames].apply(lambda x: x.std(), axis=1)
print(stddevs.describe())
stddevs.plot.hist(bins=50)
plt.title("Histogram of Standard Deviation of Wine Ratings")
plt.xlabel("Standard Deviation of Wine Ratings")
Explanation: 1)  How consistent are wine ratings?
End of explanation
medianRatings = data[ratingColNames].apply(lambda x: x.median(), axis=1).rename("Median rating")
print("Correlation:", medianRatings.corr(data["Price"]))
pandas.concat([medianRatings, data["Price"]], axis=1)\
    .plot\
    .scatter(x="Price",
             y="Median rating",
             s=0.25,
             xlim=(0, 1000),
             ylim=(medianRatings.min(), medianRatings.max()),
             title="Median Rating vs. Price for All Wines")
Explanation: While the figure above shows that the standard deviations of wine ratings is slightly skewed to the left, the figure appears to be approximately a normal distribution. 
The mean of standard deviations (approximately 23.0) and median of standard deviations (approximately 24.4) are quite high, suggesting that there is a high deviation among the wine ratings (which are normalized to be out of 100). 
The data thus suggest that wine ratings are not very consistent among different critics for the same wines. 
2) Do ratings actually justify the price at all price points?
End of explanation
luxuryWines = data[data["Price"] > data["Price"].quantile(0.9)] 
medianRatings = luxuryWines[ratingColNames].apply(lambda x: x.median(), axis=1).rename("Median rating")
print("Correlation:", medianRatings.corr(data["Price"]))
pandas.concat([medianRatings, data["Price"]], axis=1)\
    .plot\
    .scatter(x="Price",
             y="Median rating",
             s=1,
             xlim=(luxuryWines["Price"].min(), luxuryWines["Price"].max()),
             ylim=(medianRatings.min(), medianRatings.max()),
             title="Median Rating vs. Price for Luxury Wines")\
    .set_xlabel("Price (€)")
Explanation: The figure suggests that there may be a possible trend that higher-priced wines have higher ratings. In general, low-priced wines seem to achieve more lower scores than high-priced wines (it should be noted, however, that there are also more low-priced wines than high-priced wines in the data). Also, it is interesting that above a price point of around 400 euros, a median rating below 50 (out of a scale of 100) does not appear.
However, the figure also shows that many lower priced wines also achieve high ratings. In fact, some low-priced wines are shown to achieve ratings just as high as those of higher-priced wines. Even if there is a trend, the figure suggests a weak correlation, which is also supported by the correlation calculation between price and median rating of approximately 0.429. Furthermore, the data appear to be heteroscedastic; there is a greater variation in median rating among cheaper wines than more highly-priced wines.
As a result, we can cannot conclude whether ratings justify the price of wines at all price points, as there does not appear to be a statistically significant trend in the dataset.
3) Do ratings justify the price of luxury wines?
End of explanation
wineByYear = data.groupby(data["Year"]).median()
medianRatings = wineByYear[ratingColNames].apply(lambda x: x.median(), axis=1).rename("Median rating")
print(medianRatings.sort_values(ascending=False))
medianRatings.plot(title="Median Rating of Wines by Year of Production").set_ylabel("Median rating")
Explanation: To be categorized as a luxury wine, the price of the wine has been categorized as within the top decile of wine prices. 
Once again, the figure produced suggests that there is at most a very weak trend between price and median ratings of expensive wines. In fact, the correlation between price and median rating among luxury wines is lower than that between price and median rating among all wines. 
Overall, this suggests that higher or lower ratings do not seem to affect the price of wines. In addition, the data suggest that higher-priced wines do not necessarily yield higher critic ratings.
4)  What years yielded the highest rated wines?
End of explanation
wineByRegion = data.groupby(data["AOC"]).median()
medianRatings = wineByRegion[ratingColNames].apply(lambda x: x.median(), axis=1).rename("Median rating")
sortedByRating = medianRatings.sort_values(ascending=False)
print(sortedByRating)
sortedByRating.plot.barh(title="Median Rating of Wines by Region")
plt.xlabel("Median rating")
plt.gca().invert_yaxis()
Explanation: Given the dataset of wines from 2004 to 2016, the figure above suggests that 2015 and 2016 yielded wines with higher ratings. Yearly ratings were based on median ratings. From a numerical standpoint, the median rating of 2015 vintage wines is approximately 66.7 (on a scale of 100): this is only approximately 0.769 higher than the median rating of 2016 vintage wines. Interestingly enough, the high ratings in 2015 followed a year of quite low ratings in 2014. 
5)  Which regions yield higher-rated wines?
End of explanation
sortedByMedianPrice = data.groupby(data["AOC"]).median()["Price"].sort_values(ascending=False)
print(sortedByMedianPrice)
sortedByMedianPrice.plot.barh(title="Median Price of Wines by Region")
plt.xlabel("Median price (€)")
plt.gca().invert_yaxis()
Explanation: By looking at the median ratings of wines by region ("AOC" in the dataset), Sauternes appears to yield the highest rated wines (numerically, Sauternes's median rating is approximately 69.811 out of a scale of 100). The difference in median rating between Sauternes and Pauillac is quite high (approximately 7.55), suggesting that Sauternes produces wines that could potentially be considered better than the other locations.
Following Sauternes, Pauillac and St-Julien take the top three regions for producing highly-rated wine according to median wine ratings.
6)  Which regions yield more expensive wines?
It should be noted that the price of wines are right-skewed because of a few extremely expensive vintage wines. As such, both mean and median price appear (the mean is considered to take into account the effect of outliers):
End of explanation
sortedByMeanPrice = data.groupby(data["AOC"]).mean()["Price"].sort_values(ascending=False)
print(sortedByMeanPrice)
sortedByMeanPrice.plot.barh(title="Mean Price of Wines by Region")
plt.xlabel("Median price (€)")
plt.gca().invert_yaxis()
Explanation: Based on median prices, the figure identifies that Pomerol produces the most expensive wines (with a median wine price of 44 euros). Following Pomerol are St-Julien and Sauternes (with median wine prices of 42.5 and 39.0, respectively).
End of explanation
medianRatings = data[ratingColNames].apply(lambda x: x.median(), axis=1).rename("Median rating")
print("Correlation:", medianRatings.corr(data["+/-"]))
pandas.concat([medianRatings, data], axis=1)\
    .plot\
    .scatter(y="+/-",
             x="Median rating",
             s=1,
             ylim=(data["+/-"].min(), data["+/-"].max()),
             xlim=(medianRatings.min(), medianRatings.max()), title="Appreciation Amount vs. Median Rating")\
    .set_ylabel("Appreciation amount (%)")
Explanation: Based on mean prices, Pauillac produces the most expensive wines (with a mean price of approximately 101.74 euros). The change in region can be explained by the fact that some wines are much more highly priced than others and thus skew the data.
Overall, Pomerol appears to be a top region to produce highly-priced wines since it appeared in both regions producing expensive wines based on mean and median prices. Pauillac was also placed fourth for producing highly-priced wines based on median prices. 
7)  What characteristics of wines tend to make wines appreciate over time?
In order to determine what characteristics affect how wine appreciates over time, critic ratings, year, and place of origin are considered:
End of explanation
appreciationByYear = data.groupby(data["Year"]).mean()["+/-"]
print(appreciationByYear)
appreciationByYear.plot.bar()
plt.title("Mean Appreciation Amount by Year of Production")
plt.ylabel("Mean appreciation amount (%)")
Explanation: In terms of critic ratings, the figure above suggests that median ratings and wine appreciation amounts have at most a very weak trend. In fact, there is a correlation of only approximately 0.279, which suggests that critic ratings likely are unrelated to whether a wine will appreciate and by how much.
End of explanation
appreciationByRegion = data.groupby(data["AOC"]).mean()["+/-"].sort_values(ascending=False)
print(appreciationByRegion)
appreciationByRegion.plot.barh()
plt.title("Mean Appreciation Amount by Region")
plt.xlabel("Mean appreciation amount (%)")
plt.gca().invert_yaxis()
Explanation: In terms of year, the figure above shows that wines produced in 2005, 2009, 2010, 2014, and 2015 appreciated. The wines of 2009 seem to have appreciated the most (by a mean appreciation of approximately 68.13%). 2005 wines appreciated a similar amount at approximately 66.43%. 
It should be noted that there is a lack of 2004 and 2016 appreciation data. Thus, there appears to be 0% appreciation for both years.
End of explanation
<END_TASK>