<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Examples
Step1:  Building on our discussion of modules from last week, we'll use the my_dataset module that I have prepared as a basis.  This module is largely identical to what we have built out in previous weeks.
Step2:  We'll use a read_csv function that is provided in our module to feed into a Dataset object.  This loads in our full dataset, and we'll take a look at the columns it generates.
Note that we're using a different encoding.  In particular, this is because the file is encoded in cp1252, which is a Windows-specific extension that includes some characters that can't be interpreted directly in Unicode.  If we didn't specify this, it would crash.
Step3:  Let's see what data is available.
Step4:  Wow!  OK, so, we have a column for each year that data is available.  Other than that, we have an indicator code, a country code, and then the longer names of those two things.  How many indicator codes are there?  (Spoiler alert
Step5:  We'll convert our columns like usual, but this time we'll "blacklist" the ones we don't want to convert to float.
Step6:  Let's do some checks to make sure we've got a full set of values, by checking that we have N_countries by N_indicators.
Step7:  Awesome, we're good.
It's a bit unwieldy to have a bunch of columns.  We're going to join all the years, in order, so that we can have just a single column for that.  This column will be N-wide, where N is the number of years.
To do this, we'll create a list of arrays, and then use np.array to turn them into a single array.
Step8:  But it's the wrong shape, as we'll see
Step9:  So we'll transpose it (switching first and last axes) and then stick it into our dataset as the column "values".
Step10:  Just to make sure, let's double-check that our values column shows up.
Step11:  Let's get started.  I've picked out an indicator we can use, which is the population in the largest city.
Step12:  What's the population in just the US in this city over time?
Step13:  We'll need some years to plot against, so we will use np.arange to generate the values to supply to our plotting routines.  This routine drops off the final value, so we have to go to N+1 where N is the final value we want.
Step14:  Great.  That kind of works as we want.  Now let's clean it up, put it in a function, and look at a couple different countries.  And, while we're at it, we'll go ahead and make the indicator name a parameter, too.
We'll plug this into the ipywidgets.interact function momentarily.
Step15:  What countries and indicators do we have to choose from?  And, let's pick a couple at random.
Step16:  Now we'll test our routine.
Step17:  Note something here -- it's masking data that doesn't exist in the dataset.  Those points appear as discontinuities.  This isn't always what we want; for instance, we may instead wish to show those as a continuous, but irregularly spaced, line plot.  However, in this case, we want to emphasize the discontinuity.
Now, we'll use ipywidgets to change which country and indicator we plot.  This isn't incredibly useful, since the number of indicators is intimidatingly long, but we will use it as a quick way to explore some of the data.
Note that we're calling tolist() on these.  If we just feed in arrays, the interact function gets a bit confused.
Step18:  Let's just toss everything into a single plot.  This is a bad idea, and let's demonstrate why.
Step19:  Instead, we'll create a function that plots only those whose countries whose values were greater than those in the US in a given year.  For instance, if you specify 1968, only those countries whose population was greater than the US in 1968 will be plotted.
Step20:  Again, we'll use ipywidgets.interact to do this.
Step21:  Now let's see if we can get the top N countries for a given indicator.  Note that we're doing a couple things here that we've done in the past, but in a more optimized form.
Inside our routine, we're filtering out the indicator name that we're given.  We then compute the maximum along all years for each country, then we sort that maximum.  We then iterate over the sorted list, in opposite order, and plot those countries.
Step22:  Let's see which indicators might be interesting -- specifically, let's take a look at all the indicators that have "Population" in their name.
Step23:  We have lots of options.  So, let's filter out a couple of them -- we'll get the three components of population.  These should all add up to 100%, so we'll use them as an example of how to do stacked plots.
The three components we'll pull out are the age between 0-14, 15-64, and 65 and above.  We'll do this for the United States.  (But, remember, you could extend this to select the country as well!)
Step24:  We'll make a simple plot first, just to see the trends.
Step25:  Probably not a huge surprise, since the age range 15-64 is quite big.  Let's verify that these sum up to 100%.
Step26:  Yup, looks like.  Now we'll make a stackplot.  This is a plot where we're showing the area included in each, and putting one on top of the other.  This type of plot is useful for when you want to show the variation in composition over time of different quantities.
Step27:  We can plot both the line plots and the stacked plots next to each other to get a more visual comparison between them.
Step28:  What about things that don't "stack" to a constant value, but a value varying over time?  We can also use a stack plot here, which shows the trendlines, but note that it's not always going to be as clearly demonstrative of the relative percentages.  While it shows composition, extracting from that the specific composition is not trivial.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
Explanation: Examples: Week 7
This week, we will apply some of our discussions around filtering, splitting and so on to build out comparisons between different variables within the World Bank Economic Indicators dataset.
This dataset, which covers 1960-2016, 264 countries and 1452 variables, can be obtained at the Worldbank Data site.
End of explanation
import sys
sys.path.insert(0, "/srv/nbgrader/data/WDI/")
from my_dataset import Dataset, read_csv
import numpy as np
import matplotlib.pyplot as plt
import ipywidgets
plt.rcParams["figure.figsize"] = (6, 5)
Explanation: Building on our discussion of modules from last week, we'll use the my_dataset module that I have prepared as a basis.  This module is largely identical to what we have built out in previous weeks.
End of explanation
data = Dataset(read_csv("/srv/nbgrader/data/WDI/WDI_Data.csv",
                        encoding="cp1252"))
Explanation: We'll use a read_csv function that is provided in our module to feed into a Dataset object.  This loads in our full dataset, and we'll take a look at the columns it generates.
Note that we're using a different encoding.  In particular, this is because the file is encoded in cp1252, which is a Windows-specific extension that includes some characters that can't be interpreted directly in Unicode.  If we didn't specify this, it would crash.
End of explanation
data.columns()
Explanation: Let's see what data is available.
End of explanation
np.unique(data["Indicator Code"])
np.unique(data["Indicator Code"]).size
np.unique(data["Country Code"]).size
Explanation: Wow!  OK, so, we have a column for each year that data is available.  Other than that, we have an indicator code, a country code, and then the longer names of those two things.  How many indicator codes are there?  (Spoiler alert: I put the number up above already!)
End of explanation
mapping = {"Country Code": "str", "Country Name": "str",
           "Indicator Code": "str", "Indicator Name": "str"}
for c in data.columns():
    data.convert(c, mapping.get(c, "float"))
Explanation: We'll convert our columns like usual, but this time we'll "blacklist" the ones we don't want to convert to float.
End of explanation
data["1964"].shape
383328/264.
Explanation: Let's do some checks to make sure we've got a full set of values, by checking that we have N_countries by N_indicators.
End of explanation
values = []
for i in range(1960, 2017):
    values.append(data[str(i)])
values = np.array(values)
Explanation: Awesome, we're good.
It's a bit unwieldy to have a bunch of columns.  We're going to join all the years, in order, so that we can have just a single column for that.  This column will be N-wide, where N is the number of years.
To do this, we'll create a list of arrays, and then use np.array to turn them into a single array.
End of explanation
values.shape
Explanation: But it's the wrong shape, as we'll see:
End of explanation
values = values.transpose()
data.data["values"] = values
Explanation: So we'll transpose it (switching first and last axes) and then stick it into our dataset as the column "values".
End of explanation
data.columns()
Explanation: Just to make sure, let's double-check that our values column shows up.
End of explanation
# EN.URB.LCTY
pop_city = data.filter_eq("Indicator Code", "EN.URB.LCTY")
pop_city["values"].shape
Explanation: Let's get started.  I've picked out an indicator we can use, which is the population in the largest city.
End of explanation
us_pop_city = pop_city.filter_eq("Country Name", 'United States')
Explanation: What's the population in just the US in this city over time?
End of explanation
years = np.arange(1960, 2017)
plt.plot(years, us_pop_city["values"][0,:], '.-')
Explanation: We'll need some years to plot against, so we will use np.arange to generate the values to supply to our plotting routines.  This routine drops off the final value, so we have to go to N+1 where N is the final value we want.
End of explanation
def plot_country_indicator(country, indicator):
    indicator_value = data.filter_eq("Indicator Name", indicator)
    country_value = indicator_value.filter_eq("Country Name", country)
    plt.plot(years, country_value["values"][0,:], '.-')
    plt.xlabel("Year")
    plt.ylabel(indicator)
Explanation: Great.  That kind of works as we want.  Now let's clean it up, put it in a function, and look at a couple different countries.  And, while we're at it, we'll go ahead and make the indicator name a parameter, too.
We'll plug this into the ipywidgets.interact function momentarily.
End of explanation
indicators = np.unique(data["Indicator Name"])
countries = np.unique(data["Country Name"])
indicators[50], countries[100]
Explanation: What countries and indicators do we have to choose from?  And, let's pick a couple at random.
End of explanation
plot_country_indicator(countries[100], indicators[50])
Explanation: Now we'll test our routine.
End of explanation
ipywidgets.interact(plot_country_indicator,
                   country=countries.tolist(),
                    indicator=indicators.tolist())
Explanation: Note something here -- it's masking data that doesn't exist in the dataset.  Those points appear as discontinuities.  This isn't always what we want; for instance, we may instead wish to show those as a continuous, but irregularly spaced, line plot.  However, in this case, we want to emphasize the discontinuity.
Now, we'll use ipywidgets to change which country and indicator we plot.  This isn't incredibly useful, since the number of indicators is intimidatingly long, but we will use it as a quick way to explore some of the data.
Note that we're calling tolist() on these.  If we just feed in arrays, the interact function gets a bit confused.
End of explanation
plt.plot(years, pop_city["values"].transpose())
Explanation: Let's just toss everything into a single plot.  This is a bad idea, and let's demonstrate why.
End of explanation
def greater_than_year(year = 2010):
    year = str(year)
    greater_than_us = pop_city.filter_gt(year, us_pop_city[year])
    for i, country in enumerate(greater_than_us["Country Name"]):
        plt.plot(years, greater_than_us["values"][i,:], label=country)
    plt.ylim(0, np.nanmax(pop_city["values"]))
    plt.legend()
Explanation: Instead, we'll create a function that plots only those whose countries whose values were greater than those in the US in a given year.  For instance, if you specify 1968, only those countries whose population was greater than the US in 1968 will be plotted.
End of explanation
ipywidgets.interact(greater_than_year, year = (1960, 2016))
Explanation: Again, we'll use ipywidgets.interact to do this.
End of explanation
def plot_country_indicator_top(indicator, top = 5):
    indicator_value = data.filter_eq("Indicator Name", indicator)
    max_value = np.nanmax(indicator_value["values"], axis=1)
    max_value[np.isnan(max_value)] = -1e90
    max_indices = np.argsort(max_value)
    for ind in reversed(max_indices[-top:]):
        plt.plot(years, indicator_value["values"][ind,:], '.-',
                label=indicator_value["Country Name"][ind])
    plt.xlabel("Year")
    plt.legend()
    plt.ylabel(indicator)
ipywidgets.interact(plot_country_indicator_top,
                   country=countries.tolist(),
                    indicator=indicators.tolist())
Explanation: Now let's see if we can get the top N countries for a given indicator.  Note that we're doing a couple things here that we've done in the past, but in a more optimized form.
Inside our routine, we're filtering out the indicator name that we're given.  We then compute the maximum along all years for each country, then we sort that maximum.  We then iterate over the sorted list, in opposite order, and plot those countries.
End of explanation
for indicator in indicators:
    if "Population" in indicator:
        print(indicator)
Explanation: Let's see which indicators might be interesting -- specifically, let's take a look at all the indicators that have "Population" in their name.
End of explanation
united_states = data.filter_eq("Country Name", "United States")
us_pop_0014 = united_states.filter_eq("Indicator Name", "Population ages 0-14 (% of total)")
us_pop_1564 = united_states.filter_eq("Indicator Name", "Population ages 15-64 (% of total)")
us_pop_6500 = united_states.filter_eq("Indicator Name", "Population ages 65 and above (% of total)")
Explanation: We have lots of options.  So, let's filter out a couple of them -- we'll get the three components of population.  These should all add up to 100%, so we'll use them as an example of how to do stacked plots.
The three components we'll pull out are the age between 0-14, 15-64, and 65 and above.  We'll do this for the United States.  (But, remember, you could extend this to select the country as well!)
End of explanation
plt.plot(years, us_pop_0014["values"][0,:], label="0-14")
plt.plot(years, us_pop_1564["values"][0,:], label="15-64")
plt.plot(years, us_pop_6500["values"][0,:], label="65+")
plt.legend()
plt.xlabel("Years")
plt.ylabel("Percentage")
Explanation: We'll make a simple plot first, just to see the trends.
End of explanation
us_pop_0014["values"][0,:] + us_pop_1564["values"][0,:] + us_pop_6500["values"][0,:]
Explanation: Probably not a huge surprise, since the age range 15-64 is quite big.  Let's verify that these sum up to 100%.
End of explanation
plt.stackplot(years,
              us_pop_0014["values"][0,:],
              us_pop_1564["values"][0,:],
              us_pop_6500["values"][0,:],
             labels = ["0-14", "15-64", "65+"])
plt.legend(loc="center left")
plt.xlabel("Years")
plt.ylabel("Percentage")
plt.xlim(1960, 2015)
plt.ylim(0, 100)
Explanation: Yup, looks like.  Now we'll make a stackplot.  This is a plot where we're showing the area included in each, and putting one on top of the other.  This type of plot is useful for when you want to show the variation in composition over time of different quantities.
End of explanation
baseline = us_pop_1564["values"][0,:]
plt.subplot(2,1,1)
plt.stackplot(years,
              us_pop_0014["values"][0,:],
              us_pop_1564["values"][0,:],
              us_pop_6500["values"][0,:],
             labels = ["0-14", "15-64", "65+"])
plt.legend(loc="center left")
plt.xlabel("Years")
plt.ylabel("Percentage")
plt.xlim(1960, 2015)
plt.ylim(0, 100)
plt.subplot(2,1,2)
plt.plot(years, us_pop_0014["values"][0,:]/baseline)
plt.plot(years, us_pop_1564["values"][0,:]/baseline)
plt.plot(years, us_pop_6500["values"][0,:]/baseline)
Explanation: We can plot both the line plots and the stacked plots next to each other to get a more visual comparison between them.
End of explanation
pop_urban = united_states.filter_eq("Indicator Name", "Population in urban agglomerations of more than 1 million")
pop_total = united_states.filter_eq("Indicator Name", "Population, total")
pop_urban = pop_urban["values"][0,:]
pop_total = pop_total["values"][0,:]
pop_non_urban = pop_total - pop_urban
plt.stackplot(years, pop_non_urban, pop_urban, labels=["Non-Urban", "Urban"])
plt.xlabel("Years")
plt.ylabel("People")
plt.legend()
Explanation: What about things that don't "stack" to a constant value, but a value varying over time?  We can also use a stack plot here, which shows the trendlines, but note that it's not always going to be as clearly demonstrative of the relative percentages.  While it shows composition, extracting from that the specific composition is not trivial.
End of explanation
<END_TASK>