<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Module 10
Step1:  Ratio and logarithm
If you use linear scale to visualize ratios, it can be quite misleading.
Let's first create some ratios.
Step2:  Q
Step3:  Q
Step4:  Log-binning
Let's first see what happens if we do not use the log scale for a dataset with a heavy tail. 
Q
Step5:  If you simply call hist() method with a dataframe object, it identifies all the numeric columns and draw a histogram for each. 
Q
Step6:  As we can see, a majority of the columns are not normally distributed. In particular, if you look at the worldwide gross variable, you only see a couple of meaningful data from the histogram. Is this a problem of resolution? How about increasing the number of bins?
Step7:  Maybe a bit more useful, but it doesn't tell anything about the data distribution above certain point. How about changing the vertical scale to logarithmic scale?
Step8:  Now, let's try log-bin. Recall that when plotting histgrams we can specify the edges of bins through the bins parameter. For example, we can specify the edges of bins to [1, 2, 3, ... , 10] as follows.
Step9:  Here, we can specify the edges of bins in a similar way. Instead of specifying on the linear scale, we do it on the log space. Some useful resources
Step10:  Because there seems to be movie(s) that made $0, and because log(0) is undefined & log(1) = 0, let's add 1 to the variable.
Step11:  Now we can plot a histgram with log-bin. Set both axis to be log-scale.
Step12:  What is going on? Is this the right plot?
Q
Step13:  Q
Step14:  We can also try in semilog scale (only one axis is in a log-scale), where the horizontal axis is linear.
Step15:  A straight line in semilog scale means exponential decay (cf. a straight line in log-log scale means power-law decay). So it seems like the amount of money a movie makes across the world follows roughly an exponential distribution, while there are some outliers that make insane amount of money. 
Q
<ASSISTANT_TASK:>
Python Code:
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import numpy as np
import scipy.stats as ss
import vega_datasets
Explanation: Module 10: Logscale
End of explanation
x = np.array([1,    1,   1,  1, 10, 100, 1000])
y = np.array([1000, 100, 10, 1, 1,  1,   1   ])
ratio = x/y
print(ratio)
Explanation: Ratio and logarithm
If you use linear scale to visualize ratios, it can be quite misleading.
Let's first create some ratios.
End of explanation
X = np.arange(len(ratio))
# Implement
Explanation: Q: Plot on the linear scale using the scatter() function. Also draw a horizontal line at ratio=1 for a reference.
End of explanation
# Implement
Explanation: Q: Is this a good visualization of the ratio data? Why? Why not? Explain.
Q: Can you fix it?
End of explanation
# TODO: Implement the functionality mentioned above 
# The following code is just a dummy. You should load the correct dataset from vega_datasets package. 
movies = pd.DataFrame({"Worldwide_Gross": np.random.sample(200), "IMDB_Rating": np.random.sample(200)})
Explanation: Log-binning
Let's first see what happens if we do not use the log scale for a dataset with a heavy tail. 
Q: Load the movie dataset from vega_datasets and remove the NaN rows based on the following three columns: IMDB_Rating, IMDB_Votes, Rotten_Tomatoes_Rating.
End of explanation
# Implement
Explanation: If you simply call hist() method with a dataframe object, it identifies all the numeric columns and draw a histogram for each. 
Q: draw all possible histograms of the movie dataframe. Adjust the size of the plots if needed.
End of explanation
ax = movies["Worldwide_Gross"].hist(bins=200)
ax.set_xlabel("World wide gross")
ax.set_ylabel("Frequency")
Explanation: As we can see, a majority of the columns are not normally distributed. In particular, if you look at the worldwide gross variable, you only see a couple of meaningful data from the histogram. Is this a problem of resolution? How about increasing the number of bins?
End of explanation
ax = movies["Worldwide_Gross"].hist(bins=200)
ax.set_yscale('log')
ax.set_xlabel("World wide gross")
ax.set_ylabel("Frequency")
Explanation: Maybe a bit more useful, but it doesn't tell anything about the data distribution above certain point. How about changing the vertical scale to logarithmic scale?
End of explanation
movies["IMDB_Rating"].hist(bins=range(0,11))
Explanation: Now, let's try log-bin. Recall that when plotting histgrams we can specify the edges of bins through the bins parameter. For example, we can specify the edges of bins to [1, 2, 3, ... , 10] as follows.
End of explanation
min(movies["Worldwide_Gross"])
Explanation: Here, we can specify the edges of bins in a similar way. Instead of specifying on the linear scale, we do it on the log space. Some useful resources:
Google query: python log-bin
numpy.logspace
numpy.linspace vs numpy.logspace
Hint: since $10^{\text{start}} = \text{min(Worldwide_Gross)}$, $\text{start} = \log_{10}(\text{min(Worldwide_Gross)})$
End of explanation
movies["Worldwide_Gross"] = movies["Worldwide_Gross"]+1.0
# TODO: Replace the dummy value of bins using np.logspace.  
# Create 20 bins that cover the whole range of the dataset. 
bins = [1.0, 2.0, 4.0]
bins
Explanation: Because there seems to be movie(s) that made $0, and because log(0) is undefined & log(1) = 0, let's add 1 to the variable.
End of explanation
ax = (movies["Worldwide_Gross"]+1.0).hist(bins=bins)
ax.set_yscale('log')
ax.set_xscale('log')
ax.set_xlabel("World wide gross")
ax.set_ylabel("Frequency")
Explanation: Now we can plot a histgram with log-bin. Set both axis to be log-scale.
End of explanation
# Implement
Explanation: What is going on? Is this the right plot?
Q: explain and fix
End of explanation
# TODO: Implement functionality mentioned above
# You must replace the dummy values with the correct code. 
worldgross_sorted = np.random.sample(200)
Y = np.random.sample(200)
Explanation: Q: Can you explain the plot? Why are there gaps?
CCDF
CCDF is a nice alternative to examine distributions with heavy tails. The idea is same as CDF, but the direction of aggregation is opposite. For a given value x, CCDF(x) is the number (fraction) of data points that are same or larger than x. To write code to draw CCDF, it'll be helpful to draw it by hand by using a very small, toy dataset. Draw it by hand and then think about how each point in the CCDF plot can be computed. 
Q: Draw a CCDF of worldwide gross data in log-log scale
End of explanation
plt.xlabel("World wide gross")
plt.ylabel("CCDF")
plt.plot(worldgross_sorted,Y)
plt.yscale('log')
Explanation: We can also try in semilog scale (only one axis is in a log-scale), where the horizontal axis is linear.
End of explanation
# Implement
Explanation: A straight line in semilog scale means exponential decay (cf. a straight line in log-log scale means power-law decay). So it seems like the amount of money a movie makes across the world follows roughly an exponential distribution, while there are some outliers that make insane amount of money. 
Q: Which is the most successful movie in our dataset?
You can use the following
idxmax(): https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.idxmax.html
loc: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html
End of explanation
<END_TASK>