<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  Overview
Step2:  Since we have 5000 sample cookies instead of the original 40000, we can adjust accordingly using calculate probability. For these two evaluation metric, we need number of users who click "Start Now" button, and calculated as
Step3:  Next, standard deviation for Gross conversion is
Step4:  and for Net Conversion,
Step5:  Gross Conversion and Net Conversion, their empirical variance should approximate analytical variance, because the unit of analysis and unit of diversion is the same, cookie-ids/user-ids.
Sizing
Number of Samples vs. Power
Gross Conversion. Baseline
Step6:  Duration vs. Exposure
Fraction
Step7:  Next, we count the total views and clicks for both control and experiment groups.
Step8:  For count like number of cookies and number of clicks in "Start free trial" button, we can do confidence interval around the fraction we expect in control group, and actual fraction as the observed outcome. Since we expect control and experiment to have equal proportion, we set the the expected proportion to be 0.5. Both invariant metrics, the confidence interval for sanity checks use the function below.
Step9:  Now for sanity checks confidence interval of number of cookies who views the page,
Step10:  The actual proportion is
Step11:  Since we know that 0.5006 is within the interval, then experiment pass sanity checks for number of cookies.
Next, we calculate confidence interval of number of clicks at "Start free trial" button.
Step12:  And the actual proportion,
Step13:  Again 0.5006 is within the interval, so our experiment also pass the sanity check.
For our sanity check with ctp, is a little different calculation. Using simple count earlier, we know that if we setup our experiment in a proper way, the true proportion of control group should be 0.5. Since we don't know the true proportion of ctp control group, we build confidence interval around the control group, and ctp experiment as observed outcome. If the experiment change and ctp experiment is outside ctp control confidence interval, then our experiment failed sanity checks. Thus we can't continue our analysis.
Step14:  And as you can see, click-through-probability of the experiment is still within the confidence interval of click-through-probability control groups. Since we have passed all of the sanity checks, we can continue to analyze the experiment.
Effect Size Test
Did not use Bonferroni correction
Gross Conversion
Bounds =  (-0.0291, -0.0120)
Statistical Significance? Yes
Practical Significance? Yes
Net Conversion
Bounds =  (-0.0116,0.0019)
Statistical Significance? No
Practical Significance? No
Step15:  Gross Conversion
Keep in mind that observed_difference can be negative
Step16:  The observed difference is outside the confidence interval. And the observed difference also above 0.01 dmin, minimum detectable effect. We should definitely launch.
Net Conversion
Step17:  The observed difference is within the confidence interval so it's not statiscally significant and also not practically significant. We may fail or continue with our results.
Sign Test
Did not use Bonferroni correction
Gross Conversion
p-value = 0.0026
Statistical Significance? Yes
Net Conversion
p-value = 0.6776
Statistical Significance? No
Sign Test is also a test that must be confirmed with effect size test. I'm using Online Calculator to calculate the  binomial p-value, whether the probability of experiment is higher than control groups. If we simulate it, what ar the odds. If the probability is so rare, that isn't likely due to chance, then the experiment succeed, provided significance level, which I choose to be 5%.
I'm using helper function, to compare probability day-to-day whether the metric in question is smaller for group than the experiment.
Step18:  Gross Conversion
Count the gross conversion, I got,
Step19:  Net Conversion
<ASSISTANT_TASK:>
Python Code:
baselines= Unique cookies to view page per day:	40000
Unique cookies to click "Start free trial" per day:	3200
Enrollments per day:	660
Click-through-probability on "Start free trial":	0.08
Probability of enrolling, given click:	0.20625
Probability of payment, given enroll:	0.53
Probability of payment, given click:	0.1093125
lines  = baselines.split('\n')
d_baseline = dict([(e.split(':\t')[0],float(e.split(':\t')[1])) for e in lines])
Explanation: Overview: Free Trial Screener
At the time of this experiment, Udacity courses currently have two options on the home page: "start free trial", and "access course materials". If the student clicks "start free trial", they will be asked to enter their credit card information, and then they will be enrolled in a free trial for the paid version of the course. After 14 days, they will automatically be charged unless they cancel first. If the student clicks "access course materials", they will be able to view the videos and take the quizzes for free, but they will not receive coaching support or a verified certificate, and they will not submit their final project for feedback.
In the experiment, Udacity tested a change where if the student clicked "start free trial", they were asked how much time they had available to devote to the course. If the student indicated 5 or more hours per week, they would be taken through the checkout process as usual. If they indicated fewer than 5 hours per week, a message would appear indicating that Udacity courses usually require a greater time commitment for successful completion, and suggesting that the student might like to access the course materials for free. At this point, the student would have the option to continue enrolling in the free trial, or access the course materials for free instead. This screenshot shows what the experiment looks like.
The hypothesis was that this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time—without significantly reducing the number of students to continue past the free trial and eventually complete the course. If this hypothesis held true, Udacity could improve the overall student experience and improve coaches' capacity to support students who are likely to complete the course.
The unit of diversion is a cookie, although if the student enrolls in the free trial, they are tracked by user-id from that point forward. The same user-id cannot enroll in the free trial twice. For users that do not enroll, their user-id is not tracked in the experiment, even if they were signed in when they visited the course overview page.
<!--TEASER_END-->
Background
Start free trial to allow paid version
Access course materials for free
experiment
users see this popup after click
if they're not committed, warning them.
We have two initial hypothesis.
“..this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time”
”..without significantly reducing the number of students to continue past the free trial and eventually complete the course. “
Unit of diversion: cookie
users in free trial tracked by user-id
same user-id can't enroll in free trial twice
users not enroll can't tracked.
Experiment Design
Metric Choice
Invariant metrics
Number of cookies
Number of clicks
Click through probability
Sanity Check is useful when we want to make sure that the data filtered for experiment and control group is the same. This can be done using the right invariance metric. These three metrics shouldn't change because it's outside of the experiment, in a sense that these metric calculated all before the experiment begin.
Number of cookies who views the page should be the same when Udacity experiment. They haven't click the "Start Now" button and see "Free Trial Screener" experiment. So number of cookies can be used as invariant metrics. When users click the button, they also haven't yet see the experiment that Udacity does, so number of clicks shouldn't change between experiment and control groups.
Since the experiments only occurs after the users click the "Start Now" button, its click-through-probability also have to be the same for each experiment and control group. We know that number of cookies and number of clicks has to be the same, then click-thorough-probability also has to be the same.
Besides cookie-id, there is also user-id. But user-id is not a good invariant, because Udacity also open to unregistered users to view page until after click of a button.
Evaluation metrics
Gross Conversion
Net Conversion
We have two initial hypothesis.
“..this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time”
”..without significantly reducing the number of students to continue past the free trial and eventually complete the course. “
For evaluation metrics, I choose Gross Conversion, Retention, and Net Conversion. All of these metrics are a good evaluation metrics since they change when the experiment change, and since each of the metrics has user-ids as the unit of analysis, should be much smaller standard error since Udacity also using it as the cookie of diversion. 
Gross conversion is the number of user-ids to complete checkout and enroll in the free trial divided by number of unique cookies to click the "Start free trial" button. After the visitors click the button, they should see the screener, hence the warning. It should be makes other visitors that doesn't have serious commitment back down and cancel it right away.
Retention is number of user-ids to remain enrolled past the 14-day boundary (and thus make at least one payment) divided by number of user-ids to complete checkout. The experiment intend to focus the visitors that only want to make a serious commitment. The retention rate should be higher for experiment group than the control group.
Net conversion is number of user-ids to remain enrolled past the 14-day boundary (and thus make at least one payment) divided by the number of unique cookies to click the "Start free trial" button. Net Conversion also true, since the experiment intend to see higher conversion rate for students to continue (at least make one payment) than the users that only click the button, that doesn't even see the warning experiment given.
Ouf of these metrics, Retention turns out have a longer duration, which is 118 days. This takes too long, and it's not something Udacity willing to give for the experiment. So Retention will be excluded.
The first part is what Gross Conversion does, we should expect after the experiment, Gross Conversion shows significantly reduce the number who left trial because they don’t have time commitment. The experiment group should be significantly different than control group.
The second part is what Net Conversion does, we should expect after experiment, the metric shows insignificantly reduce the number of students who at least make one payment.  The experiment group should not significantly different than control group.
Measuring Standard Deviation
Gross Conversion: 0.02
Net Conversion 0.0156
Expect analytical variance match empirical variance because unit of analysis and unit of diversion is same.
To calculate standard deviation, we use this formula
Python
Formula = np.sqrt(p * (1-p) / n)
and using baseline data below.
End of explanation
n = 5000
n_click = n * d_baseline['Click-through-probability on "Start free trial"']
n_click
Explanation: Since we have 5000 sample cookies instead of the original 40000, we can adjust accordingly using calculate probability. For these two evaluation metric, we need number of users who click "Start Now" button, and calculated as
End of explanation
p = d_baseline['Probability of enrolling, given click']
round(np.sqrt(p * (1-p) / n_click),4)
Explanation: Next, standard deviation for Gross conversion is
End of explanation
p = d_baseline['Probability of payment, given click']
round(np.sqrt(p * (1-p) / n_click),4)
Explanation: and for Net Conversion,
End of explanation
(27411 * 2) / d_baseline['Click-through-probability on "Start free trial"']
Explanation: Gross Conversion and Net Conversion, their empirical variance should approximate analytical variance, because the unit of analysis and unit of diversion is the same, cookie-ids/user-ids.
Sizing
Number of Samples vs. Power
Gross Conversion. Baseline: 0.20625 dmin: 0.01 = 25.839 cookies who clicks.
Net Conversion. Baseline: 0.1093125 dmin: 0.0075 = 27,411 cookies who clicks.
Not using Bonferroni correction.
Using alpha = 0.05 and beta 0.2
The pageviews needed then will be:
685275 impression.
We feed it into sample size calculator. 
We can use bigger number, so the minimum required cookies is sufficient. The sample size is only for one group, so output from the calculator must be doubled to get the enough pageviews. Since this only the user who clicks, we calculate number of pageviews using CTP. The pageviews needed then will be:
End of explanation
control = pd.read_csv('control_data.csv')
experiment = pd.read_csv('experiment.csv')
control.head()
experiment.head()
Explanation: Duration vs. Exposure
Fraction: 0.8 (Low risk)
Duration: 22 days (40000 pageviews/day)
The fraction of experiment exposure to Udacity visitors will be 80%. The experiment isn't risky enough that may potentially leaked as blog news or article. It doesn't really big a news, as Udacity only want to put little warning to the users. Because only 40000 pageviews each day can be gathered, the duration will be 22 days.
This is where Retention metric fail for our evaluation metrics. It has a longer duration, which is 118 days. This takes too long, and it's not something Udacity willing to give for the experiment. So Retention will be excluded.
Experiment Analysis
Sanity Checks
Number of Cookies:
Bounds = (0.4988,0.5012)
Observed = 0.5006
Passes? Yes
Number of clicks on “Start free trial”:
Bounds = (0.4959,0.5041)
Observed = 0.5005
Passes? Yes
Click-through-probability on “Start free trial”:
Bounds = (0.0812,0.0830)
Observed = 0.0821
Passes? Yes
Since we have passed all of the sanity checks, we can continue to analyze the experiment.
We do sanity checks to ensure that both experiment and control groups have equal proportion. It's the metric that shouldn't change when experiment change, which is invariant metrics that we chose earlier. First let's see the data that we want to analyze both at control and experiment.
End of explanation
control_views = control.Pageviews.sum()
control_clicks = control.Clicks.sum()
experiment_views = experiment.Pageviews.sum()
experiment_clicks = experiment.Clicks.sum()
Explanation: Next, we count the total views and clicks for both control and experiment groups.
End of explanation
def sanity_check_CI(control,experiment,expected):
    SE = np.sqrt((expected*(1-expected))/(control + experiment))
    ME = 1.96 * SE
    return (expected-ME,expected+ME)
    
Explanation: For count like number of cookies and number of clicks in "Start free trial" button, we can do confidence interval around the fraction we expect in control group, and actual fraction as the observed outcome. Since we expect control and experiment to have equal proportion, we set the the expected proportion to be 0.5. Both invariant metrics, the confidence interval for sanity checks use the function below.
End of explanation
sanity_check_CI(control_views,experiment_views,0.5)
Explanation: Now for sanity checks confidence interval of number of cookies who views the page,
End of explanation
float(control_views)/(control_views+experiment_views)
Explanation: The actual proportion is
End of explanation
sanity_check_CI(control_clicks,experiment_clicks,0.5)
Explanation: Since we know that 0.5006 is within the interval, then experiment pass sanity checks for number of cookies.
Next, we calculate confidence interval of number of clicks at "Start free trial" button.
End of explanation
float(control_clicks)/(control_clicks+experiment_clicks)
Explanation: And the actual proportion,
End of explanation
ctp_control = float(control_clicks)/control_views
ctp_experiment = float(experiment_clicks)/experiment_views
# %%R
c = 28378
n = 345543
CL = 0.95
pe = c/n
SE = sqrt(pe*(1-pe)/n)
z_star = round(qnorm((1-CL)/2,lower.tail=F),digits=2)
ME = z_star * SE
c(pe-ME, pe+ME)
ctp_experiment
Explanation: Again 0.5006 is within the interval, so our experiment also pass the sanity check.
For our sanity check with ctp, is a little different calculation. Using simple count earlier, we know that if we setup our experiment in a proper way, the true proportion of control group should be 0.5. Since we don't know the true proportion of ctp control group, we build confidence interval around the control group, and ctp experiment as observed outcome. If the experiment change and ctp experiment is outside ctp control confidence interval, then our experiment failed sanity checks. Thus we can't continue our analysis.
End of explanation
get_gross = lambda group: float(group.dropna().Enrollments.sum())/ group.Clicks.sum()
get_net = lambda group: float(group.dropna().Payments.sum())/ group.Clicks.sum()
Explanation: And as you can see, click-through-probability of the experiment is still within the confidence interval of click-through-probability control groups. Since we have passed all of the sanity checks, we can continue to analyze the experiment.
Effect Size Test
Did not use Bonferroni correction
Gross Conversion
Bounds =  (-0.0291, -0.0120)
Statistical Significance? Yes
Practical Significance? Yes
Net Conversion
Bounds =  (-0.0116,0.0019)
Statistical Significance? No
Practical Significance? No
End of explanation
print('N_cont = %i'%control.dropna().Clicks.sum())
print('X_cont = %i'%control.dropna().Enrollments.sum())
print('N_exp =  %i'%experiment.dropna().Clicks.sum())
print('X_exp =  %i'%experiment.dropna().Enrollments.sum())
X_exp/N_exp
X_cont/N_cont
#%%R
N_cont = 17293
X_cont = 3785
N_exp = 17260
X_exp = 3423
observed_diff = X_exp/N_exp - X_cont/N_cont
# print(observed_diff)
p_pool = (X_cont+X_exp)/(N_cont+N_exp)
SE = sqrt( (p_pool*(1-p_pool)) * ((1/N_cont) + (1/N_exp)))
ME = 1.96 * SE
# print(p_pool)
c(observed_diff-ME, observed_diff+ME)
observed_diff
Explanation: Gross Conversion
Keep in mind that observed_difference can be negative
End of explanation
print('N_cont = %i'%control.dropna().Clicks.sum())
print('X_cont = %i'%control.dropna().Payments.sum())
print('N_exp =  %i'%experiment.dropna().Clicks.sum())
print('X_exp =  %i'%experiment.dropna().Payments.sum())
X_exp/N_exp
X_cont/N_cont
#%%R
N_cont = 17293
X_cont = 2033
N_exp = 17260
X_exp = 1945
observed_diff = X_exp/N_exp - X_cont/N_cont
# print(observed_diff)
p_pool = (X_cont+X_exp)/(N_cont+N_exp)
SE = sqrt( (p_pool*(1-p_pool)) * ((1/N_cont) + (1/N_exp)))
ME = 1.96 * SE
# print(p_pool)
c(observed_diff-ME, observed_diff+ME)
observed_diff
Explanation: The observed difference is outside the confidence interval. And the observed difference also above 0.01 dmin, minimum detectable effect. We should definitely launch.
Net Conversion
End of explanation
compare_prob = lambda col: ((control.dropna()[col] / control.dropna().Clicks) <
                            (experiment.dropna()[col]/experiment.dropna().Clicks))
                                   
Explanation: The observed difference is within the confidence interval so it's not statiscally significant and also not practically significant. We may fail or continue with our results.
Sign Test
Did not use Bonferroni correction
Gross Conversion
p-value = 0.0026
Statistical Significance? Yes
Net Conversion
p-value = 0.6776
Statistical Significance? No
Sign Test is also a test that must be confirmed with effect size test. I'm using Online Calculator to calculate the  binomial p-value, whether the probability of experiment is higher than control groups. If we simulate it, what ar the odds. If the probability is so rare, that isn't likely due to chance, then the experiment succeed, provided significance level, which I choose to be 5%.
I'm using helper function, to compare probability day-to-day whether the metric in question is smaller for group than the experiment.
End of explanation
compare_prob('Enrollments').value_counts()
Explanation: Gross Conversion
Count the gross conversion, I got,
End of explanation
compare_prob('Payments').value_counts()
Explanation: Net Conversion
End of explanation
<END_TASK>