<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 The Meterstick package provides a concise and flexible syntax to describe and execute
routine data analysis tasks. The easiest way to learn to use Meterstick is by example.
For External users
You can open this notebook in Google Colab.
Installation
You can install from pip for the stable version
Step1:  or from GitHub for the latest version.
Step2:  Demo Starts
Step3:  Simple Metrics
There are many built-in simple Metrics in Meterstick. They directly operate on a DataFrame.
Sum
Step4:  Count
Step5:  Dot (inner product)
Step6:  It can also be normalized.
Step7:  Max
Step8:  Min
Step9:  Mean
Step10:  Weighted Mean
Step11:  Quantile
Step12:  Interpolation
You can specify how you want to interpolate the quantile. It could be any of (‘linear’, ‘lower’, ‘higher’, ‘midpoint’, ‘nearest’).
Step13:  Weighted Quantile
Step14:  Variance
Step15:  Biased Variance
The default Variance is unbiased, namely, the divisor used in calculations is N - 1. You could set unbiased=False to use N as the divisor.
Step16:  Weighted Variance
Step17:  Standard Deviation
Step18:  Biased Standard Deviation
Similar to biased Variance, it's possible to compute biased standard deviation.
Step19:  Weighted Standard Deviation
Step20:  Coefficient of Variation
Step21:  Correlation
Step22:  Weighted Correlation
Step23:  Covariance
Step24:  Weighted Covariance
Step25:  Slicing
You can group your DataFrame and compute the Metrics on slices.
Step26:  Multiple Metrics
You can put multiple Metrics into a MetricList and compute them together. It's not only makes your codes terser, it might make the computation much faster. See Caching section for more infomation.
Step27:  Arithmetic of Metrics
You can do many arithmetic operations on Metrics. It can also be between a Metric and a scalar. You can call set_name() to give your composite Metric a new name. Internally, we operate on the results returned by Metrics with return_dataframe=False to avoid incompatible DataFrame columns names. However, if both Metrics return DataFrames even when return_dataframe is set to False, you might get lots of NAs. The solution is use rename_columns() to unify the column names. See section "Compare the standard errors between Jackknife and Bootstrap" for an example.
Add
Step28:  Divide
Step29:  Ratio
Since division between two Sums is common, we make a Ratio() Metric as a syntax sugar. Its third arg is the name for the Metric and is optional.
Step30:  We also support many other common arithmetic operations.
Step31:  Output Format
There are two options for you to control the format of the return.
1. return_dataframe
Step32:  Operations
An Operation is a special type of Metric that is built on top of another Metric (called a "child").  A Metric is anything that has the compute_on() method, so the child doesn't need to be a simple Metric like Sum. It could be a MetricList, a composite Metric, or even another Operation.
Distribution
Compute the child Metric on a DataFrame grouped by a column, then normalize the numbers to 1 within group.
Step33:  It's equal to
Step34:  Distribution has an alias Normalize.
Step35:  Cumulative Distribution
Similar to Distribution except that it returns the cumulative sum after normalization, but unlike Distribution the order of the cumulating column matters. As the result, we always sort the column and there is an 'order' arg for you to customize the ordering.
Step36:  PercentChange
Computes the percent change to a certain group on the DataFrame returned by the child Metric. The returned value is the # of percent points.
Step37:  You can include the base group in your result.
Step38:  You can also specify multiple columns as the condition columns, then your base value should be a tuple.
Step39:  Absolute Change
Very similar to PercentChange, but the absolute difference is returned.
Step40:  You can also include the base group in your result.
Step41:  Cochran-Mantel-Haenszel statistics
Please refer to the Wikepedia page for its definition. Besides the condition column and baseline key that PercentChange and AbsoluteChange take, CMH also needs a column to  stratify. The child Metric must be a ratio of two single-column Metrics or CMH doesn't make sense. So instead of passing
AbsoluteChange(MetricList([a, b])) / AbsoluteChange(MetricList([c, d])),
please use
MetricList([AbsoluteChange(a) / AbsoluteChange(c),
          AbsoluteChange(b) / AbsoluteChange(d)]).
Step42:  CUPED
It computes the absolute change that has been adjusted using the CUPED approach. It provides an unbiased estimate to the absolute change with lower variance.
Let's see how it works on a fake data with preperiod metrics that are correlated with postperiod metrics and the effect of the experiment is small and noisy.
Step43:  CUPED essentially fits a linear model of Postperiod metric ~ 1 + preperiod metric and uses it to control for the variance in the preperiod.
Step44:  We can see that CUPED's result is similar to the absolute change but has smaller variance.
Step45:  PrePostChange
It computes the percent change that has been adjusted using the PrePost approach. It's similar to CUPED but control for treatment groups additionally. Essentially, it fits
Postperiod metric ~ 1 + is_treated * preperiod metric, or more verbosely,
Postperiod metric = β1 + β2 * is_treated  + β3 * preperiod metric + β4 * is_treated * preperiod metric.
Note that the estimate of β2 will be the estimate of treatment effect and the control arm metric can be estimated using β1 if we centered preperiod metric. As the result, β2 / β1 will be the estimate of the percent change that PrePostChange returns.
Step46:  Standard Errors
Jackknife
Unlike all Metrics we have seen so far, Jackknife returns a multiple-column DataFrame because by default we return point estimate and standard error.
Step47:  You can also specify a confidence level, the we'll return the confidence interval. The returned DataFrame also comes with a display() method for visualization which will highlight significant changes. To customize the display(), please take a look at confidence_interval_display_demo.ipynb.
Step48:  Bootstrap
The output is similar to Jackknife. The different args are
  - unit
Step49:  Models
Meterstick also has built-in support for model fitting. The module is not imported by default, so you need to manually import it.
Step50:  Linear Regression
Step51:  What Model(y, x, groupby).compute_on(data) does is
1. Computes MetricList((y, x)).compute_on(data, groupby).
2. Fits the underlying sklearn model on the result from #1.
Step52:  Ridge Regression
Step53:  Lasso Regression
Step54:  Logistic Regression
Step55:  If y is not binary, by default a multinomal model is fitted. The behavior can be controlled via the 'multinomial' arg.
Step56:  Classes are the unique values of y.
Step57:  Wrapping sklearn models into Meterstick provides the ability to combine Models with other built-in Metrics and Operations. For example, you can Jackknife the Model to get the uncertainty of coefficients.
Step58:  Pipeline
You have already seen this. Instead of
Jackknife(PercentChange(MetricList(...))).compute_on(df)
you can write
MetricList(...) | PercentChange() | Jackknife() | compute_on(df)
which is more intuitive. We overwrite the "|" operator on Metric and the __call__() of Operation so a Metric can be pipelined to an Operation. As Operation is a special kind of Metric, so it can bu further pipelined to another Operation. At last, compute_on() takes a Metric from the pipeline and is equavalent to calling metric.compute_on().
Filter
There is a "where" arg for Metric. It'll be passed to df.query() at the beginning of compute_on(df). By default the filter is not reflected in the name of Metric so same Metrics with different filters would have same column names in the returned DataFrames. It makes combining them easy.
Step59:  It's equivalent to
Step60:  SQL
You can easily get SQL query for all built-in Metrics and Operations, except for weighted Quantile/CV/Correlation/Cov, by calling
to_sql(sql_table, split_by).
You can also directly execute the query by calling
compute_on_sql(sql_table, split_by, execute, melted),
where execute is a function that can execute SQL queries. The return is very similar to compute_on().
The dialect it uses is the standard SQL in Google Cloud's BigQuery.
Step61:  Custom Metric
We provide many Metrics out of box but we understand there are cases you need more, so we make it easy for you to write you own Metrics.
First you need to understand the dataflow of a DataFrame when it's passed to compute_on(). The dataflow looks like this.
        &lt;-------------------------------------------compute_on(handles caching)----------------------------------------------&gt;
        &lt;-------------------------------------compute_through-----------------------------------&gt;                            |
        |                                              &lt;------compute_slices------&gt;             |                            |
        |                                              |-&gt; slice1 -&gt; compute |    |             |                            |
    df -&gt; df.query(where) -&gt; precompute -&gt; split_data -|-&gt; slice2 -&gt; compute | -&gt; pd.concat  -&gt; postcompute -&gt; manipulate -&gt; final_compute
                                                       |-&gt; ...             |
In summary, compute() operates on a slice of data and hence only takes one arg, df. While precompute(), postcompute(), compute_slices(), compute_through() and final_compute() operate on the whole DataFrame so they take the df that has been processed by the dataflow till them and the split_by passed to compute_on(). final_compute() also has access to the original df passed to compute_on() for you to make additional manipulation. manipulate() does common data manipulation like melting and cleaning. Besides wrapping all the computations above, compute_on() also caches the result from compute_through(). Please refer to the section of Caching for more details.
Depending on your case, you can overwrite all the methods, but we suggest you NOT to overwrite compute_on() because it might mess up the caching mechanism, nor manipulate(), because it might not work well with other Metrics' data manipulation. Here are some rules to help you decide.
1. If your Metric has no vectorization over slices, overwrite compute() which only takes one arg, df. To overwrite, you can either create a new class inheriting from Metric or just pass a lambda function into Metric.
2. If you have vectorization logic over slices, overwrite compute_slices().
3. As compute() operates on a slice of data, it doesn't have access to the columns to split_by and the index value of the slice. If you need them, overwrite compute_with_split_by(self, df, split_by, slice_name), which is just a wrapper of compute(), but has access to split_by and the value of current slice, slice_name.
4. The data passed into manipulate() should be a number, a pd.Series, or a wide/unmelted pd.DataFrame.
5. split_data() returns (sub_dataframe, corresponding slice value). You might want to overwrite it for non-vectorized Operations. See section Linear Regression for examples.
Also there are some requirements.
1. Your Metric shouldn't change the input DataFrame inplace or it might not work with other Metrics.
2. Your Metric shouldn't rely on the index of the input DataFrame if you want it to work with Jackknife. The reason is Jackknife might reset the index.
No Vectorization
Step62:  CustomSum doesn't have vectorization. It loops through the DataFrame and sum on every slice. As the result, it's slower than vectorized summation.
Step63:  With Vectorization
We can do better. Let's implement a Sum with vectorization.
Step64:  Precompute, postcompute and final_compute
They are useful when you need to preprocess and postprocess the data.
Step65:  Overwrite using Lambda Functions
For one-off Metrics, you can also overwrite precompute, compute, postcompute, compute_slices and final_compute by passing them to Metric() as lambda functions.
Step67:  Custom Operation
Writing a custom Operation is a bit more complex. Take a look at the Caching section below as well. Typically an Operation first computes its children Metrics with expanded split_by. Here are some rules to keep in mind.
1. Always use compute_on and compute_child to compute the children Metrics. They handle caching so your Operation can interact with other Metrics correctly.
2. If the Operation extends the split_by when computing children Metrics, you need to register the extra columns added in the __init__().
3. The extra columns should come after the original split_by.
4. If you really cannot obey #2 or #3, you need to overwrite Operation.flush_children(), or it won't work with Jackknife and Bootstrap.
5. Try to vectorize the Operation as much as possible. At least you can compute the children Metrics in a vectorized way by calling compute_child(). It makes the caching of the children Metrics more available.
6. Jackknife takes shortcuts when computing leave-one-out (LOO) estimates for Sum, Mean and Count, so if you want your Operation to work with Jackknife fast, delegate computations to Sum, Mean and Count as much as possible. See section Linear Regression for a comparison.
7. For the same reason, you computation logic should avoid using input df other than in compute_on() and compute_child(). When cutting corners, Jackknife emits None as the input df for LOO estimation. The compute_on() and compute_child() functions know to read from cache but other functions may not know what to do. If your Operation uses df outside the compute_on() and compute_child() functions, you have either to
      * ensure that your computation doesn't break when df is None.
      * set attribute 'precomputable_in_jk' to False (which will force the jackknife to be computed the manual way, which is slower).
Let's see Distribution for an example.
Step68:  SQL Generation
If you want the custom Metric to generate SQL query, you need to implement to_sql() or get_sql_and_with_clause(). The latter is more common and recommended. Please refer to built-in Metrics to see how it should be implemented. Here we show two examples, one for Metric and the other for Operation.
Step70:  For an Operation, you ususally call the child metrics' get_sql_and_with_clause() to get the subquery you need.
Step71:  Caching
tl;dr
Step72:  Now let's see what heppens if we reuse sum_clicks.
Step73:  Then sum_clicks only gets computed once. For Metics that are not quite compatible, you can still put them in a MeticList and set return_dataframe to False to maximize the caching.
Step74:  If you really cannot compute Metrics together, you can use a cache_key.
Step75:  The resulte are cached in ctr, a composite Metric, as well as its children, the Sum Metrics.
Step76:  You can flush the cache by calling flush_cache(key, split_by=None, recursive=True, prune=True), where "recursive" means if you want to flush the cache of the children Metrics as well, and "prune" means if the key is not found in current Metric, do you still want to flush the children Metrics or stop early. It's useful when a high level Metric appears in several places then during the flushing we will hit it multiple times. We can save time by stop early.
Step77:  Though ctr's cache has been flushed, we can still compute ctr from cache because all its children are cached.
Step78:  We won't be able to re-compute ctr if we recursively flush its cache.
Step79:  However, the behavior becomes subtle when Operation is involved.
Step80:  Note that it's sum_clicks.compute_on(df, 'country') instead of sum_clicks.compute_on(df) got saved in the cache. The reason is we need the former not the latter to compute the PercentChange. Using sum_clicks.compute_on(df, cache_key=42) will always give you the right result so it's not a big issue, just might confuse you sometime.
Step81:  Advanced Examples
Click Split
Step82:  Difference in differences
Step83:  Compare the standard errors between Jackknife and Bootstrap
Step86:  Linear Regression
Here we fit a linear regression on mean values of groups. We show two versions, the former delgates computations to Mean so its Jackknife is faster than the latter which doesn't delegate.
Step87:  LOWESS
Step88:  Coefficient Shrikage
<ASSISTANT_TASK:>
Python Code:
!pip install meterstick
Explanation: The Meterstick package provides a concise and flexible syntax to describe and execute
routine data analysis tasks. The easiest way to learn to use Meterstick is by example.
For External users
You can open this notebook in Google Colab.
Installation
You can install from pip for the stable version
End of explanation
!git clone https://github.com/google/meterstick.git
import sys, os
sys.path.append(os.getcwd())
Explanation: or from GitHub for the latest version.
End of explanation
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from meterstick import *
np.random.seed(42)
platform = ('Desktop', 'Mobile', 'Tablet')
exprs = ('ctrl', 'expr')
country = ('US', 'non-US')
size = 1000
impressions = np.random.randint(10, 20, size)
clicks = impressions * 0.1 * np.random.random(size)
df = pd.DataFrame({'impressions': impressions, 'clicks': clicks})
df['platform'] = np.random.choice(platform, size=size)
df['expr_id'] = np.random.choice(exprs, size=size)
df['country'] = np.random.choice(country, size=size)
df['cookie'] = np.random.choice(range(5), size=size)
df.loc[df.country == 'US', 'clicks'] *= 2
df.loc[(df.country == 'US') & (df.platform == 'Desktop'), 'impressions'] *= 4
df.head()
Explanation: Demo Starts
End of explanation
Sum('clicks').compute_on(df)
Explanation: Simple Metrics
There are many built-in simple Metrics in Meterstick. They directly operate on a DataFrame.
Sum
End of explanation
Count('country').compute_on(df)
Count('country', distinct=True).compute_on(df)
Explanation: Count
End of explanation
Dot('clicks', 'impressions').compute_on(df)
Explanation: Dot (inner product)
End of explanation
Dot('clicks', 'clicks', True).compute_on(df)
Explanation: It can also be normalized.
End of explanation
Max('clicks').compute_on(df)
Explanation: Max
End of explanation
Min('clicks').compute_on(df)
Explanation: Min
End of explanation
Mean('clicks').compute_on(df)
Explanation: Mean
End of explanation
Mean('clicks', 'impressions').compute_on(df)
Explanation: Weighted Mean
End of explanation
Quantile('clicks').compute_on(df) # Default is median.
Quantile('clicks', 0.2).compute_on(df)
Quantile('clicks', (0.2, 0.5)).compute_on(df) # Quantile can take multiple quantiles.
Explanation: Quantile
End of explanation
Quantile('clicks', 0.5, interpolation='higher').compute_on(df)
Explanation: Interpolation
You can specify how you want to interpolate the quantile. It could be any of (‘linear’, ‘lower’, ‘higher’, ‘midpoint’, ‘nearest’).
End of explanation
Quantile('clicks', weight='impressions').compute_on(df)
Explanation: Weighted Quantile
End of explanation
Variance('clicks').compute_on(df)
Explanation: Variance
End of explanation
Variance('clicks', unbiased=False).compute_on(df)
Explanation: Biased Variance
The default Variance is unbiased, namely, the divisor used in calculations is N - 1. You could set unbiased=False to use N as the divisor.
End of explanation
Variance('clicks', weight='impressions').compute_on(df)
Explanation: Weighted Variance
End of explanation
StandardDeviation('clicks').compute_on(df)
Explanation: Standard Deviation
End of explanation
StandardDeviation('clicks', False).compute_on(df)
Explanation: Biased Standard Deviation
Similar to biased Variance, it's possible to compute biased standard deviation.
End of explanation
StandardDeviation('clicks', weight='impressions').compute_on(df)
Explanation: Weighted Standard Deviation
End of explanation
CV('clicks').compute_on(df)
Explanation: Coefficient of Variation
End of explanation
Correlation('clicks', 'impressions').compute_on(df)
Explanation: Correlation
End of explanation
Correlation('clicks', 'impressions', weight='impressions').compute_on(df)
Explanation: Weighted Correlation
End of explanation
Cov('clicks', 'impressions').compute_on(df)
Explanation: Covariance
End of explanation
Cov('clicks', 'impressions', weight='impressions').compute_on(df)
Explanation: Weighted Covariance
End of explanation
Sum('clicks').compute_on(df, 'country')
Mean('clicks').compute_on(df, ['platform', 'country'])
Explanation: Slicing
You can group your DataFrame and compute the Metrics on slices.
End of explanation
MetricList((Sum('clicks'), Count('clicks'))).compute_on(df)
Explanation: Multiple Metrics
You can put multiple Metrics into a MetricList and compute them together. It's not only makes your codes terser, it might make the computation much faster. See Caching section for more infomation.
End of explanation
(Sum('clicks') + 1).compute_on(df)
sum((Sum('clicks'), Sum('impressions'), 1)).compute_on(df)
sum((Sum('clicks'), Sum('impressions'), 1)).set_name('meaningless sum').compute_on(df)
Explanation: Arithmetic of Metrics
You can do many arithmetic operations on Metrics. It can also be between a Metric and a scalar. You can call set_name() to give your composite Metric a new name. Internally, we operate on the results returned by Metrics with return_dataframe=False to avoid incompatible DataFrame columns names. However, if both Metrics return DataFrames even when return_dataframe is set to False, you might get lots of NAs. The solution is use rename_columns() to unify the column names. See section "Compare the standard errors between Jackknife and Bootstrap" for an example.
Add
End of explanation
(Sum('clicks') / Sum('impressions')).compute_on(df)
Explanation: Divide
End of explanation
Ratio('clicks', 'impressions', 'ctr').compute_on(df)
Explanation: Ratio
Since division between two Sums is common, we make a Ratio() Metric as a syntax sugar. Its third arg is the name for the Metric and is optional.
End of explanation
MetricList(
    (Sum('clicks') - 1,
     -Sum('clicks'),
     2 * Sum('clicks'),
     Sum('clicks')**2,
     2**Mean('clicks'),
     (Mean('impressions')**Mean('clicks')).set_name('meaningless power'))
    ).compute_on(df, melted=True)
Explanation: We also support many other common arithmetic operations.
End of explanation
Sum('clicks').compute_on(df, return_dataframe=False)
Count('clicks').compute_on(df, ['platform', 'country'], return_dataframe=False)
Mean('clicks').compute_on(df, melted=True)
MetricList((Sum('clicks'), Count('clicks'))).compute_on(df, 'country')
Quantile('clicks', [0.2, 0.7]).compute_on(df, 'country', melted=True)
# Don't worry. We will talk more about the pipeline operator "|" later.
(MetricList((Sum('clicks'), Count('clicks')))
 | Jackknife('cookie')
 | compute_on(df, 'country'))
(MetricList((Sum('clicks'), Count('clicks')))
 | Bootstrap(n_replicates=100)
 | compute_on(df, 'country', melted=True))
Explanation: Output Format
There are two options for you to control the format of the return.
1. return_dataframe: Default True, if False, we try to return a scalar or pd.Series. For complex Metrics it might have no effect and a DataFrame is always returned. For example, all Metrics in the Operations section below always return a DataFrame.
return_dataframe has different effect on MetricList. If False, MetricList will return a list of DataFrames instead of trying to concat them. This is a convenient way to compute incompatible Metrics together to maximize caching (see section Caching also). There is an attribute "children_return_dataframe" in MetricList which will be passed to children Metrics as their return_dataframe so you can get a list of numbers or pd.Series.
2. melted: Dafault False. It decides if the returned DataFrame is in wide/unmelted or long/melted form. It doesn't have effect if the return is not a DataFrame. 
  - Long/melted means the leftmost index is 'Metric' so
`MetricList((m1, m2)).compute_on(df, melted=True).loc[m1.name] ≡ m1.compute_on(df, melted=True)`
Wide/unmelted means the outermost column index is 'Metric' so
MetricList((m1, m2)).compute_on(df)[m1.name] ≡ m1.compute_on(df)
End of explanation
Distribution('country', Sum('clicks')).compute_on(df)
Explanation: Operations
An Operation is a special type of Metric that is built on top of another Metric (called a "child").  A Metric is anything that has the compute_on() method, so the child doesn't need to be a simple Metric like Sum. It could be a MetricList, a composite Metric, or even another Operation.
Distribution
Compute the child Metric on a DataFrame grouped by a column, then normalize the numbers to 1 within group.
End of explanation
(Sum('clicks').compute_on(df, 'country') /
 Sum('clicks').compute_on(df, return_dataframe=False))
Explanation: It's equal to
End of explanation
Normalize('country', Sum('clicks')).compute_on(df)
Explanation: Distribution has an alias Normalize.
End of explanation
CumulativeDistribution('country', MetricList(
    (Sum('clicks'), Sum('impressions')))).compute_on(df)
CumulativeDistribution(
    'country', Sum('clicks'), order=('non-US', 'US')).compute_on(df, 'platform')
CumulativeDistribution(
    'country', MetricList((Sum('clicks'), Sum('impressions')))
).compute_on(df, melted=True)
Explanation: Cumulative Distribution
Similar to Distribution except that it returns the cumulative sum after normalization, but unlike Distribution the order of the cumulating column matters. As the result, we always sort the column and there is an 'order' arg for you to customize the ordering.
End of explanation
PercentChange('country', 'US', Mean('clicks')).compute_on(df)
mean = Mean('clicks').compute_on(df, 'country')
(mean.loc['non-US'] / mean.loc['US'] - 1) * 100
Explanation: PercentChange
Computes the percent change to a certain group on the DataFrame returned by the child Metric. The returned value is the # of percent points.
End of explanation
PercentChange(
    'country',
    'US',
    MetricList((Count('clicks'), Count('impressions'))),
    include_base=True).compute_on(df, 'platform')
Explanation: You can include the base group in your result.
End of explanation
PercentChange(
    ['country', 'platform'],
    ('US', 'Desktop'),
    MetricList((Count('clicks'), Count('impressions'))),
    include_base=True).compute_on(df)
Explanation: You can also specify multiple columns as the condition columns, then your base value should be a tuple.
End of explanation
AbsoluteChange('country', 'US', Mean('clicks')).compute_on(df)
Explanation: Absolute Change
Very similar to PercentChange, but the absolute difference is returned.
End of explanation
AbsoluteChange(
    'country', 'US', Count('clicks'), include_base=True).compute_on(
        df, 'platform', melted=True)
Explanation: You can also include the base group in your result.
End of explanation
ctr = Ratio('clicks', 'impressions')
MH('country', 'US', 'platform', ctr).compute_on(df)  # stratified by platform
Explanation: Cochran-Mantel-Haenszel statistics
Please refer to the Wikepedia page for its definition. Besides the condition column and baseline key that PercentChange and AbsoluteChange take, CMH also needs a column to  stratify. The child Metric must be a ratio of two single-column Metrics or CMH doesn't make sense. So instead of passing
AbsoluteChange(MetricList([a, b])) / AbsoluteChange(MetricList([c, d])),
please use
MetricList([AbsoluteChange(a) / AbsoluteChange(c),
          AbsoluteChange(b) / AbsoluteChange(d)]).
End of explanation
np.random.seed(42)
exprs = ('ctrl', 'expr')
n = 10000
df_prepost = pd.DataFrame({'impressions': np.random.randint(10, 30, n)})
df_prepost['expr_id'] = np.random.choice(exprs, size=n)
df_prepost['cookie'] = np.random.choice(range(20), size=n)
# Preperiod correlates with postperiod.
df_prepost['pre_impressions'] = np.random.normal(df_prepost.impressions, 3)
# Add small and noisy improvments.
df_prepost.loc[df_prepost.expr_id == 'expr', 'impressions'] += np.random.randint(-2, 4, size=len(df_prepost.loc[df_prepost.expr_id == 'expr', 'impressions']))
abs = AbsoluteChange('expr_id', 'ctrl', Mean('impressions'))
cuped = CUPED('expr_id', 'ctrl', Mean('impressions'), Mean('pre_impressions'), 'cookie')
MetricList((abs, cuped)).compute_on(df_prepost)
Explanation: CUPED
It computes the absolute change that has been adjusted using the CUPED approach. It provides an unbiased estimate to the absolute change with lower variance.
Let's see how it works on a fake data with preperiod metrics that are correlated with postperiod metrics and the effect of the experiment is small and noisy.
End of explanation
from sklearn import linear_model
df_agg = MetricList((Mean('impressions'), Mean('pre_impressions'))).compute_on(df_prepost, ['expr_id', 'cookie'])
lm = linear_model.LinearRegression()
lm.fit(df_agg[['mean(pre_impressions)']], df_agg['mean(impressions)'])
theta = lm.coef_[0]
df_agg['adjusted'] = df_agg['mean(impressions)'] - theta * df_agg['mean(pre_impressions)']
adjusted = df_agg.groupby('expr_id').adjusted.mean()
adjusted['expr'] - adjusted['ctrl']
Explanation: CUPED essentially fits a linear model of Postperiod metric ~ 1 + preperiod metric and uses it to control for the variance in the preperiod.
End of explanation
from plotnine import ggplot, aes, geom_density, after_stat, facet_grid
data_to_plot = pd.concat([df_agg['mean(impressions)'], df_agg.adjusted], keys=['Raw', 'CUPED'], names=['Adjusted'])
data_to_plot = pd.DataFrame(data_to_plot, columns=['Value']).reset_index()
(
    ggplot(data_to_plot)
    + aes(x="Value", y=after_stat('density'), color='expr_id')
    + geom_density()
    + facet_grid('Adjusted ~ .')
)
# Jackknife is explained in the 'Standard Errors' section.
Jackknife('cookie', MetricList((abs, cuped))).compute_on(df_prepost)
# It's possible to control for multiple metrics.
CUPED('expr_id', 'ctrl', Mean('impressions'),
      [Mean('pre_impressions'), Mean('pre_impressions')**2],
      'cookie').compute_on(df_prepost)
Explanation: We can see that CUPED's result is similar to the absolute change but has smaller variance.
End of explanation
pct = PercentChange('expr_id', 'ctrl', Mean('impressions'))
prepost = PrePostChange('expr_id', 'ctrl', Mean('impressions'), Mean('pre_impressions'), 'cookie')
MetricList((pct, prepost)).compute_on(df_prepost)
df_agg = MetricList((Mean('impressions'), Mean('pre_impressions'))).compute_on(
    df_prepost, ['expr_id', 'cookie']).reset_index()
df_agg['mean(pre_impressions)'] -= df_agg['mean(pre_impressions)'].mean()
df_agg['is_treated'] = df_agg.expr_id == 'expr'
df_agg['interaction'] = df_agg.is_treated * df_agg['mean(pre_impressions)']
lm = linear_model.LinearRegression()
lm.fit(df_agg[['is_treated', 'mean(pre_impressions)', 'interaction']],
       df_agg['mean(impressions)'])
beta1 = lm.intercept_
beta2 = lm.coef_[0]
beta2 / beta1 * 100
# Jackknife is explained in the 'Standard Errors' section.
Jackknife('cookie', MetricList((pct, prepost))).compute_on(df_prepost)
Explanation: PrePostChange
It computes the percent change that has been adjusted using the PrePost approach. It's similar to CUPED but control for treatment groups additionally. Essentially, it fits
Postperiod metric ~ 1 + is_treated * preperiod metric, or more verbosely,
Postperiod metric = β1 + β2 * is_treated  + β3 * preperiod metric + β4 * is_treated * preperiod metric.
Note that the estimate of β2 will be the estimate of treatment effect and the control arm metric can be estimated using β1 if we centered preperiod metric. As the result, β2 / β1 will be the estimate of the percent change that PrePostChange returns.
End of explanation
Jackknife('cookie', MetricList((Sum('clicks'), Sum('impressions')))).compute_on(df)
metrics = MetricList((Sum('clicks'), Sum('impressions')))
Jackknife('cookie', metrics).compute_on(df, 'country', True)
Explanation: Standard Errors
Jackknife
Unlike all Metrics we have seen so far, Jackknife returns a multiple-column DataFrame because by default we return point estimate and standard error.
End of explanation
Jackknife('cookie', metrics, 0.9).compute_on(df)
res = (
    MetricList((Ratio('clicks', 'impressions', 'ctr'), Sum('clicks')))
    | PercentChange('country', 'US')
    | Jackknife('cookie', confidence=0.9)
    | compute_on(df, 'platform'))
res.display()
Explanation: You can also specify a confidence level, the we'll return the confidence interval. The returned DataFrame also comes with a display() method for visualization which will highlight significant changes. To customize the display(), please take a look at confidence_interval_display_demo.ipynb.
End of explanation
np.random.seed(42)
Bootstrap(None, Sum('clicks'), 100).compute_on(df)
np.random.seed(42)
Bootstrap('cookie', Sum('clicks'), 100).compute_on(df, 'country')
np.random.seed(42)
Bootstrap('cookie', Sum('clicks'), 100, 0.95).compute_on(df, 'country')
np.random.seed(42)
res = (
    MetricList((Ratio('clicks', 'impressions', 'ctr'), Sum('impressions')))
    | AbsoluteChange('country', 'US')
    | Bootstrap(None, n_replicates=100, confidence=0.9)
    | compute_on(df, 'platform'))
res.display()
Explanation: Bootstrap
The output is similar to Jackknife. The different args are
  - unit: If None, we bootstrap on rows. Otherwise we do a block bootstrap. The unique values in unit column will be used as the resampling buckets.
  - n_replicates: The number of resamples. Default to 10000, which is recommended in Tim Hesterberg's What Teachers Should Know About the Bootstrap. Here we use a smaller number for faster demonstration.
End of explanation
from meterstick.models import *
Explanation: Models
Meterstick also has built-in support for model fitting. The module is not imported by default, so you need to manually import it.
End of explanation
m = LinearRegression(Mean('clicks'), Mean('impressions'), 'platform')
m.compute_on(df)
Explanation: Linear Regression
End of explanation
from sklearn import linear_model
x = Mean('impressions').compute_on(df, 'platform')
y = Mean('clicks').compute_on(df, 'platform')
m = linear_model.LinearRegression().fit(x, y)
print(m.coef_, m.intercept_)
Explanation: What Model(y, x, groupby).compute_on(data) does is
1. Computes MetricList((y, x)).compute_on(data, groupby).
2. Fits the underlying sklearn model on the result from #1.
End of explanation
# x can also be a list of Metrics or a MetricList.
m = Ridge(
    Mean('clicks'),
    [Mean('impressions'), Variance('clicks')],
    'platform',
    alpha=2)
m.compute_on(df, melted=True)
Explanation: Ridge Regression
End of explanation
m = Lasso(
    Mean('clicks'),
    Mean('impressions'),
    'platform',
    fit_intercept=False,
    alpha=5)
m.compute_on(df, 'country')
Explanation: Lasso Regression
End of explanation
m = LogisticRegression(Count('clicks'), Mean('impressions'), 'country')
m.compute_on(df, melted=True)
Explanation: Logistic Regression
End of explanation
m = LogisticRegression(Count('clicks'), Mean('impressions'), 'platform', name='LR')
m.compute_on(df, melted=True)
Explanation: If y is not binary, by default a multinomal model is fitted. The behavior can be controlled via the 'multinomial' arg.
End of explanation
Count('clicks').compute_on(df, 'platform')
Explanation: Classes are the unique values of y.
End of explanation
(LinearRegression(
    Mean('clicks'),
    [Mean('impressions'), Variance('impressions')],
    'country',
    name='lm')
 | AbsoluteChange('platform', 'Desktop')
 | Jackknife('cookie', confidence=0.9)
 | compute_on(df)).display()
Explanation: Wrapping sklearn models into Meterstick provides the ability to combine Models with other built-in Metrics and Operations. For example, you can Jackknife the Model to get the uncertainty of coefficients.
End of explanation
clicks_us = Sum('clicks', where='country == "US"')
clicks_not_us = Sum('clicks', where='country != "US"')
(clicks_not_us - clicks_us).compute_on(df)
Explanation: Pipeline
You have already seen this. Instead of
Jackknife(PercentChange(MetricList(...))).compute_on(df)
you can write
MetricList(...) | PercentChange() | Jackknife() | compute_on(df)
which is more intuitive. We overwrite the "|" operator on Metric and the __call__() of Operation so a Metric can be pipelined to an Operation. As Operation is a special kind of Metric, so it can bu further pipelined to another Operation. At last, compute_on() takes a Metric from the pipeline and is equavalent to calling metric.compute_on().
Filter
There is a "where" arg for Metric. It'll be passed to df.query() at the beginning of compute_on(df). By default the filter is not reflected in the name of Metric so same Metrics with different filters would have same column names in the returned DataFrames. It makes combining them easy.
End of explanation
Sum('clicks') | AbsoluteChange('country', 'US') | compute_on(df)
Explanation: It's equivalent to
End of explanation
MetricList((Sum('X', where='Y > 0'), Sum('X'))).to_sql('T', 'grp')
m = MetricList((Sum('clicks'), Mean('impressions')))
m = AbsoluteChange('country', 'US', m)
m.compute_on(df, 'platform')
from sqlalchemy import create_engine
engine = create_engine('sqlite://', echo=False)
df.to_sql('T', con=engine)
# Meterstick uses a different SQL dialect from SQLAlchemy, so this doesn't
# always work.
m.compute_on_sql('T', 'platform', execute=lambda sql: pd.read_sql(sql, engine))
Explanation: SQL
You can easily get SQL query for all built-in Metrics and Operations, except for weighted Quantile/CV/Correlation/Cov, by calling
to_sql(sql_table, split_by).
You can also directly execute the query by calling
compute_on_sql(sql_table, split_by, execute, melted),
where execute is a function that can execute SQL queries. The return is very similar to compute_on().
The dialect it uses is the standard SQL in Google Cloud's BigQuery.
End of explanation
class CustomSum(Metric):
  def __init__(self, var):
    name = 'custom sum(%s)' % var
    super(CustomSum, self).__init__(name)
    self.var = var
  def compute(self, df):
    return df[self.var].sum()
CustomSum('clicks').compute_on(df, 'country')
Sum('clicks').compute_on(df, 'country')
Explanation: Custom Metric
We provide many Metrics out of box but we understand there are cases you need more, so we make it easy for you to write you own Metrics.
First you need to understand the dataflow of a DataFrame when it's passed to compute_on(). The dataflow looks like this.
        &lt;-------------------------------------------compute_on(handles caching)----------------------------------------------&gt;
        &lt;-------------------------------------compute_through-----------------------------------&gt;                            |
        |                                              &lt;------compute_slices------&gt;             |                            |
        |                                              |-&gt; slice1 -&gt; compute |    |             |                            |
    df -&gt; df.query(where) -&gt; precompute -&gt; split_data -|-&gt; slice2 -&gt; compute | -&gt; pd.concat  -&gt; postcompute -&gt; manipulate -&gt; final_compute
                                                       |-&gt; ...             |
In summary, compute() operates on a slice of data and hence only takes one arg, df. While precompute(), postcompute(), compute_slices(), compute_through() and final_compute() operate on the whole DataFrame so they take the df that has been processed by the dataflow till them and the split_by passed to compute_on(). final_compute() also has access to the original df passed to compute_on() for you to make additional manipulation. manipulate() does common data manipulation like melting and cleaning. Besides wrapping all the computations above, compute_on() also caches the result from compute_through(). Please refer to the section of Caching for more details.
Depending on your case, you can overwrite all the methods, but we suggest you NOT to overwrite compute_on() because it might mess up the caching mechanism, nor manipulate(), because it might not work well with other Metrics' data manipulation. Here are some rules to help you decide.
1. If your Metric has no vectorization over slices, overwrite compute() which only takes one arg, df. To overwrite, you can either create a new class inheriting from Metric or just pass a lambda function into Metric.
2. If you have vectorization logic over slices, overwrite compute_slices().
3. As compute() operates on a slice of data, it doesn't have access to the columns to split_by and the index value of the slice. If you need them, overwrite compute_with_split_by(self, df, split_by, slice_name), which is just a wrapper of compute(), but has access to split_by and the value of current slice, slice_name.
4. The data passed into manipulate() should be a number, a pd.Series, or a wide/unmelted pd.DataFrame.
5. split_data() returns (sub_dataframe, corresponding slice value). You might want to overwrite it for non-vectorized Operations. See section Linear Regression for examples.
Also there are some requirements.
1. Your Metric shouldn't change the input DataFrame inplace or it might not work with other Metrics.
2. Your Metric shouldn't rely on the index of the input DataFrame if you want it to work with Jackknife. The reason is Jackknife might reset the index.
No Vectorization
End of explanation
%%timeit
CustomSum('clicks').compute_on(df, 'country')
%%timeit
Sum('clicks').compute_on(df, 'country')
%%timeit
df.groupby('country')['clicks'].sum()
Explanation: CustomSum doesn't have vectorization. It loops through the DataFrame and sum on every slice. As the result, it's slower than vectorized summation.
End of explanation
class VectorizedSum(Metric):
  def __init__(self, var):
    name = 'vectorized sum(%s)' % var
    super(VectorizedSum, self).__init__(name = name)
    self.var = var
  def compute_slices(self, df, split_by):
    if split_by:
      return df.groupby(split_by)[self.var].sum()
    return df[self.var].sum()
VectorizedSum('clicks').compute_on(df, 'country')
%%timeit
VectorizedSum('clicks').compute_on(df, 'country')
Explanation: With Vectorization
We can do better. Let's implement a Sum with vectorization.
End of explanation
class USOnlySum(Sum):
  def precompute(self, df, split_by):
    return df[df.country == 'US']
  def postcompute(self, data, split_by):
    print('Inside postcompute():')
    print('Input data: ', data)
    print('Input split_by: ', split_by)
    print('\n')
    return data
  def final_compute(self, res, melted, return_dataframe, split_by, df):
    # res is the result processed by the dataflow till now. df is the original
    # DataFrme passed to compute_on().
    print('Inside final_compute():')
    for country in df.country.unique():
      if country not in res.index:
        print('Country "%s" is missing!' % country)
    return res
USOnlySum('clicks').compute_on(df, 'country')
Explanation: Precompute, postcompute and final_compute
They are useful when you need to preprocess and postprocess the data.
End of explanation
normalize = metrics.Sum('clicks', postcompute=lambda res, split_by: res / res.sum())
normalize.compute_on(df, 'country')
# The above is equivalent to Normalize by 'country'.
Normalize('country', Sum('clicks')).compute_on(df)
Explanation: Overwrite using Lambda Functions
For one-off Metrics, you can also overwrite precompute, compute, postcompute, compute_slices and final_compute by passing them to Metric() as lambda functions.
End of explanation
class Distribution(Operation):
  Computes the normalized values of a Metric over column(s).
  Attributes:
    extra_index: A list of column(s) to normalize over.
    children: A tuple of a Metric whose result we normalize on. And all other
      attributes inherited from Operation.
  
  def __init__(self,
               over: Union[Text, List[Text]],
               child: Optional[Metric] = None,
               **kwargs):
    self.over = over
    # The 3rd argument is the extra column that will be added to split_by. It'll
    # be converted to a list then assigned to self.extra_index.
    super(Distribution, self).__init__(child, 'Distribution of {}', over,
                                       **kwargs)
  def compute_slices(self, df, split_by=None):
    # extra_index is after the split_by.
    lvls = split_by + self.extra_index if split_by else self.extra_index
    res = self.compute_child(df, lvls)
    total = res.groupby(level=split_by).sum() if split_by else res.sum()
    return res / total
Explanation: Custom Operation
Writing a custom Operation is a bit more complex. Take a look at the Caching section below as well. Typically an Operation first computes its children Metrics with expanded split_by. Here are some rules to keep in mind.
1. Always use compute_on and compute_child to compute the children Metrics. They handle caching so your Operation can interact with other Metrics correctly.
2. If the Operation extends the split_by when computing children Metrics, you need to register the extra columns added in the __init__().
3. The extra columns should come after the original split_by.
4. If you really cannot obey #2 or #3, you need to overwrite Operation.flush_children(), or it won't work with Jackknife and Bootstrap.
5. Try to vectorize the Operation as much as possible. At least you can compute the children Metrics in a vectorized way by calling compute_child(). It makes the caching of the children Metrics more available.
6. Jackknife takes shortcuts when computing leave-one-out (LOO) estimates for Sum, Mean and Count, so if you want your Operation to work with Jackknife fast, delegate computations to Sum, Mean and Count as much as possible. See section Linear Regression for a comparison.
7. For the same reason, you computation logic should avoid using input df other than in compute_on() and compute_child(). When cutting corners, Jackknife emits None as the input df for LOO estimation. The compute_on() and compute_child() functions know to read from cache but other functions may not know what to do. If your Operation uses df outside the compute_on() and compute_child() functions, you have either to
      * ensure that your computation doesn't break when df is None.
      * set attribute 'precomputable_in_jk' to False (which will force the jackknife to be computed the manual way, which is slower).
Let's see Distribution for an example.
End of explanation
class SumWithSQL(SimpleMetric):
  def __init__(self,
               var: Text,
               name: Optional[Text] = None,
               where: Optional[Text] = None,
               **kwargs):
    super(SumWithSQL, self).__init__(var, name, 'sum({})', where, **kwargs)
    self._sum = Sum(var, name, where, **kwargs)
  def compute_slices(self, df, split_by):
    return self._sum.compute_slices(df, split_by)
  # All the SQL-related classes, like Datasource, Filters, Columns, and so on,
  # are defined in sql.py.
  def get_sql_and_with_clause(self, table: Datasource, split_by: Columns,
                              global_filter: Filters, indexes: Columns,
                              local_filter: Filters, with_data: Datasources):
    del indexes  # unused
    # Always starts with this line unless you know what you are doing.
    local_filter = Filters([self.where, local_filter]).remove(global_filter)
    columns = Column(self.var, 'SUM({})', self.name, local_filter)
    # Returns a Sql instance and the WITH clause it needs.
    return Sql(columns, table, global_filter, split_by), with_data
m = Sum('clicks') - SumWithSQL('clicks', 'custom_sum')
m.compute_on_sql('T', 'platform', execute=lambda sql: pd.read_sql(sql, engine))
Explanation: SQL Generation
If you want the custom Metric to generate SQL query, you need to implement to_sql() or get_sql_and_with_clause(). The latter is more common and recommended. Please refer to built-in Metrics to see how it should be implemented. Here we show two examples, one for Metric and the other for Operation.
End of explanation
class DistributionWithSQL(Operation):
  def __init__(self,
               over: Union[Text, List[Text]],
               child: Optional[Metric] = None,
               **kwargs):
    super(DistributionWithSQL, self).__init__(child, 'Distribution of {}', over,
                                              **kwargs)
  def compute_slices(self, df, split_by=None):
    lvls = split_by + self.extra_index if split_by else self.extra_index
    res = self.compute_child(df, lvls)
    total = res.groupby(level=split_by).sum() if split_by else res.sum()
    return res / total
  def get_sql_and_with_clause(self,
                              table: Datasource,
                              split_by: Columns,
                              global_filter: Filters,
                              indexes: Columns,
                              local_filter: Filters,
                              with_data: Datasources):
    Gets the SQL query and WITH clause.
    The query is constructed by
    1. Get the query for the child metric.
    2. Keep all indexing/groupby columns unchanged.
    3. For all value columns, get
      value / SUM(value) OVER (PARTITION BY split_by).
    Args:
      table: The table we want to query from.
      split_by: The columns that we use to split the data.
      global_filter: The Filters that can be applied to the whole Metric tree.
      indexes: The columns that we shouldn't apply any arithmetic operation.
      local_filter: The Filters that have been accumulated so far.
      with_data: A global variable that contains all the WITH clauses we need.
    Returns:
      The SQL instance for metric, without the WITH clause component.
      The global with_data which holds all datasources we need in the WITH
        clause.
    
    # Always starts with this line unless you know what you are doing.
    local_filter = Filters([self.where, local_filter]).remove(global_filter)
    # The intermediate tables needed by child metrics will be added to with_data
    # in-place.
    child_sql, with_data = self.children[0].get_sql_and_with_clause(
        table, indexes, global_filter, indexes, local_filter, with_data)
    child_table = sql.Datasource(child_sql, 'DistributionRaw')
    # Always use the alias returned by with_data.add(), because if the with_data
    # already holds a different table that also has 'DistributionRaw' as its
    # alias, we'll use a different alias for the child_table, which is returned
    # by with_data.add().
    child_table_alias = with_data.add(child_table)
    groupby = sql.Columns(indexes.aliases, distinct=True)
    columns = sql.Columns()
    for c in child_sql.columns:
      if c.alias in groupby:
        continue
      col = sql.Column(c.alias) / sql.Column(
          c.alias, 'SUM({})', partition=split_by.aliases)
      col.set_alias('Distribution of %s' % c.alias_raw)
      columns.add(col)
    return sql.Sql(groupby.add(columns), child_table_alias), with_data
m = DistributionWithSQL('country', Sum('clicks'))
m.to_sql('T')
Explanation: For an Operation, you ususally call the child metrics' get_sql_and_with_clause() to get the subquery you need.
End of explanation
class SumWithTrace(Sum):
  def compute_through(self, data, split_by):
    print('Computing %s...' % self.name)
    return super(SumWithTrace, self).compute_through(data, split_by)
sum_clicks = SumWithTrace('clicks', 'sum of clicks')
ctr = SumWithTrace('clicks') / SumWithTrace('impressions')
MetricList((sum_clicks, ctr)).compute_on(df)
Explanation: Caching
tl;dr: Reuse Metrics as much as possible and compute them together.
Computation can be slow so it'd nice if we pass in the same DataFrame multiple
times the computation is actually only done once. The difficulty is that
DataFrame is mutable so it's hard to decide whether we really saw this DataFrame
before. However, in one round of compute_on(), the DataFrame shouldn't change
(our Metrics never change the original DataFrame and your custom Metrics
shouldn't either), so we can cache the result, namely, a Metric appearing in
multiple places will only be computed once. This all happens automatically so
you don't need to worry about it. If you really cannot compute all your Metrics
in one round, there is a "cache_key" arg in compute_on(). What it does is
if the key is in cache, just read the cache;
if not, compute and save the result to cache under the key.
Note:
1. All we check is cache_key, nothing more, so it's your responsibility to
make sure same key really corresponds to the same input DataFrame AND split_by.
2. The caching and retrieving happen in all levels of Metrics, so
PercentChange(..., Sum('x')).compute_on(df, cache_key='foo')
not only cache the percent change to PercentChange's cache, but also cache
Sum('x').compute_through(df)
to Sum('x')'s cache. Note it's the output of compute_through() is cached so we
do't need to re-compute just because you change "melted" from True to False.
3. Anything that can be a key of a dict can be used as cache_key, except '_RESERVED' and tuples like ('_RESERVED', ...).
First, let's illustrate that when we don't reuse Metrics, everything gets
computed once as expected.
End of explanation
sum_clicks = SumWithTrace('clicks', 'sum of clicks')
ctr = sum_clicks / SumWithTrace('impressions')
MetricList((sum_clicks, ctr)).compute_on(df)
Explanation: Now let's see what heppens if we reuse sum_clicks.
End of explanation
sum_clicks = SumWithTrace('clicks', 'sum of clicks')
jk, s = MetricList(
    [Jackknife('cookie', sum_clicks), sum_clicks],
    children_return_dataframe=False).compute_on(
        df, return_dataframe=False)
print(s)
jk
Explanation: Then sum_clicks only gets computed once. For Metics that are not quite compatible, you can still put them in a MeticList and set return_dataframe to False to maximize the caching.
End of explanation
sum_clicks = SumWithTrace('clicks', 'sum of clicks')
ctr = sum_clicks / SumWithTrace('impressions')
sum_clicks.compute_on(df, 'country', cache_key='foo')
ctr.compute_on(df, 'country', cache_key='foo')
Explanation: If you really cannot compute Metrics together, you can use a cache_key.
End of explanation
sum_clicks = SumWithTrace('clicks', 'sum of clicks')
ctr = sum_clicks / SumWithTrace('impressions')
MetricList((sum_clicks, ctr)).compute_on(df, cache_key='foo')
print('sum_clicks cached: ', sum_clicks.get_cached('foo'))
print('ctr cached: ', ctr.get_cached('foo'))
ctr.compute_on(None, cache_key='foo')
Explanation: The resulte are cached in ctr, a composite Metric, as well as its children, the Sum Metrics.
End of explanation
sum_clicks = SumWithTrace('clicks', 'sum of clicks')
ctr = sum_clicks / SumWithTrace('impressions')
MetricList((sum_clicks, ctr)).compute_on(df, cache_key='foo')
ctr.flush_cache('foo', recursive=False)
sum_clicks.compute_on(None, cache_key='foo')  # sum is not flushed.
ctr.in_cache('foo')
Explanation: You can flush the cache by calling flush_cache(key, split_by=None, recursive=True, prune=True), where "recursive" means if you want to flush the cache of the children Metrics as well, and "prune" means if the key is not found in current Metric, do you still want to flush the children Metrics or stop early. It's useful when a high level Metric appears in several places then during the flushing we will hit it multiple times. We can save time by stop early.
End of explanation
ctr.compute_on(None, cache_key='foo')
ctr.in_cache('foo')
Explanation: Though ctr's cache has been flushed, we can still compute ctr from cache because all its children are cached.
End of explanation
ctr.flush_cache('foo')
sum_clicks.compute_on(None, cache_key='foo')  # sum is flushed too.
Explanation: We won't be able to re-compute ctr if we recursively flush its cache.
End of explanation
sum_clicks = SumWithTrace('clicks')
PercentChange('country', 'US', sum_clicks).compute_on(df, cache_key=42)
sum_clicks.compute_on(None, 'country', cache_key=42)
Explanation: However, the behavior becomes subtle when Operation is involved.
End of explanation
sum_clicks.compute_on(df, cache_key=42)
sum_clicks.compute_on(df, 'country', cache_key=42)
Explanation: Note that it's sum_clicks.compute_on(df, 'country') instead of sum_clicks.compute_on(df) got saved in the cache. The reason is we need the former not the latter to compute the PercentChange. Using sum_clicks.compute_on(df, cache_key=42) will always give you the right result so it's not a big issue, just might confuse you sometime.
End of explanation
np.random.seed(42)
df['duration'] = np.random.random(len(df)) * 200
long_clicks = Sum('clicks', where='duration > 60')
short_clicks = Sum('clicks', where='duration < 30')
click_split = (long_clicks / short_clicks).set_name('click split')
click_split | Jackknife('cookie') | compute_on(df, 'country')
Explanation: Advanced Examples
Click Split
End of explanation
np.random.seed(42)
df['period'] = np.random.choice(('preperiod', 'postperiod'), size=size)
sum_clicks = Sum('clicks')
ctr = sum_clicks / Sum('impressions')
metrics = (sum_clicks, ctr)
preperiod_clicks = MetricList(metrics, where='period == "preperiod"')
postperiod_clicks = MetricList(metrics, where='period == "postperiod"')
pct = PercentChange('platform', 'Desktop')
did = (pct(postperiod_clicks) - pct(preperiod_clicks)).rename_columns(
    ['clicks% DID', 'ctr% DID'])
Jackknife('cookie', did).compute_on(df)
Explanation: Difference in differences
End of explanation
np.random.seed(42)
sum_clicks = Sum('clicks')
ctr = sum_clicks / Sum('impressions')
metrics = MetricList((sum_clicks, ctr))
(Jackknife('cookie', metrics) /
 Bootstrap('cookie', metrics, 100)).rename_columns(
     pd.MultiIndex.from_product(
         (('sum(clicks)', 'ctr'), ('Value', 'SE')))).compute_on(df, 'country')
Explanation: Compare the standard errors between Jackknife and Bootstrap
End of explanation
np.random.seed(42)
size = 1000000
df_lin = pd.DataFrame({'grp': np.random.choice(range(10), size=size)})
df_lin['x'] = df_lin.grp + np.random.random(size=size)
df_lin['y'] = 2 * df_lin.x + np.random.random(size=size)
df_lin['cookie'] = np.random.choice(range(20), size=size)
df_lin_mean = df_lin.groupby('grp').mean()
plt.scatter(df_lin_mean.x, df_lin_mean.y)
plt.show()
from sklearn import linear_model
class LinearReg(Operation):
  def __init__(self, x, y, grp):
    self.lm = linear_model.LinearRegression()
    # Delegate most of the computations to Mean Metrics.
    child = MetricList((Mean(x), Mean(y)))
    self.grp = grp
    # Register grp as the extra_index.
    super(LinearReg, self).__init__(child, '%s ~ %s' % (y, x), grp)
  def split_data(self, df, split_by=None):
    The 1st element in yield will be passed to compute().
    if not split_by:
      yield self.compute_child(df, self.grp), None
    else:
      # grp needs to come after split_by.
      child = self.compute_child(df, split_by + [self.grp])
      keys, indices = list(zip(*child.groupby(split_by).groups.items()))
      for i, idx in enumerate(indices):
        yield child.loc[idx.unique()].droplevel(split_by), keys[i]
  def compute(self, df):
    self.lm.fit(df.iloc[:, [0]], df.iloc[:, 1])
    return pd.Series((self.lm.coef_[0], self.lm.intercept_))
lr = LinearReg('x', 'y', 'grp')
Jackknife('cookie', lr, 0.95).compute_on(df_lin)
class LinearRegSlow(Metric):
  def __init__(self, x, y, grp):
    self.lm = linear_model.LinearRegression()
    # Doesn't delegate.
    self.x = x
    self.y = y
    self.grp = grp
    super(LinearRegSlow, self).__init__('%s ~ %s' % (y, x))
  def split_data(self, df, split_by=None):
    The 1st element in yield will be passed to compute().
    idx = split_by + [self.grp] if split_by else self.grp
    mean = df.groupby(idx).mean()
    if not split_by:
      yield mean, None
    else:
      keys, indices = list(zip(*mean.groupby(split_by).groups.items()))
      for i, idx in enumerate(indices):
        yield mean.loc[idx.unique()].droplevel(split_by), keys[i]
  def compute(self, df):
    self.lm.fit(df.iloc[:, [0]], df.iloc[:, 1])
    return pd.Series((self.lm.coef_[0], self.lm.intercept_))
lr_slow = LinearRegSlow('x', 'y', 'grp')
Jackknife('cookie', lr_slow, 0.95).compute_on(df_lin)
%%timeit
Jackknife('cookie', lr, 0.95).compute_on(df_lin)
%%timeit
Jackknife('cookie', lr_slow, 0.95).compute_on(df_lin)
Explanation: Linear Regression
Here we fit a linear regression on mean values of groups. We show two versions, the former delgates computations to Mean so its Jackknife is faster than the latter which doesn't delegate.
End of explanation
# Mimics that measurements, y, are taken repeatedly at a fixed grid, x.
np.random.seed(42)
size = 10
x = list(range(5))
df_sin = pd.DataFrame({'x': x * size, 'cookie': np.repeat(range(size), len(x))})
df_sin['y'] = np.sin(df_sin.x) + np.random.normal(scale=0.5, size=len(df_sin.x))
df_sin.head(10)
import statsmodels.api as sm
lowess = sm.nonparametric.lowess
class Lowess(Metric):
  def __init__(self, x, y, name=None, where=None):
    self.x = x
    self.y = y
    name = name or 'LOWESS(%s ~ %s)' % (y, x)
    super(Lowess, self).__init__(name, where=where)
  def compute(self, data):
    lowess_fit = pd.DataFrame(
        lowess(data[self.y], data[self.x]), columns=[self.x, self.y])
    return lowess_fit.drop_duplicates().reset_index(drop=True)
Lowess('x', 'y') | compute_on(df_sin)
jk = Lowess('x', 'y') | Jackknife('cookie', confidence=0.9) | compute_on(df_sin)
point_est = jk[('y', 'Value')]
ci_lower = jk[('y', 'Jackknife CI-lower')]
ci_upper = jk[('y', 'Jackknife CI-upper')]
plt.scatter(df_sin.x, df_sin.y)
plt.plot(x, point_est, c='g')
plt.fill_between(
    x, ci_lower,
    ci_upper,
    color='g',
    alpha=0.5)
plt.show()
Explanation: LOWESS
End of explanation
from plotnine import ggplot, geom_point, geom_ribbon, aes, ylab
y = Mean('clicks')
x = [Mean('impressions'), Variance('impressions')]
grpby = 'platform'
baseline = LinearRegression(y, x, grpby, fit_intercept=False)
shrinkage = [(Ridge(y, x, grpby, a, False) / baseline).rename_columns(
    ('%s::mean(impressions)' % a, '%s::var(impressions)' % a))
             for a in range(10)]
jk = (MetricList(shrinkage)
  | Jackknife('cookie', confidence=0.95)
  | compute_on(df, melted=True)).reset_index()
jk[['penalty', 'X']] = jk.Metric.str.split('::', expand=True)
jk.penalty = jk.penalty.astype(int)
(ggplot(jk, aes('penalty', 'Value', color='X'))
  + ylab('Shrinkage')
  + geom_point()
  + geom_ribbon(
     aes(ymin='Jackknife CI-lower', ymax='Jackknife CI-upper', fill='X'),
     alpha=0.1))
Explanation: Coefficient Shrikage
End of explanation
<END_TASK>