<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Compute GitHub Stats
Notebook setup
Step1:  Setup Authorization
If you are using a service account run
%%bash
Activate Service Account provided by Kubeflow.
gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
If you are running using user credentials
gcloud auth application-default login
Step3:  Unique PR Creators
Step4:  Number Prs
Step6:  Release stats per release (quarter)
Compute stats about a release 
We do this based on time
You can see a sample of the payload at https
Step8:  Get a list of distinct actions
Here's a list of events in the api
It looks like these are different from the ones in the github archive
Step10:  New Issues Last 28 Days
Step12:  GetSomeSampleIssue Events
Step14:  Get some sample pull request events
Want to inspect the data
Step16:  Get Distinct Types
<ASSISTANT_TASK:>
Python Code:
import os
import subprocess
if os.path.exists("/var/run/secrets/kubernetes.io/serviceaccount"):
    subprocess.check_call(["pip", "install", "--user", "-r", "requirements.txt"], stderr=subprocess.STDOUT, bufsize=1)
# NOTE: The RuntimeWarnings (if any) are harmless. See ContinuumIO/anaconda-issues#6678.
import altair as alt
from pandas.io import gbq
import pandas as pd
import numpy as np
from importlib import reload
import itertools
import getpass
import subprocess
# Configuration Variables. Modify as desired.
PROJECT = subprocess.check_output(["gcloud", "config", "get-value", "project"]).strip().decode()
#matplotlib
Explanation: Compute GitHub Stats
Notebook setup
End of explanation
import datetime
month = datetime.datetime.now().month
year = datetime.datetime.now().year
num_months = 12
months = []
for i in range(num_months):
    months.append("\"{0}{1:02}\"".format(year, month))
    month -= 1
    if month == 0:
        month = 12
        year -=1
Explanation: Setup Authorization
If you are using a service account run
%%bash
Activate Service Account provided by Kubeflow.
gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
If you are running using user credentials
gcloud auth application-default login
End of explanation
query = 
SELECT
    DATE(created_at) AS pr_date,
    actor.id,
    actor.login,
    JSON_EXTRACT(payload, '$.pull_request.user.id') as user_id,
    JSON_EXTRACT(payload, '$.pull_request.id') as pr_id,
    JSON_EXTRACT(payload, '$.pull_request.merged') as merged
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})
    AND type = 'PullRequestEvent'
    AND org.login = 'kubeflow'
    AND JSON_EXTRACT(payload, '$.action') IN ('"closed"')
.format(",".join(months))
all_prs=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
# Filter PRs to merged PRs
v=all_prs["merged"].values == 'true'
merged_all_prs = all_prs.iloc[v]
p=pd.Series(data=merged_all_prs["user_id"].values,index=merged_all_prs["pr_date"])
p=p.sort_index()
# Some solutions here: https://stackoverflow.com/questions/46470743/how-to-efficiently-compute-a-rolling-unique-count-in-a-pandas-time-series
# Need to figure out how to do a time based window
# TODO(jlewi): Is there a bug in the rolling window computation? creators ends up having the same number
# of rows as p; so we end up with multiple datapoints for each day; but the values aren't the same for
# each day. What is causing this effect?
creators = p.rolling('28d').apply(lambda arr: pd.Series(arr).nunique())
# We need to group the days. Rolling window will create a point for each data point
creators_df = pd.DataFrame({"day": creators.index, "num_authors": creators.values})
creators_df = creators_df.groupby("day", as_index=False).max()
import altair as alt
chart = alt.Chart(creators_df, title= "Unique PR Authors (Last 28 Days)")
line = chart.mark_line().encode(
  x= alt.X('day', title = "Day"),
  y=alt.Y("num_authors", title="# Unique Authors"),    
)
point = line + line.mark_point()
point.interactive()
Explanation: Unique PR Creators
End of explanation
pr_impulse=pd.Series(data=merged_all_prs["pr_id"].values,index=merged_all_prs["pr_date"])
pr_impulse=pr_impulse.sort_index()
unique_prs = pr_impulse.rolling('28d').apply(lambda arr: pd.Series(arr).nunique())
prs_df = pd.DataFrame({"day": unique_prs.index, "num_prs": unique_prs.values})
prs_df = prs_df.groupby("day", as_index=False).max()
chart = alt.Chart(prs_df, title= "Merged PRs (Last 28 Days)")
line = chart.mark_line().encode(
  x= alt.X('day', title = "Day"),
  y=alt.Y("num_prs", title="# PRs"),    
)
point = line + line.mark_point()
point.interactive()
Explanation: Number Prs
End of explanation
release_months = []
year = 2019
for month in range(8, 11):
    release_months.append("\"{0}{1:02}\"".format(year, month))
        
query = 
SELECT
    DATE(created_at) AS pr_date,
    actor.id,
    actor.login,
    JSON_EXTRACT(payload, '$.pull_request.merged') as merged,
    JSON_EXTRACT(payload, '$.pull_request.id') as pr_id,
    JSON_EXTRACT(payload, '$.pull_request.url') as pr_url,
    JSON_EXTRACT(payload, '$.pull_request.user.id') as user_id
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})
    AND type = 'PullRequestEvent'
    AND org.login = 'kubeflow'
    AND JSON_EXTRACT(payload, '$.action') IN ('"closed"')
.format(",".join(release_months))
prs=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
# Filter PRs to merged PRs
v=prs["merged"].values == 'true'
merged_prs = prs.iloc[v]
unique_pr_logins = prs["user_id"].unique()
unique_prs = prs["pr_id"].unique()
merged_unique_logins = merged_prs["user_id"].unique()
merged_unique_prs = merged_prs["pr_id"].unique()
print("Number of unique pr authors (merged & unmerged) {0}".format(unique_pr_logins.shape))
print("Number of unique prs (merged & unmerged) {0}".format(unique_prs.shape))
print("Number of unique pr authors (merged) {0}".format(merged_unique_logins.shape))
print("Number of unique prs (merged) {0}".format(merged_unique_prs.shape))
Explanation: Release stats per release (quarter)
Compute stats about a release 
We do this based on time
You can see a sample of the payload at https://api.github.com/repos/kubeflow/pipelines/pulls/1038
End of explanation
query = 
SELECT
    distinct JSON_EXTRACT(payload, '$.action')
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})    
.format(",".join(months))
actions=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
actions
Explanation: Get a list of distinct actions
Here's a list of events in the api
It looks like these are different from the ones in the github archive
End of explanation
query = 
SELECT
    DATE(created_at) AS issue_date,
    actor.id,
    actor.login,
    JSON_EXTRACT(payload, '$.pull_request.id') as issue_id,
    JSON_EXTRACT(payload, '$.pull_request.url') as issue_url  
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})
    AND type = 'IssuesEvent'
    AND org.login = 'kubeflow'
    AND JSON_EXTRACT(payload, '$.action') IN ('"opened"')
.format(",".join(months))
issues=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
issue_counts=issues["issue_date"].value_counts()
issue_counts=issue_counts.sort_index()
rolling_issue_count = issue_counts.rolling('28d').sum()
issues_df = pd.DataFrame({"day": rolling_issue_count.index, "num_issues": rolling_issue_count.values})
issues_df = issues_df.groupby("day", as_index=False).max()
# Truncate the first 28 days because it will be a windowing effect.
chart = alt.Chart(issues_df[28:], title= "New Issues (Last 28 Days)")
line = chart.mark_line().encode(
  x= alt.X('day', title = "Day"),
  y=alt.Y("num_issues", title="# issues"),    
)
point = line + line.mark_point()
point.interactive()
import matplotlib
from matplotlib import pylab
matplotlib.rcParams.update({'font.size': 22})
hf = pylab.figure()
hf.set_size_inches(18.5, 10.5)
pylab.plot(rolling_issue_count, linewidth=5)
ha = pylab.gca()
ha.set_title("New Kubeflow Issues (28 Days)")
ha.set_xlabel("Date")
ha.set_ylabel("# Of Issues")
Explanation: New Issues Last 28 Days
End of explanation
query = 
SELECT
    *
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})
    AND type = 'IssuesEvent'
    AND org.login = 'kubeflow'
    
    limit 20
.format(",".join(months))
events=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
events
Explanation: GetSomeSampleIssue Events
End of explanation
query = 
SELECT
    *
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})
    AND type = 'PullRequestEvent'
    AND org.login = 'kubeflow'
    
    limit 20
.format(",".join(months))
events=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
import pprint
import json
data = json.loads(events["payload"].values[3])
pprint.pprint(data)
data["pull_request"]["id"]
Explanation: Get some sample pull request events
Want to inspect the data
End of explanation
query = 
SELECT
    distinct type
  FROM `githubarchive.month.*`
  WHERE
    _TABLE_SUFFIX IN ({0})    
    AND org.login = 'kubeflow'
    
    limit 20
.format(",".join(months))
events=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)
events
Explanation: Get Distinct Types
End of explanation
<END_TASK>