<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 To see where we got this dataset from, check out our last notebook on scraping GitHub.
To get plotting, we start by importing the holy trinity of Python
Step1:  We read in the anaonymised data to a pands dataframe and make a second dataframe of only commits to OHW20 repos
Step2:  Commits in time
Step3:  How about zooming in on hackweek?
Step4:  Zoom closer?
Step5:  Hopefuly an indication of Oceanhackweek's global participation, not some sorely sleep deprived coders!
How many unique participants wrote commits during these weeks?
Step6:  What is the distribution of commits amongst participants like?
Step7:  Commit mesages and extensions
A basic bit of NLP with Counter and the WordCloud library
What were the most common file types pushed to git?
Step8:  This needs a bit of cleaning
Step9:  Jupyter notebooks (ipynb) overwhelmingly popular
For a slighly fancier visualisation, lets make a wordlcoud
Step10:  How about git commit messages?
Step11:  That's it! (for now)
I have only scratched the surface of the data you can get from scraping github here. For more ideas on what to do with this data, check out the end of the other notebook
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
Explanation: To see where we got this dataset from, check out our last notebook on scraping GitHub.
To get plotting, we start by importing the holy trinity of Python
End of explanation
df = pd.read_csv('ohw_anonymised.csv', index_col='datetime', parse_dates=True)
df_ohw20 = df[df.ohw20_repo]
df
Explanation: We read in the anaonymised data to a pands dataframe and make a second dataframe of only commits to OHW20 repos
End of explanation
weekly_commits = df.author.groupby(df.index.week).count()
df_hw = df[df.index.week==33]
hw_commits = df_hw.author.groupby(df_hw.index.week).count()
plt.rcParams.update({'font.size': 18})
fig, ax = plt.subplots(figsize=(15,10))
ax.bar(weekly_commits.index, weekly_commits.values, label='Weeks of 2020')
ax.bar(hw_commits.index, hw_commits.values, label='OHW20')
ax.legend()
ax.set(xlabel='week of 2020', ylabel='Commits by OHW participants')
Explanation: Commits in time
End of explanation
daily_commits = df_hw.author.groupby(df_hw.index.day).count()
fig, ax = plt.subplots(figsize=(12,8))
ax.bar(daily_commits.index, daily_commits.values)
ax.set(xlabel='day of hackweek', ylabel='Commits by OHW participants')
ax.set_xticks(np.arange(10,17))
ax.set_xticklabels(['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']);
Explanation: How about zooming in on hackweek?
End of explanation
df_friday = df_hw[df_hw.index.day==14]
friday_commits = df_friday.author.groupby(df_friday.index.hour).count()
fig, ax = plt.subplots(figsize=(12,8))
ax.bar(friday_commits.index, friday_commits.values)
ax.set(xlabel='Hour of Friday (UTC)', ylabel='Commits by OHW participants')
Explanation: Zoom closer?
End of explanation
# severley unpythonic but it works
week_commits = []
unique_users = []
for week in weekly_commits.index:
    sub_df = df[df.index.week==week]
    unique_users.append(len(sub_df.groupby('author').count()))
fig, ax = plt.subplots(figsize=(15,10))
ax.bar(weekly_commits.index, unique_users)
ax.bar(33, max(unique_users))
ax.set(xlabel='week of 2020', ylabel='Unique OHW20 participants with commits')
Explanation: Hopefuly an indication of Oceanhackweek's global participation, not some sorely sleep deprived coders!
How many unique participants wrote commits during these weeks?
End of explanation
from collections import Counter
c = Counter(df_ohw20[df_ohw20.ohw20_repo].author)
MVP = c.most_common()
mvp_names, mvp_nums = [], []
for tup in MVP:
    mvp_names.append(tup[0])
    mvp_nums.append(tup[1])
fig, ax = plt.subplots(figsize=(14,8))
ax.bar(mvp_names, mvp_nums)
ax.set(xticks=[], ylabel='Number of commits per participant');
Explanation: What is the distribution of commits amongst participants like?
End of explanation
from collections import Counter
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
all_commit_strings = ''.join(df_ohw20.message)
extensions = ''.join(str(df_ohw20.extensions.values))
extensions
Explanation: Commit mesages and extensions
A basic bit of NLP with Counter and the WordCloud library
What were the most common file types pushed to git?
End of explanation
extensions_no_newline = extensions.replace('\n','')
extensions_no_quotes = extensions_no_newline.replace("'", "")
extensions_no_commas = extensions_no_quotes.replace(",", "")
extensions_no_commas
extensions_list = extensions_no_commas.split(" ")
c = Counter(extensions_list)
top16 = c.most_common(16)
word, counts = [], []
for tup in top16:
    word.append(tup[0])
    counts.append(tup[1])
fig, ax = plt.subplots(figsize=(12,8))
ax.bar(word,counts)
plt.setp(ax.get_xticklabels(), ha="right", rotation=45);
ax.set_title("Occurence of filetypes in commits to OHW20 repos")
Explanation: This needs a bit of cleaning
End of explanation
stopwords = set(STOPWORDS)
wc = WordCloud(background_color="white", max_words=16,collocations=False,
               stopwords=stopwords, contour_width=3, contour_color='steelblue')
wordcloud = wc.generate(extensions_no_commas)
fig, ax = plt.subplots(figsize=(12,8))
ax.imshow(wordcloud, interpolation='bilinear')
ax.axis("off")
Explanation: Jupyter notebooks (ipynb) overwhelmingly popular
For a slighly fancier visualisation, lets make a wordlcoud
End of explanation
wc = WordCloud(background_color="white", max_words=33,collocations=False,
               stopwords=stopwords, contour_width=3, contour_color='steelblue')
wordcloud = wc.generate(all_commit_strings)
fig, ax = plt.subplots(figsize=(12,8))
ax.imshow(wordcloud, interpolation='bilinear')
ax.axis("off")
Explanation: How about git commit messages?
End of explanation
HTML(html)
Explanation: That's it! (for now)
I have only scratched the surface of the data you can get from scraping github here. For more ideas on what to do with this data, check out the end of the other notebook
End of explanation
<END_TASK>