<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 After this, we have to tell Git which information we want. We can do this via the <tt>pretty-format</tt> option. 
For each commit, we choose to create a header line with the following commit info (by using <tt>--pretty=format
Step1:  Data Wrangling
OK, but now we have a <strike>problem</strike> data wrangling challenge. We have the commit info as well as the statistic for the modified file in one column, but they don't belong together. What we want is to have the commit info along with the file statistics in separate columns to get some serious analysis started.
Commit info
Let's treat the commit info first. Luckily, we set some kind of anchor or marker to identify the commit info
Step2:  With this, we can focus on extracting the information of a commit info row. The next command could be looking a little frightening, but don't worry. We go through it step by step.
Step3:  OK, this part is ready, let's have a look at the file statistics!
We're done!
Complete code block
To much code to look through? Here is everything from above in a condensed format.
Just some milliseconds to run through, not bad!
Summary
In this notebook, I showed you how to read some non-perfect structured data via the non-character separator trick. I also showed you how to transform the rows that contain multiple kinds of data into one nicely structured <tt>DataFrame</tt>.
Now that we have the Git repository <tt>DataFrame</tt>, we can do some nice things with it e. g. visualizing the code churn of a project, but that's a story for another notebook! But to give you a short preview
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
from io import StringIO
commits_raw = pd.read_csv(StringIO(log), 
    sep="#",
    header=None,
    names=['file_stats','sha', 'date', 'author'])
commits_raw.head()    
Explanation: After this, we have to tell Git which information we want. We can do this via the <tt>pretty-format</tt> option. 
For each commit, we choose to create a header line with the following commit info (by using <tt>--pretty=format:'--%h--%ad--%aN'</tt>), which gives us the following output:
<pre>
--fa1ca6f--Thu Dec 22 08:04:18 2016 +0100--feststelltaste
</pre>
It contains the SHA key, the timestamp as well as the author's name of the commit, separated by a character that isn't certaninly in these information<tt>--</tt>. My favorite separator for this is <tt>\u0012</tt>
We also want to have some details about the modifications of each file per commit. This is why we use the <tt>--numstat</tt> flag.
Together with the <tt>--all</tt> flag to get all commits and the <tt>--no-renames</tt> flag to avoid commits that only rename files, we retrieve all the needed information directly via Git.
For each other row, we got some statistics about the modified files:
<pre>
2   0   src/main/asciidoc/appendices/bibliography.adoc
</pre>
It contains the number of lines inserted, the number of lines deleted and the relative path of the file. With a little trick and a little bit of data wrangling, we can read that information into a nicely structured DataFrame.
The first entries of that file look something like this:
Let's get started!
Import the data
First, I'll show you my approach on how to read nearly everything into a <tt>DataFrame</tt>. The key is to use Pandas' <tt>read_csv</tt> for reading "non-character separated values". How to do that? We simply choose a separator that doesn't occur in the file that we want to read. My favorite character for this is the "DEVICE CONTROL TWO" character U+0012. I haven't encountered a situation yet where this character was included in a data set.
We just read our <tt>git.log</tt> file without any headers (because there are none) and give the only column a nice name.
End of explanation
commit_metadata = commits_raw[['sha', 'date', 'author']].fillna(method='ffill')
commit_metadata.head(5)
Explanation: Data Wrangling
OK, but now we have a <strike>problem</strike> data wrangling challenge. We have the commit info as well as the statistic for the modified file in one column, but they don't belong together. What we want is to have the commit info along with the file statistics in separate columns to get some serious analysis started.
Commit info
Let's treat the commit info first. Luckily, we set some kind of anchor or marker to identify the commit info: Each commit info starts with a <tt>--</tt>. So let's extract all the commit info from the original <tt>commits</tt> <tt>DataFrame</tt>.
End of explanation
file_info = commits_raw['file_stats'].dropna().str.split("\t", expand=True)
file_info.columns = ['additions', "deletions", "filename"]
file_info.head()
file_info['additions'] = pd.to_numeric(file_info['additions'], errors='coerce')
file_info['deletions'] = pd.to_numeric(file_info['deletions'], errors='coerce')
file_info.dtypes
commits = commit_metadata.join(file_info, how='right')
commits = commits.dropna()
commits.head()
commits.groupby('author').sum()[['additions']].sort_values(by='additions', ascending=False)
Explanation: With this, we can focus on extracting the information of a commit info row. The next command could be looking a little frightening, but don't worry. We go through it step by step.
End of explanation
%matplotlib inline
timed_commits = commits.set_index(pd.DatetimeIndex(commits['date']))[['insertions', 'deletions']].resample('1D').sum()
(timed_commits['insertions'] - timed_commits['deletions']).cumsum().fillna(method='ffill').plot()
%matplotlib inline
commits['author'].value_counts().plot(kind='pie', figsize=(10,10))
Explanation: OK, this part is ready, let's have a look at the file statistics!
We're done!
Complete code block
To much code to look through? Here is everything from above in a condensed format.
Just some milliseconds to run through, not bad!
Summary
In this notebook, I showed you how to read some non-perfect structured data via the non-character separator trick. I also showed you how to transform the rows that contain multiple kinds of data into one nicely structured <tt>DataFrame</tt>.
Now that we have the Git repository <tt>DataFrame</tt>, we can do some nice things with it e. g. visualizing the code churn of a project, but that's a story for another notebook! But to give you a short preview:
End of explanation
<END_TASK>