<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Context
In https
Step1:  With the <tt>git_bin</tt>, we can execute almost any Git command we like directly. In our hypothetical use case, we want to retrieve some information about the change frequency of files. For this, we need the complete history of the Git repo including statistics for the changed files (via <tt>--numstat</tt>).
We use a little trick to make sure, that the format for the file's statistics fits nicely with the commit's metadata (SHA <tt>%h</tt>, UNIX timestamp <tt>%at</tt> and author's name <tt>%aN</tt>). The <tt>--numstat</tt> option provides data for additions, deletions and the affected file name in one line, separated by the tabulator character <tt>\t</tt>
Step2:  We now read in the complete files' history in the <tt>git_log</tt> variable. Don't let confuse you by all the <tt>\t</tt> characters. 
Let's read the result into a Pandas <tt>DataFrame</tt> by using the <tt>read_csv</tt> method. Because we can't provide a file path to a CSV data, we have to use StringIO to read in our in-memory buffered content.
Pandas will read the first line of the tabular-separated "file", sees the many tabular-separated columns and parses all other lines in the same format / column layout. Additionaly, we set the <tt>header</tt> to <tt>None</tt> because we don't have one and provide nice names for all the columns that we read in.
Step3:  The last steps are easy. We fill all the empty file statistics rows with the commit's metadata.
Step4:  And drop all the commit metadata rows that don't contain file statitics.
Step5:  We are finished! This is it. 
In summary, you'll need "one-liner" for converting a Git log file output that was exported with
git log --numstat --pretty=format
Step6:  Bonus section
As a bonus, we can now convert some columns to their right data types. The columns <tt>additions</tt> and <tt>deletions</tt> columns are representing the added or deletes lines of code respectively. But there are also a few exceptions for binary files like the images. We skip these lines with the <tt>errors='coerce'</tt> option. This will lead to <tt>Nan</tt>< in the rows that will be dropped after the converstion. 
The <tt>timestamp</tt> column is a UNIX timestamp with the past seconds since January 1st 1970.
Step7:  After this, we have to tell Git which information we want. We can do this via the <tt>pretty-format</tt> option. 
For each commit, we choose to create a header line with the following commit info (by using <tt>--pretty=format
Step8:  OK, this part is ready, let's have a look at the file statistics!
We're done!
Complete code block
To much code to look through? Here is everything from above in a condensed format.
Just some milliseconds to run through, not bad!
Summary
In this notebook, I showed you how to read some non-perfect structured data via the non-character separator trick. I also showed you how to transform the rows that contain multiple kinds of data into one nicely structured <tt>DataFrame</tt>.
Now that we have the Git repository <tt>DataFrame</tt>, we can do some nice things with it e. g. visualizing the code churn of a project, but that's a story for another notebook! But to give you a short preview
Step9:  Discussion
I hope you see how easy it is to retrieve some insights from your version control system by using Python and Pandas for some data wrangling. Because it's too good to be true, there are the drawbacks of this simple approach
<ASSISTANT_TASK:>
Python Code:
import git 
GIT_LOG_FILE = r'${REPO}/spring-petclinic'
repo = git.Repo(GIT_LOG_FILE)
git_bin = repo.git
git_bin
Explanation: Context
In https://www.feststelltaste.de/reading-a-git-log-file-output-with-pandas/ I show you a way to read in Git log data with Pandas's DataFrame and GitPython.
Looking back, this was really difficult and tedious to do. So with a few tricks we can do it much more better.
The idea
There are three new ideas that I introduce here:
We use GitPython's feature to directly access an underlying Git installation. This is way more faster than using GitPython's object representation of the repository makes it possible to have everything we need in one notebook.
We use in-memory reading by using StringIO to avoid unnecessary file access. This avoids storing the Git output on disk and read is again from disc. Way more faster, too.
We also hack Pandas's <tt>read_csv</tt> method even more. This makes the transformation of the Git log as easy as pie.
Side Note
This method also scales for analyzing huge GitHub repositories like the one for Linux. In the case for the Linux repo it has as of today almost 700.000 commits. But be warned: Cloning the repo as well as retrieving the Git log informaiton takes a while. I recommend cloning my fork because I fixed some encoding errors in author's names via an extended <tt>.mailmap</tt> file that maps these weird names to real names. Otherwise Python cannot read in the Git log because of an UnicodeError exception. On Windows machines, you'll also get some weird errors while cloning. But this doesn't influence the analysis.
Reading the Git log
The first step is to connect GitPython with the Git repo. If we have an instance of the repo, we can gain access to the underlying Git installation of the operation system via <tt>repo.git</tt>.
In this case, again, we tap the Spring Pet Clinic project, a small sample application for the Spring framework.
End of explanation
git_log = git_bin.execute('git log --numstat --pretty=format:"\t\t\t%h\t%at\t%aN"')
git_log[:100]
Explanation: With the <tt>git_bin</tt>, we can execute almost any Git command we like directly. In our hypothetical use case, we want to retrieve some information about the change frequency of files. For this, we need the complete history of the Git repo including statistics for the changed files (via <tt>--numstat</tt>).
We use a little trick to make sure, that the format for the file's statistics fits nicely with the commit's metadata (SHA <tt>%h</tt>, UNIX timestamp <tt>%at</tt> and author's name <tt>%aN</tt>). The <tt>--numstat</tt> option provides data for additions, deletions and the affected file name in one line, separated by the tabulator character <tt>\t</tt>:  
<p>
<tt>1<b>\t</b>1<b>\t</b>some/file/name.ext</tt>
</p>
We use the same tabular separator <tt>\t</tt> for the format string:
<p>
<tt>%h<b>\t</b>%at<b>\t</b>%aN</tt>
</p>
And here is the trick: Additionally, we add the amount of tabulators of the file's statistics plus an additional tabulator in front of the format string to pretend that there are empty file statistics' information in front of the format string.
The results looks like this:
<p>
<tt>\t\t\t%h\t%at\t%aN</tt>
</p>
Note: If you want to export the Git log on the command line into a file to read that file later, you need to use the tabulator character xxx as separator instead of <tt>\t</tt> in the format string. Otherwise, the trick doesn't work.
OK, let's first executed the Git log export:
End of explanation
import pandas as pd
from io import StringIO
commits_raw = pd.read_csv(StringIO(git_log), 
    sep="\t",
    header=None,              
    names=['additions', 'deletions', 'filename', 'sha', 'timestamp', 'author']
    )
commits_raw.head()
Explanation: We now read in the complete files' history in the <tt>git_log</tt> variable. Don't let confuse you by all the <tt>\t</tt> characters. 
Let's read the result into a Pandas <tt>DataFrame</tt> by using the <tt>read_csv</tt> method. Because we can't provide a file path to a CSV data, we have to use StringIO to read in our in-memory buffered content.
Pandas will read the first line of the tabular-separated "file", sees the many tabular-separated columns and parses all other lines in the same format / column layout. Additionaly, we set the <tt>header</tt> to <tt>None</tt> because we don't have one and provide nice names for all the columns that we read in.
End of explanation
commits = commits_raw.fillna(method='ffill')
commits.head()
Explanation: The last steps are easy. We fill all the empty file statistics rows with the commit's metadata.
End of explanation
commits = commits.dropna()
commits.head()
Explanation: And drop all the commit metadata rows that don't contain file statitics.
End of explanation
pd.read_csv("../../spring-petclinic/git.log", 
            sep="\t", 
            header=None,
            names=[
               'additions', 
               'deletions', 
               'filename', 
               'sha', 
               'timestamp', 
               'author']).fillna(method='ffill').dropna().head()
Explanation: We are finished! This is it. 
In summary, you'll need "one-liner" for converting a Git log file output that was exported with
git log --numstat --pretty=format:"%x09%x09%x09%h%x09%at%x09%aN" &gt; git.log
into a <tt>DataFrame</tt>:
End of explanation
commits['additions'] = pd.to_numeric(commits['additions'], errors='coerce')
commits['deletions'] = pd.to_numeric(commits['deletions'], errors='coerce')
commits = commits.dropna()
commits['timestamp'] = pd.to_datetime(commits['timestamp'], unit="s")
commits.head()
commits.groupby('filename')[['timestamp']].count().sort_values(by='timestamp', ascending=False).head(10)
java_commits = commits[commits['filename'].str.endswith(".c")]
java_commits.head()
java_commits.groupby('author').sum()[['additions']].sort_values(by='additions', ascending=False).head()
commits[commits['timestamp'].max() == commits['timestamp']]
java_commits[java_commits['timestamp'].min() == java_commits['timestamp']]
commits = commits[commits['timestamp'] <= 'today']
latest = commits.sort_values(by='timestamp', ascending=False)
latest.head()
commits['today'] = pd.Timestamp('today')
commits.head()
initial_commit_date = commits[-1:]['timestamp'].values[0]
initial_commit_date
commits = commits[commits['timestamp'] >= initial_commit_date]
commits.head()
commits['age'] = commits['timestamp'] - commits['today']
commits.head()
commits.groupby('filename')[['age']].min().sort_values(by='age').head(10)
java_commits.groupby('filename')\
    .count()[['additions']]\
    .sort_values(by='additions', ascending=False).head()
ages = commits.sort_values(by='age', ascending=False).drop_duplicates(subset=['filename'])['age'] * -1
ages.head()
ages.dt.days.hist()
commits.groupby('filename')
import glob
file_list = [
    os.path.abspath(path).replace(os.sep, "/") for path in glob.iglob("../../linux/**/*.*")]
file_list[:5]
[os.path.normpath
%matplotlib inline
commits.groupby('filename')\
    .count()[['additions']]\
    .sort_values(by='additions', ascending=False)\
    .plot(kind='bar')
commits.sort_values(by='age', ascending=False).groupby('filename').first().sort_values(by='age', ascending=False)
%matplotlib inline
commits.groupby('filename')\
    .count()['additions']\
    .hist(bins=20)
commits.groupby('filename').count().sort_values(by='additions', ascending=False)
Explanation: Bonus section
As a bonus, we can now convert some columns to their right data types. The columns <tt>additions</tt> and <tt>deletions</tt> columns are representing the added or deletes lines of code respectively. But there are also a few exceptions for binary files like the images. We skip these lines with the <tt>errors='coerce'</tt> option. This will lead to <tt>Nan</tt>< in the rows that will be dropped after the converstion. 
The <tt>timestamp</tt> column is a UNIX timestamp with the past seconds since January 1st 1970.
End of explanation
commits.groupby('author').sum()[['additions']].sort_values(by='additions', ascending=False)
Explanation: After this, we have to tell Git which information we want. We can do this via the <tt>pretty-format</tt> option. 
For each commit, we choose to create a header line with the following commit info (by using <tt>--pretty=format:'--%h--%ad--%aN'</tt>), which gives us the following output:
<pre>
--fa1ca6f--Thu Dec 22 08:04:18 2016 +0100--feststelltaste
</pre>
It contains the SHA key, the timestamp as well as the author's name of the commit, separated by a character that isn't certaninly in these information<tt>--</tt>. My favorite separator for this is <tt>\u0012</tt>
We also want to have some details about the modifications of each file per commit. This is why we use the <tt>--numstat</tt> flag.
Together with the <tt>--all</tt> flag to get all commits and the <tt>--no-renames</tt> flag to avoid commits that only rename files, we retrieve all the needed information directly via Git.
For each other row, we got some statistics about the modified files:
<pre>
2   0   src/main/asciidoc/appendices/bibliography.adoc
</pre>
It contains the number of lines inserted, the number of lines deleted and the relative path of the file. With a little trick and a little bit of data wrangling, we can read that information into a nicely structured DataFrame.
The first entries of that file look something like this:
Let's get started!
Import the data
First, I'll show you my approach on how to read nearly everything into a <tt>DataFrame</tt>. The key is to use Pandas' <tt>read_csv</tt> for reading "non-character separated values". How to do that? We simply choose a separator that doesn't occur in the file that we want to read. My favorite character for this is the "DEVICE CONTROL TWO" character U+0012. I haven't encountered a situation yet where this character was included in a data set.
We just read our <tt>git.log</tt> file without any headers (because there are none) and give the only column a nice name.
Data Wrangling
OK, but now we have a <strike>problem</strike> data wrangling challenge. We have the commit info as well as the statistic for the modified file in one column, but they don't belong together. What we want is to have the commit info along with the file statistics in separate columns to get some serious analysis started.
End of explanation
%matplotlib inline
timed_commits = java_commits.set_index(pd.DatetimeIndex(java_commits['timestamp']))[['additions', 'deletions']].resample('1D').sum()
timed_commits
(timed_commits['additions'] - timed_commits['deletions']).cumsum().fillna(method='ffill').plot()
c = commits[commits['timestamp'] <= 'today']
c.sort_values(by='timestamp', ascending=False).head()
c = c\
.groupby('sha')\
.first()\
.reset_index()
c.head()
%matplotlib inline
c.set_index(
    pd.DatetimeIndex(c['timestamp'])
)['additions']\
.resample('W-SUN', convention='start')\
.count()\
.tail(500)\
.plot(kind='area', figsize=(100,7))
c.set_index(
    pd.DatetimeIndex(c['timestamp'])
)['additions']\
.resample('W-SUN', convention='start')\
.count()\
.tail(500)\
df = c.set_index(
    pd.DatetimeIndex(c['timestamp']))
df2 = df.resample('W').count().dropna()
df2.tail()
df2['month'] = df2.index.month
df2.head()
df3 = df2.groupby([df2.index.year, df2.index.month]).aggregate({'month': 'first', 'sha' : 'min'})
df3.head()
df3.groupby(df3.index).count()
Explanation: OK, this part is ready, let's have a look at the file statistics!
We're done!
Complete code block
To much code to look through? Here is everything from above in a condensed format.
Just some milliseconds to run through, not bad!
Summary
In this notebook, I showed you how to read some non-perfect structured data via the non-character separator trick. I also showed you how to transform the rows that contain multiple kinds of data into one nicely structured <tt>DataFrame</tt>.
Now that we have the Git repository <tt>DataFrame</tt>, we can do some nice things with it e. g. visualizing the code churn of a project, but that's a story for another notebook! But to give you a short preview:
End of explanation
%matplotlib inline
commits['author'].value_counts().plot(kind='pie', figsize=(10,10))
Explanation: Discussion
I hope you see how easy it is to retrieve some insights from your version control system by using Python and Pandas for some data wrangling. Because it's too good to be true, there are the drawbacks of this simple approach:
Files, not code
We observe the file change frequency, not the code change frequency: 
All files
We also didn't look up if the files we are analyzing are still existing or have been deleted in the past. Because for our change analysis,
End of explanation
<END_TASK>