<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction
In my previous blog post, we've seen how we can identify files that change together in one commit.
In this blog post, we take the analysis to an advanced level
Step1:  In our case, we only want to check the modularization of our software for Java production code. So we just leave the files that are belonging to the main source code. What to keep here exactly is very specific to your own project. With Jupyter and pandas, we can make our decisions for this transparent and thus retraceable.
Step2:  Analysis
We want to see which files are changing (almost) together. A good start for this is to create this view onto our dataset with the pivot_table method of the underlying pandas' DataFrame. 
But before this, we need a marker column that signals that a commit occurred. We can create an additional column named hit for this easily.
Step3:  Now, we can transform the data as we need it
Step4:  As already mentioned in a previous blog post, we are now able to look at our problem from a mathematician' s perspective. What we have here now with the commit_matrix is a collection of  n-dimensional vectors. Each vector represents a filename and the components/dimensions of such a vector are the commits with either the value 0 or 1. 
Calculating similarities between such vectors is a well-known problem with a variety of solutions. In our case, we calculate the distance between the various vectors with the cosines distance metric. The machine learning library scikit-learn provides us with an easy to use implementation.
Step5:  To be able to better understand the result, we add the file names from the commit_matrix as index and column index  to the dissimilarity_matrix.
Step6:  Now, we see the result in a better representation
Step7:  Because of the alphabetically ordered filenames and the "feature-first" architecture of the  software under investigation, we get the first glimpse of how changes within modules are occurring together and which are not.
To get an even better view, we can first extract the module's names with an easy string operation and use this for the indexes.
Step8:  Then, we can create another heatmap that shows the name of the modules on both axes for further evaluation. We also just take a look at a subset of the data for representational reasons.
Step9:  Discussion
Starting at the upper left, we see the "comment" module with a pretty dark area very clearly. This means, that files around this module changed together very often.
If we go to the middle left, we see dark areas between the "comment" module and the "framework" module as well as the "site" module further down. This shows a change dependency between the "comment" module and the other two (I'll explain later, why it is that way).
If we take a look in the middle of the heatmap, we see that the very dark area represents changes of the "mail" module. This module was pretty much changed without touching any other modules. This shows a nice separation of concerns.
For the "scheduling" module, we can also see that the changes occurred mostly cohesive within the module.
Another interesting aspect is the horizontal line within the "comment" region
Step10:  The result is a 2D matrix that we can plot with matplotlib to get a first glimpse of the distribution of the calculated distances.
Step11:  With the plot above, we see that the 2D transformation somehow worked. But we can't see
* which filenames are which data points
* how the modules are grouped all together
So we need to enrich the data a little bit more and search for a better, interactive visualization technique.
Let's add the filenames to the matrix as well as nice column names. We, again, add the information about the module of a source code file to the DataFrame.
Step12:  Author
Step13:  OK, here comes the ugly part
Step14:  With this nice little data structure, we can fill pygal's XY chart and create an interactive chart.
<ASSISTANT_TASK:>
Python Code:
from lib.ozapfdis.git_tc import log_numstat
GIT_REPO_DIR = "../../dropover_git/"
git_log = log_numstat(GIT_REPO_DIR)[['sha', 'file', 'author']]
git_log.head()
Explanation: Introduction
In my previous blog post, we've seen how we can identify files that change together in one commit.
In this blog post, we take the analysis to an advanced level:
We're using a more robust model for determining the similarity of co-changing source code files
We're checking the existing modularization of a software system and compare it to the change behavior of the development teams
We're creating a visualization that lets us determine the underlying, "hidden" modularization of our software system based on conjoint changes
We discuss the results for a concrete software system in detail (with more systems to come in the upcoming blog posts).
We're using Python and pandas as well as some algorithms from the machine learning library scikit-learn and the visualization libraries matplotlib, seaborn and pygal for these purposes.
The System under Investigation
For this analysis, we use a closed-source project that I developed with some friends of mine. It's called "DropOver", a web application that can manage events with features like events' sites, scheduling, comments, todos, file uploads, mail notifications and so on. The architecture of the software system mirrored the feature-based development process: You could quickly locate where code has to be added or changed because the software system's "screaming architecture". This architecture style lead you to the right place because of the explicit, feature-based modularization that was used for the Java packages/namespaces:
It's also important to know, that we developed the software almost strictly feature-based by feature teams (OK, one developer was one team in our case). Nevertheless, the history of this repository should perfectly fit for our analysis of checking the modularization based on co-changing source code files. 
The main goal of our analysis is to see if the modules of the software system were changed independently or if they were code was changed randomly across modules boundaries. If the latter would be the case, we should reorganize the software system or the development teams to let software development activities and the surrounding more naturally fit together.
Idea
We can do this kind of analysis pretty easily by using the version control data of a software system like Git. A version control system tracks each change to a file. If more files are changed within one commit, we can assume that those files somehow have something to do with each other. This could be e. g. a direct dependency because two files depend on each other or a semantic dependency which causes an underlying concepts to change across module boundaries.
In this blog post, we take the idea further: We want to find out the degree of similarity of two co-changing files, making the analysis more robust and reliable on one side, but also enabling a better analysis of bigger software systems on the other side by comparing all files of a software system with each other regarding the co-changing properties.
Data
We use a little helper library for importing the data of our project. It's a simple git log with change statistics for each commit and file (you can see here how to retrieve it if you want to do it manually).
End of explanation
prod_code = git_log.copy()
prod_code = prod_code[prod_code.file.str.endswith(".java")]
prod_code = prod_code[prod_code.file.str.startswith("backend/src/main")]
prod_code = prod_code[~prod_code.file.str.endswith("package-info.java")]
prod_code.head()
Explanation: In our case, we only want to check the modularization of our software for Java production code. So we just leave the files that are belonging to the main source code. What to keep here exactly is very specific to your own project. With Jupyter and pandas, we can make our decisions for this transparent and thus retraceable.
End of explanation
prod_code['hit'] = 1
prod_code.head()
Explanation: Analysis
We want to see which files are changing (almost) together. A good start for this is to create this view onto our dataset with the pivot_table method of the underlying pandas' DataFrame. 
But before this, we need a marker column that signals that a commit occurred. We can create an additional column named hit for this easily.
End of explanation
commit_matrix = prod_code.reset_index().pivot_table(
    index='file',
    columns='sha',
    values='hit',
    fill_value=0)
commit_matrix.iloc[0:5,50:55]
Explanation: Now, we can transform the data as we need it: For the index, we choose the filename, as columns, we choose the unique sha key of a commit. Together with the commit hits as values, we are now able to see which file changes occurred in which commit. Note that the pivoting also change the order of both indexes. They are now sorted alphabetically.
End of explanation
from sklearn.metrics.pairwise import cosine_distances
dissimilarity_matrix = cosine_distances(commit_matrix)
dissimilarity_matrix[:5,:5]
Explanation: As already mentioned in a previous blog post, we are now able to look at our problem from a mathematician' s perspective. What we have here now with the commit_matrix is a collection of  n-dimensional vectors. Each vector represents a filename and the components/dimensions of such a vector are the commits with either the value 0 or 1. 
Calculating similarities between such vectors is a well-known problem with a variety of solutions. In our case, we calculate the distance between the various vectors with the cosines distance metric. The machine learning library scikit-learn provides us with an easy to use implementation.
End of explanation
import pandas as pd
dissimilarity_df = pd.DataFrame(
    dissimilarity_matrix,
    index=commit_matrix.index,
    columns=commit_matrix.index)
dissimilarity_df.iloc[:5,:2]
Explanation: To be able to better understand the result, we add the file names from the commit_matrix as index and column index  to the dissimilarity_matrix.
End of explanation
%matplotlib inline
import seaborn as sns
sns.heatmap(
    dissimilarity_df,
    xticklabels=False,
    yticklabels=False
);
Explanation: Now, we see the result in a better representation: For each file pair, we get the distance of the commit vectors. This means that we have now a distance measure that says how dissimilar two files were changed in respect to each other.
Visualization
Heatmap
To get an overview of the result's data, we can plot the matrix with a little heatmap first.
End of explanation
modules = dissimilarity_df.copy()
modules.index = modules.index.str.split("/").str[6]
modules.index.name = 'module'
modules.columns = modules.index
modules.iloc[25:30,25:30]
Explanation: Because of the alphabetically ordered filenames and the "feature-first" architecture of the  software under investigation, we get the first glimpse of how changes within modules are occurring together and which are not.
To get an even better view, we can first extract the module's names with an easy string operation and use this for the indexes.
End of explanation
import matplotlib.pyplot as plt
plt.figure(figsize=[10,9])
sns.heatmap(modules.iloc[:180,:180]);
Explanation: Then, we can create another heatmap that shows the name of the modules on both axes for further evaluation. We also just take a look at a subset of the data for representational reasons.
End of explanation
from sklearn.manifold import MDS
# uses a fixed seed for random_state for reproducibility
model = MDS(dissimilarity='precomputed', random_state=0)
dissimilarity_2d = model.fit_transform(dissimilarity_df)
dissimilarity_2d[:5]
Explanation: Discussion
Starting at the upper left, we see the "comment" module with a pretty dark area very clearly. This means, that files around this module changed together very often.
If we go to the middle left, we see dark areas between the "comment" module and the "framework" module as well as the "site" module further down. This shows a change dependency between the "comment" module and the other two (I'll explain later, why it is that way).
If we take a look in the middle of the heatmap, we see that the very dark area represents changes of the "mail" module. This module was pretty much changed without touching any other modules. This shows a nice separation of concerns.
For the "scheduling" module, we can also see that the changes occurred mostly cohesive within the module.
Another interesting aspect is the horizontal line within the "comment" region: These files were changed independently from all other files within the module. These files were the code for an additional data storage technology that was added in later versions of the software system. This pattern repeats for all other modules more or less strongly.
With this visualization, we can get a first impression of how good our software architecture fits the real software development activities. In this case, I would say that you can see most clearly that the source code of the modules changed mostly within the module boundaries. But we have to take a look at the changes that occur in other modules as well when changing a particular module. These could be signs of unwanted dependencies and may lead us to an architectural problem.
Multi-dimensional Scaling
We can create another kind of visualization to check 
* if the code within the modules is only changed altogether and
* if not, what other modules were changed.
Here, we can help ourselves with a technique called "multi-dimensional scaling" or "MDS" for short. With MDS, we can break down an n-dimensional space to a lower-dimensional space representation. MDS tries to keep the distance proportions of the higher-dimensional space when breaking it down to a lower-dimensional space.
In our case, we can let MDS figure out a 2D representation of our dissimilarity matrix (which is, overall, just a plain multi-dimensional vector space) to see which files get change together. With this, we'll able to see which files are changes together regardless of the modules they belong to.
The machine learning library scikit-learn gives us easy access to the algorithm that we need for this task as well. We just need to say that we have a precomputed dissimilarity matrix when initializing the algorithm and then pass our dissimilarity_df DataFrame to the fit_transform method of the algorithm.
End of explanation
plt.figure(figsize=(8,8))
x = dissimilarity_2d[:,0]
y = dissimilarity_2d[:,1]
plt.scatter(x, y);
Explanation: The result is a 2D matrix that we can plot with matplotlib to get a first glimpse of the distribution of the calculated distances.
End of explanation
dissimilarity_2d_df = pd.DataFrame(
    dissimilarity_2d,
    index=commit_matrix.index,
    columns=["x", "y"])
dissimilarity_2d_df.head()
Explanation: With the plot above, we see that the 2D transformation somehow worked. But we can't see
* which filenames are which data points
* how the modules are grouped all together
So we need to enrich the data a little bit more and search for a better, interactive visualization technique.
Let's add the filenames to the matrix as well as nice column names. We, again, add the information about the module of a source code file to the DataFrame.
End of explanation
prod_code.groupby(['file', 'author'])['hit'].count().groupby(['file', 'author']).max()
dissimilarity_2d_df['module'] = dissimilarity_2d_df.index.str.split("/").str[6]
Explanation: Author
End of explanation
plot_data = pd.DataFrame(index=dissimilarity_2d_df['module'])
plot_data['value'] = tuple(zip(dissimilarity_2d_df['x'], dissimilarity_2d_df['y']))
plot_data['label'] = dissimilarity_2d_df.index
plot_data['data'] = plot_data[['label', 'value']].to_dict('records')
plot_dict = plot_data.groupby(plot_data.index).data.apply(list)
plot_dict
Explanation: OK, here comes the ugly part: We have to transform all the data to the format our interactive visualization library pygal needs for its XY chart. We need to 
* group the data my modules
* add every distance information 
 * for each file as well as
 * the filename itself  
in a specific dictionary-like data structure.
But there is nothing that can hinder us in Python and pandas. So let's do this!
We create a separate DataFrame named plot_data with the module names as index
We join the coordinates x and y into a tuple data structure
We use the filenames from dissimilarity_2d_df's index as labels
We convert both data items to a dictionary
We append each entry for a module to only on module entry
This gives us a new DataFrame with modules as index and per module a list of dictionary-like entries with 
* the filenames as labels and
* the coordinates as values.
End of explanation
import pygal
xy_chart = pygal.XY(stroke=False)
[xy_chart.add(entry[0], entry[1]) for entry in plot_dict.iteritems()]   
# uncomment to create the interactive chart
# xy_chart.render_in_browser()
xy_chart
Explanation: With this nice little data structure, we can fill pygal's XY chart and create an interactive chart.
End of explanation
<END_TASK>