<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction
Idea
The claim was that the directory structure would be very similar to each other over a period of time. We want to identify this time span by using a time-based analysis on the commits and their corresponding directory structures. We can use some advanced Git repo analysis for this task.
Data creation script
Iterates over all commits and extracts basic information about the commit like sha, author and commit date (in log.txt) as well as the file's list of a specific version (in files.txt).
For each information set, it a new directory with the sha as unique identifier is created.
```bash
cd $1
sha_list=git rev-list master
for sha in $sha_list
Step1:  We can then import the data by looping through all the files and read in the corresponding files' content. We further extract the information items we need on the fly from the path as well as the content of log.txt. The result is stored into a Pandas DataFrame for further analysis.
Step2:  For each file, we have now a row the complete commit information available for both repositories.
Step3:  Basic statistics
Let's take a look at our read-in data.
Step4:  These are the number of entries for each repository.
Step5:  The amount of commits for each repository are.
Step6:  Data preparation
We need to adopt the data to the domain analyzed. We want to create a similarity measure between the directory structure of the lerna repository and the rush componente of the web-build-tools repository. The later is a little bit tricky, because there is a shift in the directory renaming.
<ASSISTANT_TASK:>
Python Code:
import glob
file_list = glob.glob(r'C:/dev/forensic/data/**/*.txt', recursive=True)
file_list = [x.replace("\\", "/") for x in file_list]
file_list[:5]
Explanation: Introduction
Idea
The claim was that the directory structure would be very similar to each other over a period of time. We want to identify this time span by using a time-based analysis on the commits and their corresponding directory structures. We can use some advanced Git repo analysis for this task.
Data creation script
Iterates over all commits and extracts basic information about the commit like sha, author and commit date (in log.txt) as well as the file's list of a specific version (in files.txt).
For each information set, it a new directory with the sha as unique identifier is created.
```bash
cd $1
sha_list=git rev-list master
for sha in $sha_list:
do
  data_dir="../data/$1/$sha"
  mkdir -p $data_dir
  git checkout $sha
  git log -n 1 $sha > $data_dir/log.txt
  git ls-files > $data_dir/files.txt
done
```
You can store this script e. g. into extract.sh and execute it for a repository with
bash
sh execute.sh &lt;path_git_repo&gt;
and you'll get a directory / files structure like this
.
├── data
│   ├── lerna
│   │   ├── 001ec5882630cedd895f2c95a56a755617bb036c
│   │   │   ├── files.txt
│   │   │   └── log.txt
│   │   ├── 00242afa1efa43a98dc84815ac8f554ffa58d472
│   │   │   ├── files.txt
│   │   │   └── log.txt
│   │   ├── 007f20b89ae33721bd08f8bcdd0768923bcc6bc5
│   │   │   ├── files.txt
│   │   │   └── log.txt
The content is as follows:
files.txt
.babelrc
.editorconfig
.eslintrc.yaml
.github/ISSUE_TEMPLATE.md
.github/PULL_REQUEST_TEMPLATE.md
.gitignore
.npmignore
.travis.yml
CHANGELOG.md
CODE_OF_CONDUCT.md
CONTRIBUTING.md
FAQ.md
LICENSE
README.md
appveyor.yml
bin/lerna.js
doc/hoist.md
doc/troubleshooting.md
lerna.json
package.json
src/ChildProcessUtilities.js
src/Command.js
src/ConventionalCommitUtilities.js
src/FileSystemUtilities.js
src/GitUtilities.js
src/NpmUtilities.js
...
log.txt
```
commit 001ec5882630cedd895f2c95a56a755617bb036c
Author: Daniel Stockman &#100;&#97;&#110;&#105;&#101;&#108;&#115;&#64;&#122;&#105;&#108;&#108;&#111;&#119;&#103;&#114;&#111;&#117;&#112;&#46;&#99;&#111;&#109;
Date:   Thu Aug 10 09:56:14 2017 -0700
chore: fs-extra 4.x
```
With this data, we have the base for analysing a probably similar directure structure layout over time.
abd83718682d7496426bb35f2f9ca20f10c2468d,2015-12-04 23:29:27 +1100
.gitignore
LICENSE
README.md
bin/lerna.js
lib/commands/bootstrap.js
lib/commands/index.js
lib/commands/publish.js
lib/init.js
lib/progress-bar.js
package.json
Load all files with the files listings
I've executed the script for lerna as well as the web-build-tools. First, we first get all the files.txt using glob.
End of explanation
import pandas as pd
dfs = []
for files_file in file_list:
    
    try:
        files_df = pd.read_csv(files_file, names=['sha', 'timestamp'])
        files_df['project'] = files_file.split("/")[-2]
        files_df['file'] = files_df.sha
        files_df['sha'] = files_df.sha[0]
        files_df['timestamp'] = pd.to_datetime(files_df.timestamp[0])
        files_df = files_df[1:]
        files_df
        dfs.append(files_df)
    except OSError as e:
        print((e,files_file))
    
file_log = pd.concat(dfs, ignore_index=True)
file_log.head()
file_log.file = pd.Categorical(file_log.file)
file_log.info()
dir_log = dir_log[
    (dir_log.project=='lerna') & (dir_log.file.str.endswith(".js")) |
    (dir_log.project=='web-build-tools') & (dir_log.file.str.endswith(".ts"))
]
dir_log.project.value_counts()
dir_log = dir_log[dir_log.file.str.contains("/")].copy()
dir_log['last_dir'] = dir_log.file.str.split("/").str[-2]
dir_log['last_dir_id'] = pd.factorize(dir_log.last_dir)[0]
dir_log.head()
dir_log['date'] = dir_log.timestamp.dt.date
dir_log.head()
grouped = dir_log.groupby(['project', pd.Grouper(level='date', freq="D"),'last_dir_id'])[['sha']].last()
grouped.head()
grouped['existent'] = 1
grouped.head()
test = grouped.pivot_table('existent', ['project', 'date'], 'last_dir_id').fillna(0)
test.head()
lerna = test.loc['lerna'][0]
lerna
%maplotlib inline 
test.plot()
timed_log = dir_log.set_index(['timestamp', 'project'])
timed_log.head()
timed_log.resample("W").first()
%matplotlib inline
timed.\
pivot_table('last_dir_id', timed.index, 'project')\
.fillna(method='ffill').dropna().plot()
Explanation: We can then import the data by looping through all the files and read in the corresponding files' content. We further extract the information items we need on the fly from the path as well as the content of log.txt. The result is stored into a Pandas DataFrame for further analysis.
End of explanation
file_log[file_log.project == "lerna"].iloc[0]
file_log[file_log.project == "web-build-tools"].iloc[0]
Explanation: For each file, we have now a row the complete commit information available for both repositories.
End of explanation
file_log.info()
Explanation: Basic statistics
Let's take a look at our read-in data.
End of explanation
file_log.project.value_counts()
Explanation: These are the number of entries for each repository.
End of explanation
file_log.groupby('project').sha.nunique()
Explanation: The amount of commits for each repository are.
End of explanation
file_log[file_log.project=="web-build-tools"].iloc[0]
file_log[file_log.project=="web-build-tools"].file.iloc[-10:]
lerna = file_log[file_log.project == "lerna"]
lerna.info()
rush = file_log[file_log.project == "web-build-tools"]
rush.info()
from scipy.spatial.distance import hamming
def calculate_hamming(row):
    lerna = row.file_list_lerna.split("\n")
    lerna = [x.rsplit(".", maxsplit=1)[0] for x in lerna]
    rush = row.file_list_rush.split("\n")
    rush = [x.rsplit(".", maxsplit=1)[0] for x in rush]
    count = 0
    for i in lerna:
        if i in rush:
            count = count + 1
    return count
    
comp["amount"] = comp.apply(calculate_hamming, axis=1)
comp.head()
%matplotlib inline
comp.amount.plot()
comp.resample("W").amount.mean().plot()
comp[comp.amount == comp.amount.max()]
Explanation: Data preparation
We need to adopt the data to the domain analyzed. We want to create a similarity measure between the directory structure of the lerna repository and the rush componente of the web-build-tools repository. The later is a little bit tricky, because there is a shift in the directory renaming.
End of explanation
<END_TASK>