<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 set up the data directory
Step1:  data download
get opinion and cluster files from CourtListener
opinions/cluster files are saved in data_dir/raw/court/
Step2:  get the master edgelist from CL
master edgelist is saved in data_dir/raw/
Step3:  download scdb data from SCDB
scdb data is saved in data_dir/scdb
Step4:  network data
make the case metadata and edgelist
add the raw case metadata data frame to the raw/ folder
remove cases missing scdb ids
remove detroit lumber case
get edgelist of cases within desired subnetwork
save case metadata and edgelist to the experiment_dir/
Step5:  make graph
creates the network with the desired case metadata and saves it as a .graphml file in experiment_dir/
Step6:  NLP data
make case text files
grabs the opinion text for each case in the network and saves them as a text file in experiment_dir/textfiles/
Step7:  make tf-idf matrix
creates the tf-idf matrix for the corpus of cases in the network and saves them to  subnet_dir + 'nlp/'
Step8:  Load network
Step9:  compute snapshots
Step10:  add textlength
count number of words in each opinion text.
Step11:  update snapshots
<ASSISTANT_TASK:>
Python Code:
setup_data_dir(data_dir)
make_subnetwork_directory(data_dir, network_name)
Explanation: set up the data directory
End of explanation
download_op_and_cl_files(data_dir, network_name)
Explanation: data download
get opinion and cluster files from CourtListener
opinions/cluster files are saved in data_dir/raw/court/
End of explanation
download_master_edgelist(data_dir)
Explanation: get the master edgelist from CL
master edgelist is saved in data_dir/raw/
End of explanation
download_scdb(data_dir)
Explanation: download scdb data from SCDB
scdb data is saved in data_dir/scdb
End of explanation
# create the raw case metadata data frame in the raw/ folder
make_subnetwork_raw_case_metadata(data_dir, network_name)
# create clean case metadata and edgelist from raw data
clean_metadata_and_edgelist(data_dir, network_name)
Explanation: network data
make the case metadata and edgelist
add the raw case metadata data frame to the raw/ folder
remove cases missing scdb ids
remove detroit lumber case
get edgelist of cases within desired subnetwork
save case metadata and edgelist to the experiment_dir/
End of explanation
make_graph(subnet_dir, network_name)
Explanation: make graph
creates the network with the desired case metadata and saves it as a .graphml file in experiment_dir/
End of explanation
%%time
make_network_textfiles(data_dir, network_name)
Explanation: NLP data
make case text files
grabs the opinion text for each case in the network and saves them as a text file in experiment_dir/textfiles/
End of explanation
%%time
make_tf_idf(text_dir, subnet_dir + 'nlp/')
Explanation: make tf-idf matrix
creates the tf-idf matrix for the corpus of cases in the network and saves them to  subnet_dir + 'nlp/'
End of explanation
# load the graph
G = ig.Graph.Read_GraphML(subnet_dir + network_name +'_network.graphml')
G.summary()
Explanation: Load network
End of explanation
vertex_metrics = ['indegree', 'outdegree', 'degree',
                  'd_pagerank','u_pagerank',
                  'authorities', 'hubs',
                  'd_eigen', 'u_eigen']
                  #'d_betweenness', 'u_betweenness',
                  #'d_closeness', 'u_closeness']
# add recent citations
vertex_metrics += ['recentcite_' + str(t) for t in np.arange(1, 10 + 1)]
vertex_metrics += ['recentcite_' + str(t) for t in [15, 20, 25, 30, 35, 40]]
active_years = range(1900, 2015 + 1)
%%time
make_snapshot_vertex_metrics(G, active_years, vertex_metrics, subnet_dir)
Explanation: compute snapshots
End of explanation
G.vs['num_words'] = [0] * len(G.vs)
for op_id in G.vs['name']:
    
    text = open(text_dir + op_id +'.txt', 'r').read()
    num_words = len(text.split())
    
    G.vs.find(name=op_id)['num_words'] = num_words
G.write_graphml(subnet_dir + network_name +'_network.graphml')
Explanation: add textlength
count number of words in each opinion text.
End of explanation
to_add = ['rev_pagerank', 'num_words']
to_add += ['citerank_' + str(t) for t in [1, 2, 5, 10, 20, 50]]
to_add = ['polyrank_' + str(t) for t in [1, 2, 5, 10, 20, 50]]
%%time
update_snapshot_vertex_metrics(G, active_years, to_add, subnet_dir)
Explanation: update snapshots
End of explanation
<END_TASK>