<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 This script is dedicated to querying all needed statistics for the project.
Step1:  Helpers
Step2:  Distribution of job posts among job titles
Step3:  Job posts distribution among standard job titles
Step4:  Statistics for Domains
Note
Step5:  Why no. of job titles in IT is reduced a lot after std?
Step6:  Statistics for functions
Note
<ASSISTANT_TASK:>
Python Code:
import my_util as my_util; from my_util import *
HOME_DIR = 'd:/larc_projects/job_analytics/'
DATA_DIR = HOME_DIR + 'data/clean/'
title_df = pd.read_csv(DATA_DIR + 'new_titles_2posts_up.csv')
Explanation: This script is dedicated to querying all needed statistics for the project.
End of explanation
def distTitle(agg_df, for_domain=False, for_func=False):
    fig = plt.figure()
    plt.hist(agg_df.n_title)
    mean_n_title = round(agg_df.n_title.mean(), 1)
    xl = '# job titles' + r'$(\mu = {})$'.format(mean_n_title)
    plt.xlabel(xl, fontsize=16); 
    if for_domain: plt.ylabel('# domains', fontsize=16)
    if for_func: plt.ylabel('# functions', fontsize=16)
    plt.grid(True)
    return fig
def aggBy(col, title_df):
    by_col = title_df.groupby(col)
    print('# {}: {}'.format(col, by_col.ngroups) )
    
    agg_df = by_col.agg({'title': 'nunique','non_std_title': 'nunique','n_post': sum})
    agg_df = agg_df.rename(columns={'title': 'n_title', 
                                    'std_title': 'n_std_title'}).reset_index()
    return agg_df
Explanation: Helpers
End of explanation
title_stats = pd.read_csv(DATA_DIR + 'stats_job_titles.csv')
titles = title_stats['title']
print('# titles: %d' %len(titles))
by_n_post = pd.read_csv(DATA_DIR + 'stats_job_post_dist.csv')
by_n_post.head()
Explanation: Distribution of job posts among job titles
End of explanation
by_n_post_after_std = title_stats.groupby('n_post').agg({'title': len})
by_n_post_after_std = by_n_post_after_std.rename(columns={'title': 'n_title_after_std'}).reset_index()
quantile(by_n_post_after_std.n_post)
fig = vizJobPostDist(by_n_post)
plt.savefig(RES_DIR + 'fig/dist_job_post_by_title.pdf')
plt.show(); plt.close()
print('# job titles with >= 2 posts: {}'.format(title_df.shape[0]) )
Explanation: Job posts distribution among standard job titles
End of explanation
by_domain_agg = aggBy('domain', title_df)
by_domain_agg.sort_values('n_title', ascending=False, inplace=True)
by_domain_agg.to_csv(DATA_DIR + 'stats_domains.csv', index=False)
by_domain_agg.describe().round(1).to_csv(DATA_DIR + 'tmp/domain_desc.csv')
by_domain_agg.describe().round(1)
plt.close('all')
fig = distTitle(by_domain_agg, for_domain=True)
fig.set_tight_layout(True)
plt.savefig(DATA_DIR + 'title_dist_by_domain.pdf')
plt.show(); plt.close()
Explanation: Statistics for Domains
Note: The domains are domains of job titles with >= 2 posts.
End of explanation
title_df.query('domain == "information technology"').sort_values('std_title')
Explanation: Why no. of job titles in IT is reduced a lot after std?
End of explanation
by_func_agg = aggBy('pri_func', title_df)
by_func_agg.sort_values('n_title', ascending=False, inplace=True)
by_func_agg.to_csv(DATA_DIR + 'stats_pri_funcs.csv', index=False)
by_func_agg.describe().round(1).to_csv(DATA_DIR + 'tmp/func_desc.csv')
by_func_agg.describe().round(1)
by_func_agg.head(10)
fig = distTitle(by_func_agg, for_func=True)
fig.set_tight_layout(True)
plt.savefig(DATA_DIR + 'title_dist_by_func.pdf')
plt.show(); plt.close()
sum(title_df.domain == 'information technology')
title_df.std_title[title_df.pri_func == 'technician'].nunique()
job_df = pd.read_csv(DATA_DIR + 'jobs.csv')
print job_df.shape
job_df.head(1)
full_job_df = pd.read_csv(DATA_DIR + 'job_posts.csv')
print full_job_df.shape
full_job_df.head(1)
full_job_df = pd.merge(full_job_df, job_df[['job_id', 'doc']])
print full_job_df.shape
print('# job ids including dups: %d' %len(full_job_df.job_id))
print('# unique job ids: %d' % full_job_df.job_id.nunique())
full_job_df.head(1)
full_job_df.to_csv(DATA_DIR + 'job_posts.csv', index=False)
Explanation: Statistics for functions
Note: Functions are limited to those of job titles with >= 2 posts.
End of explanation
<END_TASK>