<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Predicting publications
In this section, we will try to predict whether an user will be an author. If they are an author, we will try to predict the number of stories they would write based off observable characteristics, such as the number of years have they been on the site, how much they read, and whether or not they are in a community.
Step1:  Linear Modelling
For this problem, we will use regression analysis. That is, we have some information at hand, and we want to use that information to predict something. This works if we there's some underlying relationship between our information and the thing we want to predict.
More specifically, we will use the following models
Step2:  Logit model
The first approach that we will try is the logit model, or logistic regression, as explained above.
$Pr(Author_i) = \dfrac{1}{1-e^{-x_i^t \beta}}$
Here, the probability that user $i$ is an author is a function of a vector of characteristics $x$. These characteristics will be things that we can observe. In particular, we chose the following
Step3:  Results indicate there is some correlation between two of the independent variables
Step4:  The data is clustered around the zeros. Let's try a log transformation.
Step5:  Regression Model
Step6:  The log transformations helped increase the fit from and R-squared of ~0.05 to ~0.20.
From these results, we can see that
Step7:  Without 'fs', we lost some information but not much
<ASSISTANT_TASK:>
Python Code:
# opens raw data
with open ('../data/clean_data/df_profile', 'rb') as fp:
    df = pickle.load(fp)
# creates copy with non-missing observations
df_active = df.loc[df.status != 'inactive', ].copy()
Explanation: Predicting publications
In this section, we will try to predict whether an user will be an author. If they are an author, we will try to predict the number of stories they would write based off observable characteristics, such as the number of years have they been on the site, how much they read, and whether or not they are in a community.
End of explanation
# examines status of users
status = df_active['status'].value_counts()
# plots chart
status.plot.pie(autopct='%.f', figsize=(5,5))
plt.ylabel('')
plt.show()
Explanation: Linear Modelling
For this problem, we will use regression analysis. That is, we have some information at hand, and we want to use that information to predict something. This works if we there's some underlying relationship between our information and the thing we want to predict.
More specifically, we will use the following models:
Logistic regression
Logistic regression lets us predict the probability that variable is (or is not) something. The model is as follows:
$Pr(Y_i=1) = \dfrac{1}{1+e^{-(\beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + ... + \beta_k X_{ik} + \epsilon_i)}}$
in which $Y_i$ is $i$th observation of the dependent variable (variable of interest), and $X_{ij}$ is the $i$th observation of the $j$th independent variable.
For our model to work, we would need a few assumptions. These are:
Linearity between odds ratio and independent variables / correct functional form
Strict exogeneity $E[\epsilon|X] = 0$
No multicollinearity
Linear regression
Linear regression lets us predict the value of something. The model is as follows:
$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + ... + \beta_k X_{ik} + \epsilon_i$
in which $Y_i$ is $i$th observation of the dependent variable (variable of interest), and $X_{ij}$ is the $i$th observation of the $j$th independent variable.
For this model to work, we would need a few assumptions. These are:
Linearity between dependent and independent variables / correct functional form
Strict exogeneity $E[\epsilon|X] = 0$
No multicollinearity
Normally distributed errors
Spherical errors (homoscedasticity)
Probability of being an author
Blind guess
Before introducing any type of model, we can already see from our sample data that out of all active users, only ~18% of them are authors. As such, if we simply guess "not author" for all users, we will be correct ~82% of the time. Our goal is to come up with a classification method that has greater accuracy.
End of explanation
# displays correlation matrix
df_active.corr()
# creates design_matrix 
X = df_active
X['intercept'] = 1
# displays variance inflation factor
vif_results = pd.DataFrame()
vif_results['VIF Factor'] = [vif(X.values, i) for i in range(X.shape[1])]
vif_results['features'] = X.columns
vif_results
Explanation: Logit model
The first approach that we will try is the logit model, or logistic regression, as explained above.
$Pr(Author_i) = \dfrac{1}{1-e^{-x_i^t \beta}}$
Here, the probability that user $i$ is an author is a function of a vector of characteristics $x$. These characteristics will be things that we can observe. In particular, we chose the following:
* the number of years since they first joined the site
* the number of favorite authors+stories they have (logarithmically scaled)
* whether or not they have written a profile
* whether or not they are in a community
One more characteristic that we would also like to later include, but have not retrieved yet the data for:
* whether or not they have signed a review
The reasoning behind these variables is simple -- we assume that the longer and more involved an user is on the site, the more likely they are to publish.
From there, we randomly split up our data into two subsets: one to used as training (to build our model), and one to be used as testing (to test the accuracy of the model we built).
Results
The model produced by the training data has statistically significant positive $\beta$ coefficients for all our explanatory variables. This is the relationship we expected. However, when we use our model to predict the testing set, we achieve only ~86% accuracy. This is an improvement from blind guessing, but only slightly.
Estimating number of stories written
Multicollinearity
End of explanation
sns.pairplot(data=df_active, y_vars=['st'], x_vars=['fa', 'fs', 'age'])
plt.show()
Explanation: Results indicate there is some correlation between two of the independent variables: 'fa' and 'fs', implying one of them may not be necessary in the model.
Nonlinearity
We know from earlier distributions that some of the variables are heavily right-skewed. We created some scatter plots to confirm that the assumption of linearity holds.
End of explanation
# takes log transformation
df_active['st'] = np.log(df_active['st']+1)
df_active['fa'] = np.log(df_active['fa']+1)
df_active['fs'] = np.log(df_active['fs']+1)
sns.pairplot(data=df_active, y_vars=['st'], x_vars=['fa', 'fs', 'age'])
plt.show()
Explanation: The data is clustered around the zeros. Let's try a log transformation.
End of explanation
# runs OLS regression
formula = 'st ~ fa + fs + cc + age'
reg = smf.ols(data=df_active, formula=formula).fit()
print(reg.summary())
Explanation: Regression Model
End of explanation
# runs OLS regression
formula = 'st ~ fa + cc + age'
reg = smf.ols(data=df_active, formula=formula).fit()
print(reg.summary())
Explanation: The log transformations helped increase the fit from and R-squared of ~0.05 to ~0.20.
From these results, we can see that:
A 1% change in number of authors favorited is associated with a ~15% change in the number of stories written.
A 1% change in number of stories favorited is associated with a ~4% change in the number of stories written.
Being in a community is associated with a ~0.7 increase in the number of stories written.
One more year on the site is associated with a ~3% change in the number of stories written.
We noted earlier that 'fa' and 'fs' had a correlation of ~0.7. As such, we reran the regression without 'fa' first, then again without 'fs'. The model without 'fs' yielded a better fit (R-squared), as well as AIC and BIC.
End of explanation
def graph(formula, x_range):  
    y = np.array(x_range)
    x = formula(y)
    plt.plot(y,x)  
graph(lambda x : (np.exp(reg.params[0]+reg.params[1]*(np.log(x-1)))-1), 
      range(2,100,1))
graph(lambda x : (np.exp(reg.params[0]+reg.params[1]*(np.log(x-1))+reg.params[2])-1), 
      range(2,100,1))
plt.show() 
ages = [0, 1, 5, 10, 15]
for age in ages:
    graph(lambda x : (np.exp(reg.params[0]+reg.params[1]*(np.log(x-1))+reg.params[3]*age)-1), 
          range(2,100,1))
plt.show() 
Explanation: Without 'fs', we lost some information but not much:
A 1% change in number of authors favorited is associated with a ~20% change in the number of stories written.
Being in a community is associated with a ~0.7 increase in the number of stories written.
One more year on the site is associated with a ~3% change in the number of stories written.
All these results seem to confirm a basic intuition that the more active an user reads (as measured by favoriting authors and stories), the likely it is that user will write more stories. Being longer on the site and being part of a community is also correlated to publications.
To get a sense of the actual magnitude of these effects, let's attempt some plots:
End of explanation
<END_TASK>