<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Outlier detection
Detect outliers using linear regression model and statsmodels. Based on Stackoverflow question.
Step1:  Test data
Here were just making some fake data. First of all, make a list of x values from 0 to 29. Next, use the x values to generate some y data that is salted with some randomness. Finally, change the 7th value to a value that is clearly an outlier from the rest.
Step2:  Regression
Here we're just doing an ordinary least squares method to fit the data. The "data ~ x" is just saying that 'data' (which is the y values) are directly related to 'x' values. This formalism apparently implies that data = m*x + b.
Step3:  Test for Outliers
Here we're using our regression results to do a test for outliers. In this case, I guess the default is a Bonferroni outlier test. We're only printing off test results where the third column is less than 0.05.
Find outliers
Step4:  Figure
Step5:  Create a function and test it
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import statsmodels.api as sm # For some reason this import is necessary...
import statsmodels.formula.api as smapi
import statsmodels.graphics as smgraph
import matplotlib.pyplot as plt
%matplotlib inline
Explanation: Outlier detection
Detect outliers using linear regression model and statsmodels. Based on Stackoverflow question.
End of explanation
x = np.arange(30, dtype=float)
# Make some y data with random noise
y = 10 * x + 5.0*np.random.randn(30)
# Add outlier #
y[10] = 180.
y[20] = 130
plt.plot(x, y, 'o')
Explanation: Test data
Here were just making some fake data. First of all, make a list of x values from 0 to 29. Next, use the x values to generate some y data that is salted with some randomness. Finally, change the 7th value to a value that is clearly an outlier from the rest.
End of explanation
# Make fit #
regression = smapi.ols("data ~ x", data=dict(data=y, x=x)).fit()
regression.summary()
Explanation: Regression
Here we're just doing an ordinary least squares method to fit the data. The "data ~ x" is just saying that 'data' (which is the y values) are directly related to 'x' values. This formalism apparently implies that data = m*x + b.
End of explanation
test = regression.outlier_test()
test
print('Bad data points (bonf(p) < 0.05):')
test[test['bonf(p)'] < 0.05]
outliers = test[test['bonf(p)'] < 0.05].index.values
outliers
Explanation: Test for Outliers
Here we're using our regression results to do a test for outliers. In this case, I guess the default is a Bonferroni outlier test. We're only printing off test results where the third column is less than 0.05.
Find outliers
End of explanation
figure = smgraph.regressionplots.plot_fit(regression, 1)
line = smgraph.regressionplots.abline_plot(model_results=regression, ax=figure.axes[0])
plt.plot(outliers, y[outliers], 'xm', label='outliers', ms=14)
plt.legend(loc=0);
Explanation: Figure
End of explanation
import statsmodels.formula.api as smapi
def get_outliers(features, target):
    regression = smapi.ols("target ~ features", data=locals()).fit()
    test = regression.outlier_test()
    outliers = test[test['bonf(p)'] < 0.05]
    return list(outliers.index.values)
def test_outliers():
    x = np.arange(30, dtype=float)
    # Make some y data with random noise
    y = 10 * x + 5.0*np.random.randn(30)
    # Add outlier
    y[10] = 180.
    y[20] = 130  
    outliers = [10, 20]
    prediction = get_outliers(features=x, target=y)
    assert outliers == prediction
    
test_outliers()
Explanation: Create a function and test it
End of explanation
<END_TASK>