<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Create entry points to spark
Step1:  Linear regression without cross-valiation
Import data
Step2:  Transform data structure
Step3:  Build linear regression model
Step4:  Fit the model
Step5:  Prediction
Step6:  Module evaluation
Step7:  Compare results with R
The comparison below shows that the linear regression analyses from pyspark and R obtained very close results.
```{r}
intercept and coefficients from R
advertise = read.csv('data/Advertising.csv', header = TRUE)
lr_ad = lm(Sales~., data = advertise)
lr_ad$coefficients
(Intercept)           TV        Radio    Newspaper 
 2.938889369  0.045764645  0.188530017 -0.001037493
intercept and coefficents from pyspark
lr_model.intercept
2.9388893694594134
lr_model.coefficients
DenseVector([0.0458, 0.1885, -0.001])
R squared from R
summary(lr_ad)$r.squared
0.8972106
R squared from pyspark
evaluator.evaluate(ad_pred, {evaluator.metricName
Step8:  Build cross-validation model
Step9:  Fit cross-validation model
Step10:  Prediction
Step11:  Evaluation
Step12:  Intercept and coefficients
Step13:  Get parameter values from the best model
Parameters can be extracted by calling the java property.
<ASSISTANT_TASK:>
Python Code:
from pyspark import SparkContext
sc = SparkContext(master = 'local')
from pyspark.sql import SparkSession
spark = SparkSession.builder \
          .appName("Python Spark SQL basic example") \
          .config("spark.some.config.option", "some-value") \
          .getOrCreate()
Explanation: Create entry points to spark
End of explanation
ad = spark.read.csv('data/Advertising.csv', header=True, inferSchema=True)
ad.show(5)
Explanation: Linear regression without cross-valiation
Import data
End of explanation
from pyspark.ml.linalg import Vectors
ad_df = ad.rdd.map(lambda x: [Vectors.dense(x[0:3]), x[-1]]).toDF(['features', 'label'])
ad_df.show(5)
Explanation: Transform data structure
End of explanation
from pyspark.ml.regression import LinearRegression
lr = LinearRegression(featuresCol = 'features', labelCol = 'label')
Explanation: Build linear regression model
End of explanation
lr_model = lr.fit(ad_df)
Explanation: Fit the model
End of explanation
pred = lr_model.transform(ad_df)
pred.show(5)
Explanation: Prediction
End of explanation
from pyspark.ml.evaluation import RegressionEvaluator 
evaluator = RegressionEvaluator(predictionCol='prediction', labelCol='label')
evaluator.setMetricName('r2').evaluate(pred)
Explanation: Module evaluation
End of explanation
training, test = ad_df.randomSplit([0.8, 0.2], seed=123)
Explanation: Compare results with R
The comparison below shows that the linear regression analyses from pyspark and R obtained very close results.
```{r}
intercept and coefficients from R
advertise = read.csv('data/Advertising.csv', header = TRUE)
lr_ad = lm(Sales~., data = advertise)
lr_ad$coefficients
(Intercept)           TV        Radio    Newspaper 
 2.938889369  0.045764645  0.188530017 -0.001037493
intercept and coefficents from pyspark
lr_model.intercept
2.9388893694594134
lr_model.coefficients
DenseVector([0.0458, 0.1885, -0.001])
R squared from R
summary(lr_ad)$r.squared
0.8972106
R squared from pyspark
evaluator.evaluate(ad_pred, {evaluator.metricName: "r2"})
0.897210638178952
```
Linear regression with cross-validation
Training and test datasets
End of explanation
##=====build cross valiation model======
# estimator
lr = LinearRegression(featuresCol = 'features', labelCol = 'label')
# parameter grid
from pyspark.ml.tuning import ParamGridBuilder
param_grid = ParamGridBuilder().\
    addGrid(lr.regParam, [0, 0.5, 1]).\
    addGrid(lr.elasticNetParam, [0, 0.5, 1]).\
    build()
    
# evaluator
evaluator = RegressionEvaluator(predictionCol='prediction', labelCol='label', metricName='r2')
# cross-validation model
from pyspark.ml.tuning import CrossValidator
cv = CrossValidator(estimator=lr, estimatorParamMaps=param_grid, evaluator=evaluator, numFolds=4)
Explanation: Build cross-validation model
End of explanation
cv_model = cv.fit(training)
Explanation: Fit cross-validation model
End of explanation
pred_training_cv = cv_model.transform(training)
pred_test_cv = cv_model.transform(test)
Explanation: Prediction
End of explanation
# performance on training data
evaluator.setMetricName('r2').evaluate(pred_training_cv)
# performance on test data
evaluator.setMetricName('r2').evaluate(pred_test_cv)
Explanation: Evaluation
End of explanation
print('Intercept: ', cv_model.bestModel.intercept, "\n",
     'coefficients: ', cv_model.bestModel.coefficients)
Explanation: Intercept and coefficients
End of explanation
print('best regParam: ' + str(cv_model.bestModel._java_obj.getRegParam()) + "\n" +
     'best ElasticNetParam:' + str(cv_model.bestModel._java_obj.getElasticNetParam()))
Explanation: Get parameter values from the best model
Parameters can be extracted by calling the java property.
End of explanation
<END_TASK>