Time Series forecast on Spark
<p>So I´m trying to do power consumption forecast with time series data with Apache Spark. Sample of the data is : </p>

<pre><code>03.01.15;22:30;236,25
03.01.15;22:15;240
04.01.15;16:00;243,775
</code></pre>

<p>and so on for two year. I have Observations every 15 minutes</p>

<p>What is the best way to predict the power consumption  ? </p>

<p>I try <code>LinearRegression</code>, <code>Decision trees</code> etc. I always get huge MSEs (788). 
I try pass months, days, hours, minutes as <code>OneHotEncoder</code>. I try to do a forecast only for weeks etc. </p>

<p>Means of the data <code>year</code>, <code>month</code>, <code>day</code>:</p>

<pre><code>[2014.3996710526321,5.726973684210525,15.713815789473673]
</code></pre>

<p>Variance of the data <code>year</code>, <code>month</code>, <code>day</code>:</p>

<pre><code>[0.2403293809070049,10.218579294199253,77.46326844706495]
</code></pre>

<p>Test Mean Squared Error</p>

<pre><code>788.2397552290726
</code></pre>

<p>If I pass the values direct like <code>LabeledPoint(236.25, 2015.0,1.0,3.0,22.0,30.0))</code> the MSE goes up to <code>1280.8</code>. If I pass to the model only one observation per day being this the max value it does not help to.</p>

<p>But If I use <code>KNIME</code> and try the example for time series but with my data they do not take the dates and time in consideration, instead they lagged the power consumptions for each observation.</p>

<p>I see that Cloudera has a library for <code>Time Series</code> but I do not understand why do I need it.</p>

<p>Can someone describe the process of doing a good forecast on time series data ?  At the end I want to input a date and time and get a prediction.</p>