Implementing gradient descent for multiple variables in Octave using "sum"
<p>I'm doing Andrew Ng's course on Machine Learning and I'm trying to wrap my head around the vectorised implementation of gradient descent for multiple variables which is an optional exercise in the course.</p>

<p>This is the algorithm in question (taken from <a href="http://www.holehouse.org/mlclass/04_Linear_Regression_with_multiple_variables.html" rel="nofollow noreferrer">here</a>):</p>

<p><a href="https://i.stack.imgur.com/S3jua.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/S3jua.png" alt="enter image description here"></a></p>

<p>I just cannot do this in octave using <code>sum</code> though, I'm not sure how to multiply the sum of the hypothesis of x(i) - y(i) by the all variables xj(i). I tried different iterations of the following code but to no avail (either the dimensions are not right or the answer is wrong):</p>

<pre><code>theta = theta - alpha/m * sum(X * theta - y) * X;
</code></pre>

<p>The correct answer, however, is entirely non-obvious (to a linear algebra beginner like me anyway, from <a href="https://stackoverflow.com/questions/10479353/gradient-descent-seems-to-fail">here</a>):</p>

<pre><code>theta = theta - (alpha/m *  (X * theta-y)' * X)';
</code></pre>

<p>Is there a rule of thumb for cases where <code>sum</code> is involved that governs transformations like the above? </p>

<p>And if so, is there the opposite version of the above (i.e. going from a <code>sum</code> based solution to a general multiplication one) as I was able to come up with a correct implementation using <code>sum</code> for gradient descent for a single variable (albeit not a very elegant one):</p>

<pre><code>temp0 = theta(1) - (alpha/m * sum(X * theta - y));
temp1 = theta(2) - (alpha/m * sum((X * theta - y)' * X(:, 2)));

theta(1) = temp0;
theta(2) = temp1;
</code></pre>

<p>Please note that this only concerns vectorised implementations and although there are several questions on SO as to how this is done, my question is primarily concerned with the implementation of the algorithm in Octave using <code>sum</code>.</p>