 This discussion is restricted to the case of the square loss, though it can be extended to any convex loss.It can be shown by an easy induction that if formula_79 is the data matrix and formula_80 is the output after formula_54 steps of the SGD algorithm, then,
where formula_83 and the sequence formula_84 satisfies the recursion