Implementing steepest descent algorithm, variable step size
<p>I am trying to implement steepest descent algorithm in programming languages (C/C++/fortran).</p>

<p>For example minimization of f(x1,x2) = x1^3 + x2^3 - 2*x1*x2</p>

<ol>
<li><p>Estimate starting design point x0, iteration counter k0, convergence parameter tolerence = 0.1. 
Say this staring point is (1,0)</p></li>
<li><p>Compute gradient of f(x1,x2) at the current point x(k) as grad(f). I will use numerical differentiation here.</p>

<p>d/dx1 (f) = lim (h->0) (f(x1+h,x2) - f(x1,x2) )/h</p>

<p>This is grad(f)=(3*x1^2 - 2*x2, 3*x2^2 - 2*x1)</p>

<p>grad(f) at (0,1) is  c0 = (3,-2) </p></li>
<li><p>since L2 norm of c0 > tolerence, we proceed for next step</p></li>
<li><p>direction d0 = -c0 = (-3,2)</p></li>
<li><p>Calculate step size a. Minimize f(a) = f(x0 + a<em>d0) = (1-3a,2a) = (1-3a)^3 + (2a)^3 - 2</em>(1-3a)*(2a). I am not keeping constant step size.</p></li>
<li><p>update: new[x1,x2] = old[x1,x2]x + a*d0. </p></li>
</ol>

<p>I do not understand how to do step 5. 
I have a 1D minimization program with bisection method, and it looks like:</p>

<pre><code>program main()
    ...
    ...
    define upper, lower interval
    call function value
    ...calculations
    ...
    ...


function value (input x1in) (output xout)
    ...function is x^4 - 2x^2 + x + 10 
    xout = (xin)^4 - 2*(xin)^2 + (xin) + 10
</code></pre>

<p>In this case, looking at step 5, I cannot pass symbolic a.
Any ideas how to implement the algorithm in programming language, especially step 5? Please suggest if there is altogether different way to program this. I have seen many programs with constant step size, but I want to compute it at every step. This algorithm can be easy to implement in MATLAB ot python sympy using symbolics, but I do not want to use symbolics. 
Any suggestions appreciated. Thanks. </p>