What Is a Fréchet Derivative?

Let U and V be Banach spaces (complete normed vector spaces). The Fréchet derivative of a function f:U \to V at X\in U is a linear mapping L:U\to V such that

\notag       f(X+E) - f(X) - L(X,E) = o(\|E\|)

for all E\in U. The notation L(X,E) should be read as “the Fréchet derivative of f at X in the direction E”. The Fréchet derivative may not exist, but if it does exist then it is unique. When U = V = \mathbb{R}, the Fréchet derivative is just the usual derivative of a scalar function: L(x,e) = f'(x)e.

As a simple example, consider U = V = \mathbb{R}^{n\times n} and f(X) = X^2. From the expansion

f(X+E) - f(X) = XE + EX + E^2

we deduce that L(X,E) = XE + EX, the first order part of the expansion. If X commutes with E then L_X(E) = 2XE = 2EX.

More generally, it can be shown that if f has the power series expansion f(x) = \sum_{i=0}^\infty a_i x^i with radius of convergence r then for X,E\in\mathbb{R}^{n\times n} with \|X\| < r, the Fréchet derivative is

L(X,E) = \displaystyle\sum_{i=1}^\infty a_i             \displaystyle\sum_{j=1}^i X^{j-1} E X^{i-j}.

An explicit formula for the Fréchet derivative of the matrix exponential, f(A) = \mathrm{e}^A, is

L(A,E) = \displaystyle\int_0^1 \mathrm{e}^{A(1-s)} E \mathrm{e}^{As} \, ds.

Like the scalar derivative, the Fréchet derivative satisfies sum and product rules: if g and h are Fréchet differentiable at A then

\notag  \begin{alignedat}{2}    f &= \alpha g + \beta h  &&\;\Rightarrow\;   L_f(A,E) = \alpha L_g(A,E) + \beta L_h(A,E),\\   f &= gh  &&\;\Rightarrow\; L_f(A,E) = L_g(A,E) h(A) + g(A) L_h(A,E). \end{alignedat}

A key requirement of the definition of Fréchet derivative is that L(X,E) must satisfy the defining equation for all E. This is what makes the Fréchet derivative different from the Gâteaux derivative (or directional derivative), which is the mapping G:U \to V given by

\notag       G(X,E) = \lim_{t\to0} \displaystyle\frac{f(X+t E)-f(X)}{t}              = \frac{\mathrm{d}}{\mathrm{dt}}\Bigm|_{t=0} f(X + tE).

Here, the limit only needs to exist in the particular direction E. If the Fréchet derivative exists at X then it is equal to the Gâteaux derivative, but the converse is not true.

A natural definition of condition number of f is

\mathrm{cond}(f,X) = \lim_{\epsilon\to0}                \displaystyle\sup_{\|\Delta X\| \le \epsilon \|X\|}                \displaystyle\frac{\|f(X+\Delta X) - f(X)\|}{\epsilon\|f(X)\|},

and it can be shown that \mathrm{cond} is given in terms of the Fréchet derivative by

\mathrm{cond}(f,X) = \displaystyle\frac{\|L(X)\| \|X\|}{\|f(X)\|},


\|L(X)\| = \sup_{Z\ne0}\displaystyle\frac{\|L(X,Z)\|}{\|Z\|}.

For matrix functions, the Fréchet derivative has a number of interesting properties, one of which is that the eigenvalues of L(X) are the divided differences

\notag    f[\lambda_i,\lambda_j] = \begin{cases}    \dfrac{ f(\lambda_i)-f(\lambda_j) }{\lambda_i - \lambda_j},    & \lambda_i\ne\lambda_j, \\    f'(\lambda_i), & \lambda_i=\lambda_j,    \end{cases}

for 1\le i,j \le n, where the \lambda_i are the eigenvalues of X. We can check this formula in the case F(X) = X^2. Let (\lambda,u) be an eigenpair of X and (\mu,v) an eigenpair of X^T, so that Xu = \lambda u and X^Tv = \mu v, and let E = uv^T. Then

L(X,E) = XE + EX = Xuv^T + uv^TX = (\lambda + \mu) uv^T.

So uv^T is an eigenvector of L(X) with eigenvalue \lambda + \mu. But f[\lambda,\mu] = (\lambda^2-\mu^2)/(\lambda - \mu) = \lambda + \mu (whether or not \lambda and \mu are distinct).



Related Blog Posts


Leave a Reply


WordPress.com Logo


Google photo


Twitter picture


Facebook photo


Connecting to %s