Numerical Analysis/Differentiation/Examples

When deriving a finite difference approximation of the $$j$$th derivative of a function $$f : \mathbb{R} \rightarrow \mathbb{R}$$, we wish to find $$a_1, a_2, ..., a_n \in \mathbb{R}$$ and $$b_1, b_2, ..., b_n \in \mathbb{R}$$ such that
 * $$f^{(j)}(x_0) = h^{-j}\sum_{i=1}^{n}a_i f(x_0 + b_i h) + O(h^k) \text{ as } h \to 0$$

or, equivalently,
 * $$h^{-j}\sum_{i=1}^{n}a_i f(x_0 + b_i h) = f^{(j)}(x_0) + O(h^k) \text{ as } h \to 0$$

where $$O(h^k)$$ is the error, the difference between the correct answer and the approximation, expressed using Big-O notation. Because $$h$$ may be presumed to be small, a larger value for $$k$$ is better than a smaller value.

A general method for finding the coefficients is to generate the Taylor expansion of $$h^{-j}\sum_{i=1}^{n}a_i f(x_0 + b_i h)$$ and choose $$a_1, a_2, ..., a_n$$ and $$b_1, b_2, ..., b_n$$ such that $$f^{(j)}(x_0)$$ and the remainder term are the only non-zero terms. If there are no such coefficients, a smaller value for $$k$$ must be chosen.

For a function of $$m$$ variables $$g:\mathbb{R}^m \rightarrow \mathbb{R}$$, the procedure is similar, except $$x_0, b_1, b_2, ..., b_n$$ are replaced by points in $$\mathbb{R}^m$$ and the multivariate extension of Taylor's theorem is used.

Single-Variable
In all single-variable examples, $$x_0 \in \mathbb{R}$$ and $$f : \mathbb{R} \rightarrow \mathbb{R}$$ are unknown, and $$h \in \mathbb{R}$$ is small. Additionally, let $$f$$ be 5 times continuously differentiable on $$\mathbb{R}$$.

First Derivative
Find $$a, b, c \in \mathbb{R}$$ such that $$\frac{a f(x_0 + h) + b f(x_0 + c h)}{h}$$ best approximates $$f'(x_0)$$.

Solution: First, we find the Taylor series expansion of $$\frac{a f(x_0 + h) + b f(x_0 + c h)}{h}$$ with remainder term to be
 * $$\frac{a f(x_0 + h) + b f(x_0 + c h)}{h} = \frac{a + b}{h}f(x_0) + (a + b c) f'(x_0) + \frac{h}{2} (a + b c^2) f''(x_0) + O(h^2) \text{ as } h \to 0\,.$$

If we can find a solution to the system
 * $$\begin{align}0 &=& a + &b&\\

1 &=& a + &b c&\\ 0 &=& a + &b c^2&\end{align}$$ then we can substitute that solution into the Taylor expansion to obtain
 * $$\frac{a f(x_0 + h) + b f(x_0 + c h)}{h} = f'(x_0) + O(h^2) \text{ as } h \to 0\,.$$

The system of equations has exactly one solution: $$a = \frac{1}{2}$$, $$b = \frac{1}{-2}$$, $$c = -1$$, so
 * $$\frac{f(x_0 + h) - f(x_0 - h)}{2 h} = f'(x_0) + O(h^2) \text{ as } h \to 0\,.$$

Let $$f : \mathbb{R} \rightarrow \mathbb{R}$$ be 42 times continuously differentiable on $$\mathbb{R}$$. Find the largest $$n \in \mathbb{N}$$ such that
 * $$\frac{df}{dx}(x_0) = \frac{-f(x_0 + 2 h) + 8 f(x_0 + h, y_0) - 8 f(x_0 - h) + f(x_0 - 2 h)}{12 h} + O(h^n) \text{ as } h \to 0$$

In other words, find the order of the error of the method.

Solution: The Taylor expansion of the method is
 * $$\begin{align}\frac{-f(x_0 + 2 h) + 8 f(x_0 + h, y_0) - 8 f(x_0 - h) + f(x_0 - 2 h)}{12 h}

&= \frac{-1 + 8 - 8 + 1}{12 h}f(x_0) + \frac{-2 + 8 + 8 - 2}{12} f'(x_0) \\ &+ h\frac{(-4 + 8 - 8 + 4)}{24} f(x_0) + h^2\frac{(-8 + 8 + 8 - 8}{72} f'(x_0) \\ &+ h^3\frac{(-16 + 8 - 8 + 16}{288} f^{(4)}(x_0) + h^4\frac{(-32 + 8 + 8 - 32)}{1440} f^{(4)}(x_0) + O(h^5) \text{ as } h \to 0\,.\end{align}$$ Simplifying this algebraically gives
 * $$\frac{-f(x_0 + 2 h) + 8 f(x_0 + h, y_0) - 8 f(x_0 - h) + f(x_0 - 2 h)}{12 h} = f'(x_0) + \frac{h^4}{-30} f^{(5)}(x_0) + O(h^5) \text{ as } h \to 0\,.$$

The multiple of $$h^4$$ cannot be removed, so $$n = 4$$ and by properties of Big-O notation,
 * $$\frac{-f(x_0 + 2 h) + 8 f(x_0 + h, y_0) - 8 f(x_0 - h) + f(x_0 - 2 h)}{12 h} = f'(x_0) + O(h^4) \text{ as } h \to 0\,.$$

Second Derivative
Find $$a, b, c \in \mathbb{R}$$ such that $$\frac{a f(x_0 - h) + b f(x_0) + c f(x_0 + h)}{h^2}$$ best approximates $$f''(x_0)$$.

Solution: First, we find the Taylor series expansion of $$\frac{a f(x_0 - h) + b f(x_0) + c f(x_0 + h)}{h^2}$$, with remainder term to be
 * $$\frac{a f(x_0 - h) + b f(x_0) + c f(x_0 + h)}{h^2} = \frac{a + b + c}{h^2}f(x_0) + \frac{c - a}{h} f'(x_0) + \frac{a + c}{2} f(x_0) + \frac{h}{6} (c - a) f'(x_0) + O(h^2) \text{ as } h \to 0\,.$$

If we can find a solution to the system
 * $$\begin{align}0 &=& &a& + &b& + &c&\\

0 &=& &-a& + &0& + &c&\\ 2 &=& &a& + &0& + &c&\\ 0 &=& &-a& + &0& + &c&\end{align}$$ then we can substitute that solution into the Taylor expansion and obtain
 * $$\frac{a f(x_0 - h) + b f(x_0) + c f(x_0 + h)}{h^2} = f''(x_0) + O(h^2) \text{ as } h \to 0\,.$$

The system of equations has exactly one solution: $$a = 1$$, $$b = -2$$, $$c = 1$$ so
 * $$\frac{f(x_0 - h) - 2 f(x_0) + f(x_0 + h)}{h^2} = f''(x_0) + O(h^2) \text{ as } h \to 0\,.$$

Multivariate
In all two-variable examples, $$x_0, y_0 \in \mathbb{R}$$ and $$f : \mathbb{R}^2 \rightarrow \mathbb{R}$$ are unknown, and $$h \in \mathbb{R}$$ is small.

Non-Mixed Derivatives
Because of the nature of partial derivatives, some of them may be calculated using single-variable methods. This is done by holding constant all but one variable to form a new function of one variable. For example if $$g_y(y) = f(x_0, y)$$, then $$\frac{df}{dy}(x_0, y_0) = \frac{dg}{dy}(y_0)$$.

Find an approximation of $$\frac{d}{dy}f(x_0, y_0)$$ Solution: Because we are differentiating with respect to only one variable, we can hold x constant and use the result of one of the single-variable examples:
 * $$\frac{f(x_0, y_0 + h) - f(x_0, y_0 - h)}{2 h} = \frac{df}{dy}(x_0, y_0) + O(h^2) \text{ as } h \to 0$$

Mixed Derivatives
Mixed derivatives may require the multivariate extension of Taylor's theorem.

Let $$f : \mathbb{R}^2 \rightarrow \mathbb{R}$$ be 42 times continuously differentiable on $$\mathbb{R}^2$$ and let $$g : \mathbb{R}^3 \rightarrow \mathbb{R}$$ be defined as
 * $$g(x_0, y_0, h) = \frac{f(x_0 + h, y_0 + h) + f(x_0 - h, y_0 - h) - f(x_0 - h, y_0 + h) - f(x_0 + h, y_0 - h)}{4 h^2}\,.$$

Find the largest $$n \in \mathbb{N}$$ such that
 * $$\frac{d^2 f}{dx dy}(x_0, y_0) = g(x_0, y_0, h) + O(h^n) \text{ as } h \to 0\,.$$

In other words, find the order of the error of the approximation.

Solution: The first few terms of the multivariate Taylor expansion of $$f$$ around $$(x_0,y_0)$$ are
 * $$\begin{align}f(x_0 + x, y_0 + y) &= f(x_0, y_0)\\ &+

x \frac{df}{dx}(x_0, y_0) + y \frac{df}{dy}(x_0, y_0)\\ &+ \frac{x^2}{2} \frac{d^2f}{dx^2}(x_0, y_0) + x y \frac{d}{dy}\frac{df}{dx}(x_0, y_0) + \frac{y^2}{2} \frac{d^2f}{dy^2}(x_0, y_0)\\ &+ \frac{x^3}{6} \frac{d^3f}{dx^3}(x_0, y_0) + \frac{x^2 y}{2} \frac{d}{dy}\frac{d^2f}{dx^2}(x_0, y_0) + \frac{x y^2}{2} \frac{d^2}{dy^2}\frac{df}{dx}(x_0, y_0) + \frac{y^3}{6} \frac{d^3f}{dy^3}(x_0, y_0)\\ &+ \frac{x^4}{24} \frac{d^4f}{dx^4}(x_0, y_0) + \frac{x^3 y}{6} \frac{d}{dy}\frac{d^3 f}{dx^3}(x_0, y_0) + \frac{x^2 y^2}{4} \frac{d^2}{dy^2}\frac{d^2 f}{dx^2}(x_0, y_0) + \frac{x y^3}{6} \frac{d^3}{dy^3}\frac{d f}{dx}(x_0, y_0) + \frac{y^4}{24} \frac{d^4f}{dy^4}(x_0, y_0)\\ &+ O(x^5) + O(y^5) \text{ as } x, y \to 0\,.\end{align}$$

We substitute the expansion for $$f$$ into the approximation $$g$$ to obtain
 * $$\begin{align}g(x_0, y_0, h) &= \frac{1 + 1 - 1 - 1}{4 h^2}f(x_0, y_0) \\

& + \frac{1 - 1 + 1 - 1}{4 h}\frac{df}{dx}(x_0, y_0) + \frac{1 - 1 - 1 + 1}{4 h} \frac{df}{dy}(x_0, y_0) \\ & + \frac{1 + 1 - 1 - 1}{8} \frac{d^2f}{dx^2}(x_0, y_0) + \frac{1 + 1 + 1 + 1}{4} \frac{d}{dy}\frac{df}{dx}(x_0, y_0) + \frac{1 + 1 - 1 - 1}{8} \frac{d^2f}{dy^2}(x_0, y_0) \\ & + h \left(\frac{1 - 1 + 1 - 1}{24} \frac{d^3f}{dx^3}(x_0, y_0) + \frac{1 - 1 - 1 + 1}{8} \frac{d}{dy}\frac{d^2f}{dx^2}(x_0, y_0) + \frac{1 - 1 + 1 - 1}{8} \frac{d^2}{dy^2}\frac{df}{dx}(x_0, y_0) + \frac{1 - 1 - 1 + 1}{24} \frac{d^3f}{dy^3}(x_0, y_0) \right)\\ & + h^2 \left(\frac{1 + 1 - 1 - 1}{96} \frac{d^4f}{dx^4}(x_0, y_0) + \frac{1 + 1 + 1 + 1}{24} \frac{d}{dy}\frac{d^3 f}{dx^3}(x_0, y_0) + \frac{1 + 1 - 1 - 1}{16} \frac{d^2}{dy^2}\frac{d^2 f}{dx^2}(x_0, y_0) + \frac{1 + 1 + 1 + 1}{24} \frac{d^3}{dy^3}\frac{d f}{dx}(x_0, y_0) + \frac{1 + 1 - 1 - 1}{96} \frac{d^4f}{dy^4}(x_0, y_0) \right) \\ & + O(h^3) \text{ as } h \to 0\,.\end{align}$$

Because of the careful choices of coefficients, we can simplify this to
 * $$g(x_0, y_0, h) =

\frac{d}{dy}\frac{df}{dx}(x_0, y_0) + \frac{h^2}{6}\left(\frac{d}{dy}\frac{d^3 f}{dx^3}(x_0, y_0) + \frac{d^3}{dy^3}\frac{d f}{dx}(x_0, y_0)\right) + O(h^3) \text{ as } h \to 0\,.$$

We note that Big-O notation permits us to write the last 3 terms as $$O(h^2) \text{ as } h \to 0$$. Thus,
 * $$g(x_0, y_0, h) =

\frac{d}{dy}\frac{df}{dx}(x_0, y_0) + O(h^2) \text{ as } h \to 0\,.$$

Because the multiples of $$h^2$$ are unaffected by adding more terms to the Taylor expansion, $$n = 2$$ is the greatest natural number satisfying the conditions given in the problem.

Example Code
Implementing these methods is reasonably simple in programming languages that support higher-order functions. For example, the method from the first example may be implemented in C++ using function pointers, as follows: