The gradient and the directional derivative

The idea

The gradient collects the partial derivatives into one arrow. It points in the direction in which f climbs fastest, its length is that fastest rate, and it is perpendicular to the level curve through the point (the curve where f keeps the same value). Walking against the gradient is the steepest way down.

The math, with a comment on each formula

\nabla f(x, y) = \begin{pmatrix} f_x \\ f_y \end{pmatrix} In words: the gradient is the column of partial derivatives.

D_u f = \nabla f \cdot u = \|\nabla f\| \cos\theta \qquad (\|u\| = 1) In words: the slope in direction u is the gradient dotted with u; it is largest, \|\nabla f\|, when u points along the gradient (\theta = 0), and zero when u runs along the level curve (\theta = 90^\circ).

(x, y)_{\text{new}} = (x, y) - t\,\nabla f(x, y), \qquad t > 0 In words: a gradient step moves a little way downhill; t is the step length.

Worked example

Take f(x, y) = x^2 + 3y^2 at the point (2, 1), where f = 7. The gradient is \nabla f = \begin{pmatrix}2 x\\6 y\end{pmatrix}, which there equals \begin{pmatrix}4\\6\end{pmatrix}, with length \|\nabla f\| = 2 \sqrt{13} (about 7.211). In the direction u = (3, 4)/5 the slope is D_u f = \frac{36}{5}, smaller than the maximum, as it must be. Along (3, -2)/\sqrt{13}, perpendicular to the gradient, the slope is 0: that direction runs along the level curve x^2 + 3y^2 = 7.

One gradient step with t = 1/10 lands at (\frac{8}{5}, \frac{2}{5}), where f = \frac{76}{25}, down from 7.

Why it matters for optimization

At an unconstrained optimum the gradient is zero: no direction is downhill. With constraints, the gradient of the objective is balanced by the gradients of the active constraints (Lagrange and KKT conditions), which is how ProvenDecisions certifies smooth answers. First-order methods are repeated gradient steps.