The Hessian and classifying critical points

The idea

In two variables a critical point (both partial derivatives zero) can be a valley, a peak, or a saddle: up in one direction and down in another, like a mountain pass. The Hessian, the matrix of second partial derivatives, tells which, just as f'' does in one variable.

The math, with a comment on each formula

H = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{xy} & f_{yy} \end{pmatrix}, \qquad \det H = f_{xx} f_{yy} - f_{xy}^2 In words: the Hessian holds the curvatures; its determinant multiplies the curvatures in the two main directions (its eigenvalues).

\begin{align*} \det H > 0,\ f_{xx} > 0 &\Rightarrow \text{local min}\\ \det H > 0,\ f_{xx} < 0 &\Rightarrow \text{local max}\\ \det H < 0 &\Rightarrow \text{saddle}\\ \det H = 0 &\Rightarrow \text{no conclusion} \end{align*} In words: both eigenvalues positive is a valley, both negative a peak, opposite signs a saddle.

Worked example

Take f(x, y) = x^3 + y^3 - 3xy. The partial derivatives f_x = 3 x^{2} - 3 y and f_y = - 3 x + 3 y^{2} are both zero at two real points, (0, 0) and (1, 1). The Hessian is H = \begin{pmatrix}6 x & -3\\-3 & 6 y\end{pmatrix}.

It is only local: along the line y = 0, f = x^3 falls below any value, e.g. f(-2, 0) = -8.

Why it matters for optimization

The Hessian decides convexity: a function of several variables is convex exactly when its Hessian has no negative eigenvalue anywhere. Here H changes with the point and is indefinite at the saddle, so f is not convex, and a solver could only promise a local minimum. Newton’s method in several variables uses the Hessian in place of f''.