cover

Lagrange multipliers intuition and derivation

May 10, 2022 2 min read

Lagrange multipliers are ubiquitous in optimization theory and natural sciences, such as mechanics and statistical physics. Here I work out its intuition and derivation.

Lagrange multipliers in 2D case

Suppose that you need to find a maximum/minimum of a function f(x,y)f(x, y), constrained on condition g(x,y)=cg(x, y) = c.

Direct solution of such a system would be as follows: using g(x,y)=cg(x, y) = c condition we would express yy through xx as y=h(x)y = h(x), substitute this yy into f(x,y)=f(x,h(x))f(x, y) = f(x, h(x)) and find the optimum of f(x,h(x))f(x, h(x)) in just variable, xx.

Lagrange multipliers provide a convenient alternative to this straightforward solution in the following form:

{x,yf(x,y)=λx,yg(x,y)g(x,y)=c    {f(x,y)x=λg(x,y)xf(x,y)y=λg(x,y)yg(x,y)=c\begin{cases} \nabla_{x, y} f(x, y) = \lambda \nabla_{x, y} g(x, y) \\ g(x, y) = c \end{cases} \iff \begin{cases} \frac{\partial f(x, y)}{\partial x} = \lambda \frac{\partial g(x, y)}{\partial x} \\ \frac{\partial f(x, y)}{\partial y} = \lambda \frac{\partial g(x, y)}{\partial y} \\ g(x, y) = c \end{cases}

So, instead of substituting yy, we introduce a new variable λ\lambda and solve a system of 3 equations in 3 variables.

Why the solution of this new system is equivalent to the direct method?

The interpretation of these conditions is as follows: let us draw the levels of f(x,y)f(x, y).

At the point of g(x,y)=cg(x, y) = c curve, the gradient f(x,y)\nabla f(x, y) should be orthogonal to the curve g(x,y)=cg(x, y) = c, otherwise, it would have continued to move along the g(x,y)=cg(x, y) = c curve.

See illustration of 2D case:

In 3D case similarly we optimize a function f(x,y,z)f(x, y, z) constrained on two conditions g1(x,y,z)=c1g_1(x, y, z) = c_1 and g2(x,y,z)=c2g_2(x, y, z) = c_2.

In this case intersection of curves g1(x,y,z)=c1g_1(x, y, z) = c_1 and g2(x,y,z)=c2g_2(x, y, z) = c_2 is approximated by a line.

Again, f(x,y,z)\nabla f(x, y, z) should be orthogonal to this intersection line. See illustration in 3D case:

References:


Boris Burkov

Written by Boris Burkov who lives in Moscow, Russia, loves to take part in development of cutting-edge technologies, reflects on how the world works and admires the giants of the past. You can follow me in Telegram