m physical modeling / IICSM

PHYSICS / ENGINEERING / COMPUTING

Optimization models and methods

Derive unconstrained minima, gradients, constrained optima and least-squares estimation through twenty worked examples.

Subject library · 51 guides · derivations & worked examples

Fundamental physics

Space and flight

Materials and transport

Systems and mathematics

Marine and road vehicles

Electrical and computing

Companion site

Matter pathway: atom → solid → liquid → gas → plasma. Quantum mechanics and quantum field theory provide foundations across the pathway; they are not additional phases. This is a connected modeling route, not a universal heating curve. Actual phases depend on pressure, composition, and kinetics.

1. Stationarity and curvature

Definitions & inputs. f objective,x decision vector,g gradient,H Hessian.

  1. Taylor expansion describes local slope and curvature.

    f(x+d)≃f(x)+gTd+12dTHdf(x+d)\simeq f(x)+g^Td+\tfrac12d^THd

  2. A differentiable interior optimum must have zero first variation.

    ∇f(x∗)=0\nabla f(x^*)=0

  3. Positive curvature raises the objective for all small nonzero directions.

    H≻0⇒strict local minimumH\succ0\Rightarrow\text{strict local minimum}

Interpretation. Convexity promotes a local optimum to global; zero gradient alone is insufficient.

↑ Return to definitions and contents

2. Gradient and Newton steps

Definitions & inputs. α step length,g gradient,H invertible Hessian.

  1. Move against local slope.

    x+=x−α∇f(x)x^+=x-\alpha\nabla f(x)

  2. Minimize the local quadratic model for a Newton direction.

    dN=−H−1gd_N=-H^{-1}g

  3. The quadratic error update has factors 1−αλ; bound their magnitude below one.

    0<α<2/Lfor a positive quadratic with largest Hessian eigenvalue L0<\alpha<2/L\quad\text{for a positive quadratic with largest Hessian eigenvalue }L

Interpretation. Nonconvex or ill-conditioned problems need safeguards; Newton is not globally convergent by default.

↑ Return to definitions and contents

3. Constraints and KKT

Definitions & inputs. g_i(x)≤0 inequalities,h(x)=0 equalities,λi nonnegative multipliers,ν equality multiplier.

  1. Add weighted constraints to form the Lagrangian.

    L=f+∑iλigi+νTh\mathcal L=f+\sum_i\lambda_i g_i+\nu^Th

  2. Combine stationarity with primal and dual feasibility.

    ∇xL=0,g≤0, h=0, λ≥0\nabla_x\mathcal L=0,\quad g\le0,\ h=0,\ \lambda\ge0

  3. Complementarity says a slack inequality has zero multiplier.

    λigi=0\lambda_i g_i=0

Interpretation. Sign conventions for constraints determine multiplier signs; infeasible stationary points are not solutions.

↑ Return to definitions and contents

4. Least squares and regularization

Definitions & inputs. A design matrix,b observations,x unknowns,λ nonnegative penalty.

  1. Penalize the sum of squared residuals.

    f=12∥Ax−b∥2f=\tfrac12\|Ax-b\|^2

  2. Differentiate with respect to the fitted coefficients.

    ∇f=AT(Ax−b)=0\nabla f=A^T(Ax-b)=0

  3. Add λ‖x‖²/2 for ridge regularization.

    (ATA+λI)x=ATb(A^TA+\lambda I)x=A^Tb

Interpretation. QR or SVD is often numerically preferable to explicitly forming normal equations.

↑ Return to definitions and contents

Graphical worked example

Convex objective f=(x−3)²+2; unique unconstrained minimum at (3,2). X axis: Decision x (dimensionless). Y axis: Objective f(x) (dimensionless).
Convex objective f=(x−3)²+2; unique unconstrained minimum at (3,2). Related worked calculation · Download SVG · Plot data

Twenty worked examples

Open a problem to see its defined inputs, assumptions, equation, numerical substitution, result, and interpretation. Values are illustrative analytical exercises.

Example 01. Quadratic minimizer

Definitions & inputs. f(x)=(x−3)²+2.

  1. Choose the governing model and isolate the requested quantity.

    f′(x)=2(x−3)=0f'(x)=2(x-3)=0

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x∗=3x^*=3

  3. Evaluate the expression; the result uses the units shown.

    Result=3 \mathrm{Result}=3\ {}

Interpretation. Positive second derivative 2 confirms a unique global minimum.

↑ Return to definitions and contents
Example 02. Minimum objective

Definitions & inputs. f(x)=(x−3)²+2,at x*=3.

  1. Choose the governing model and isolate the requested quantity.

    f∗=f(x∗)f^*=f(x^*)

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    f∗=(3−3)2+2f^*=(3-3)^2+2

  3. Evaluate the expression; the result uses the units shown.

    Result=2 \mathrm{Result}=2\ {}

Interpretation. Decision value and objective value are different quantities.

↑ Return to definitions and contents
Example 03. Gradient at a point

Definitions & inputs. f(x)=x²,x=4.

  1. Choose the governing model and isolate the requested quantity.

    g=2xg=2x

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    g=2(4)g=2(4)

  3. Evaluate the expression; the result uses the units shown.

    Result=8 \mathrm{Result}=8\ {}

Interpretation. Units of the gradient are objective units per decision unit.

↑ Return to definitions and contents
Example 04. One gradient step

Definitions & inputs. f=x²,x0=4,α=.1.

  1. Choose the governing model and isolate the requested quantity.

    x1=x0−α(2x0)x_1=x_0-\alpha(2x_0)

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x1=4−0.1(8)x_1=4-0.1(8)

  3. Evaluate the expression; the result uses the units shown.

    Result=3.2 \mathrm{Result}=3.2\ {}

Interpretation. The new objective is smaller than the initial value.

↑ Return to definitions and contents
Example 05. Gradient contraction

Definitions & inputs. f=x²,α=.1.

  1. Choose the governing model and isolate the requested quantity.

    xk+1=(1−2α)xkx_{k+1}=(1-2\alpha)x_k

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    r=1−2(0.1)r=1-2(0.1)

  3. Evaluate the expression; the result uses the units shown.

    Result=0.8 \mathrm{Result}=0.8\ {}

Interpretation. The decision error contracts by 0.8 per step in this quadratic.

↑ Return to definitions and contents
Example 06. Ten gradient steps

Definitions & inputs. Same quadratic,x0=4,r=.8.

  1. Choose the governing model and isolate the requested quantity.

    x10=r10x0x_{10}=r^{10}x_0

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x10=4(0.8)10x_{10}=4(0.8)^{10}

  3. Evaluate the expression; the result uses the units shown.

    Result=0.4294967 \mathrm{Result}=0.4294967\ {}

Interpretation. The minimum is approached asymptotically for this step length.

↑ Return to definitions and contents
Example 07. Quadratic step upper bound

Definitions & inputs. f=x²/2,Hessian L=1.

  1. Choose the governing model and isolate the requested quantity.

    0<α<2/L0<\alpha<2/L

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    αupper=2/1\alpha_{upper}=2/1

  3. Evaluate the expression; the result uses the units shown.

    Result=2 \mathrm{Result}=2\ {}

Interpretation. At the endpoint the iterates oscillate without convergence.

↑ Return to definitions and contents
Example 08. Newton step for quadratic

Definitions & inputs. f=(x−3)²,x0=0,g=−6,H=2.

  1. Choose the governing model and isolate the requested quantity.

    x1=x0−g/Hx_1=x_0-g/H

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x1=0−(−6)/2x_1=0-(-6)/2

  3. Evaluate the expression; the result uses the units shown.

    Result=3 \mathrm{Result}=3\ {}

Interpretation. Newton reaches this exact quadratic minimizer in one step.

↑ Return to definitions and contents
Example 09. Newton step for quartic

Definitions & inputs. f=x⁴,x0=2,g=32,H=48.

  1. Choose the governing model and isolate the requested quantity.

    x1=x0−g/Hx_1=x_0-g/H

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x1=2−32/48x_1=2-32/48

  3. Evaluate the expression; the result uses the units shown.

    Result=1.333333 \mathrm{Result}=1.333333\ {}

Interpretation. It does not reach the nonquadratic minimum in one step.

↑ Return to definitions and contents
Example 10. Projection onto interval

Definitions & inputs. Candidate x=3,feasible interval [0,2].

  1. Choose the governing model and isolate the requested quantity.

    Π(x)=min⁡(2,max⁡(0,x))\Pi(x)=\min(2,\max(0,x))

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    Π(3)=2\Pi(3)=2

  3. Evaluate the expression; the result uses the units shown.

    Result=2 \mathrm{Result}=2\ {}

Interpretation. Projection minimizes Euclidean distance to the feasible interval.

↑ Return to definitions and contents
Example 11. Bound-constrained quadratic

Definitions & inputs. Minimize (x−3)² with x≤1.

  1. Choose the governing model and isolate the requested quantity.

    x∗=min⁡(3,1)x^*=\min(3,1)

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x∗=1x^*=1

  3. Evaluate the expression; the result uses the units shown.

    Result=1 \mathrm{Result}=1\ {}

Interpretation. The optimum lies at the active upper bound.

↑ Return to definitions and contents
Example 12. KKT multiplier

Definitions & inputs. Same objective and g=x−1≤0 at x*=1.

  1. Choose the governing model and isolate the requested quantity.

    2(x∗−3)+λ=02(x^*-3)+\lambda=0

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    λ=−2(1−3)\lambda=-2(1-3)

  3. Evaluate the expression; the result uses the units shown.

    Result=4 \mathrm{Result}=4\ {}

Interpretation. λ is nonnegative, satisfying dual feasibility.

↑ Return to definitions and contents
Example 13. Equality-constrained split

Definitions & inputs. Minimize x²+y² subject to x+y=10.

  1. Choose the governing model and isolate the requested quantity.

    2x+ν=2y+ν=0⇒x=y2x+\nu=2y+\nu=0\Rightarrow x=y

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x∗=10/2x^*=10/2

  3. Evaluate the expression; the result uses the units shown.

    Result=5 \mathrm{Result}=5\ {}

Interpretation. The other decision y is also 5.

↑ Return to definitions and contents
Example 14. Weighted resource split

Definitions & inputs. Minimize x²+4y² with x+y=10.

  1. Choose the governing model and isolate the requested quantity.

    2x=8y,x+y=102x=8y,\quad x+y=10

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    y∗=10/5y^*=10/5

  3. Evaluate the expression; the result uses the units shown.

    Result=2 \mathrm{Result}=2\ {}

Interpretation. The corresponding x=8 gives lower penalty to the more expensive y component.

↑ Return to definitions and contents
Example 15. Least-squares constant

Definitions & inputs. Observations 2,4,6; fit one common value.

  1. Choose the governing model and isolate the requested quantity.

    x∗=∑bi/nx^*=\sum b_i/n

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x∗=(2+4+6)/3x^*=(2+4+6)/3

  3. Evaluate the expression; the result uses the units shown.

    Result=4 \mathrm{Result}=4\ {}

Interpretation. The residuals sum to zero.

↑ Return to definitions and contents
Example 16. Through-origin fitted slope

Definitions & inputs. x samples 1,2; y samples 2,5; fit y=ax.

  1. Choose the governing model and isolate the requested quantity.

    a=∑xiyi/∑xi2a=\sum x_iy_i/\sum x_i^2

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    a=(1(2)+2(5))/(1+4)a=(1(2)+2(5))/(1+4)

  3. Evaluate the expression; the result uses the units shown.

    Result=2.4 \mathrm{Result}=2.4\ {}

Interpretation. An intercept was explicitly excluded.

↑ Return to definitions and contents
Example 17. Scalar ridge estimate

Definitions & inputs. Minimize (x−3)²/2+λx²/2 with λ=2.

  1. Choose the governing model and isolate the requested quantity.

    x∗=3/(1+λ)x^*=3/(1+\lambda)

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    x∗=3/3x^*=3/3

  3. Evaluate the expression; the result uses the units shown.

    Result=1 \mathrm{Result}=1\ {}

Interpretation. Regularization shrinks the estimate toward zero.

↑ Return to definitions and contents
Example 18. Two-dimensional gradient norm

Definitions & inputs. Gradient components 3 and 4.

  1. Choose the governing model and isolate the requested quantity.

    ∥g∥2=g12+g22\|g\|_2=\sqrt{g_1^2+g_2^2}

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    ∥g∥2=9+16\|g\|_2=\sqrt{9+16}

  3. Evaluate the expression; the result uses the units shown.

    Result=5 \mathrm{Result}=5\ {}

Interpretation. Convergence tolerances need scaling appropriate to variables and objective.

↑ Return to definitions and contents
Example 19. Small linear program

Definitions & inputs. Maximize 3x+2y with x,y≥0 and x+y≤4.

  1. Choose the governing model and isolate the requested quantity.

    3x+2y≤3(x+y)≤123x+2y\le3(x+y)\le12

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    fmax=3(4)+2(0)f_{max}=3(4)+2(0)

  3. Evaluate the expression; the result uses the units shown.

    Result=12 \mathrm{Result}=12\ {}

Interpretation. The upper bound is attained at x=4,y=0, proving optimality.

↑ Return to definitions and contents
Example 20. Quadratic condition number

Definitions & inputs. Positive Hessian eigenvalues 1 and 100.

  1. Choose the governing model and isolate the requested quantity.

    κ=λmax/λmin\kappa=\lambda_{max}/\lambda_{min}

  2. Insert the stated inputs in consistent units or the explicitly defined normalized units.

    κ=100/1\kappa=100/1

  3. Evaluate the expression; the result uses the units shown.

    Result=100 \mathrm{Result}=100\ {}

Interpretation. Strong curvature anisotropy can slow unpreconditioned gradient descent.

↑ Return to definitions and contents

Symbols and units

Each derivation and problem defines its own symbols and inputs. Symbols may be reused with different meanings in other subjects. Keep units consistent, retain sufficient precision during calculation, and apply the stated validity limits.