Beautiful Math
Path

Calculus intuition · Lesson 5

Gradients and learning

In many variables, the gradient packs all partial slopes into one vector that points toward steepest increase. Learning often walks the opposite way.

Core ~28 minDeep study ~78 minWhat is a partial derivative?Why descend the gradient?

Before this: Rates of change, The derivative

How this idea was born

One knob is a dial you can feel. Many knobs at once - recipe parameters, machine settings, network weights - create a foggy hillside where every direction changes the score. One-variable derivatives were enough for a single slope. Learning and design need a single object that packages every partial slope: the gradient.
Joseph-Louis Lagrange (1736-1813) sought analytic clarity in mechanics and optimization, preferring elegant algebraic forms over geometric clutter. MacTutor sketches the career. The gradient and constrained-optimization traditions that follow his spirit ask: what is the cleanest way to express how a quantity depends on many knobs? Steal the taste for clean structure when ideas get high-dimensional.
This chapter is the bridge for a modern rereader from calculus intuition to the geometry of training. You need rates of change and the derivative; the previous limits-and-optimization chapter helps with critical points and linear-step honesty. A full linear algebra course is not required. The vectors lesson in linear algebra is recommended when you want more geometry; below is a self-contained mini bridge so steepest ascent does not arrive as jargon.

Mini bridge: vectors and the gradient package

A vector here is an ordered list of numbers with a direction story: (2,−1)(2, -1) means "2 steps along the first axis, 1 step opposite the second." You can scale a vector (stretch or flip) and add vectors (tip-to-tail chaining). You do not need bases, matrices, or eigenvalues yet.
A partial derivative ∂f/∂x\partial f/\partial x freezes the other knobs and reports the slope for one knob. Pack every partial into a list and you have the gradient:
∇f=(∂f∂x,∂f∂y,…).\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \ldots\right).
That list is a vector in parameter space. Its geometric claim (when the function is smooth enough and the gradient is nonzero): it points toward steepest ascent of ff. Steepest descent is the flipped list −∇f-\nabla f. You may treat that claim as the definition of the package's job before you prove it from the multivariable chain rule.

First principles

For f(x,y)f(x, y), the partial ∂f/∂x\partial f/\partial x treats yy as fixed and differentiates in xx. The gradient
∇f=(∂f∂x,∂f∂y)\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right)
points in the direction of steepest ascent (when it exists and is nonzero). Steepest descent is −∇f-\nabla f.
Each coordinate of the gradient is a slope for one knob. Together they tell you how to nudge all knobs at once for the largest instantaneous increase of ff.
If L(w)L(w) is loss and ww are parameters, a basic learning step is
wnew=wold−η∇L(wold)w_{\text{new}} = w_{\text{old}} - \eta \nabla L(w_{\text{old}})
with step size η>0\eta > 0. Too large: overshoot (linear approximation remainder bites). Too small: crawl.
startlower loss
Gradient descent: follow the downhill direction in small steps to seek a lower loss.

Worked intuition

Imagine walking a foggy hillside with only a local slope measurement under your feet. You cannot see the global minimum. You can feel which way is steepest uphill, then step the other way. That is gradient descent as metaphor: local information, iterative hope, no guarantee of the best valley on a nonconvex map.
Networks compose functions. The gradient of a composition multiplies local Jacobians (chain rule in higher dimensions). Backpropagation organizes that bookkeeping so you do not reinvent the product rule by hand for every architecture.
Algebrica's partial-derivatives and backpropagation pages connect multivariable slopes to the chain rule in networks. The diagram above is the cartoon; those pages are the schematics.
Momentum methods (optional depth; see Distill's momentum essay) carry a kind of memory through the terrain so steps do not jitter as violently. You do not need momentum to understand the first principle: follow −∇L-\nabla L carefully.

Common confusions

Gradient points downhill. By standard convention, ∇L\nabla L points uphill on the loss surface. Descent uses the negative. Sign errors here waste days.
"The" minimum. Gradient descent finds a critical region given its start, step sizes, and noise. In deep learning the loss landscape is typically nonconvex; local minima, saddles, and flat regions are part of the lived experience. The algorithm is local, not omniscient.
Learning rate as a minor hyperparameter. η\eta is the entire personality of the walk. Enormous steps leap out of valleys. Tiny steps stall in plateaus. Schedules exist because one fixed η\eta rarely fits every phase of training.
Partial derivatives ignore interactions forever. A partial freezes other variables only for that instantaneous slope. As you step, other coordinates move too. The gradient is a local linear snapshot, not a promise about the whole journey.
Gradient magnitude as "importance" of a parameter forever. Large partials mean high local sensitivity, not permanent importance across the whole training run.
Confusing stochastic noise with true landscape features. Mini-batch gradients jitter. Not every spike is a cliff in the true risk.

History & stories

Joseph-Louis Lagrange sought analytic clarity in mechanics and optimization, preferring elegant algebraic forms over geometric clutter. MacTutor's Lagrange biography sketches the career. The gradient and constrained-optimization traditions that follow his spirit ask: what is the cleanest way to express how a quantity depends on many knobs?
You do not need eighteenth-century mechanics to use ∇\nabla. You do need the taste for structure when dimension explodes. Calculus intuition checkpoint: rates, derivatives, integrals, and now packed multivariable slopes.

Real world

Adjusting many recipe parameters to maximize taste, or tuning many machine settings to minimize defect rate, are gradient-flavored problems even when nobody writes ∇\nabla. Anytime you have a score depending on many knobs and you tweak against measured sensitivity, you are in the same family.
Foggy hill-walking remains a fair metaphor for industrial tuning: sensors give local slopes; global optimality is a hope constrained by time and risk.

Open when you want the machine-learning connection. Skip freely.

Field notes, extra examples, and glossary live here when you want more than the core path.

Carry this forward

  • Gradients pack partial slopes into a steepest-ascent vector; learning walks −∇L-\nabla L.
  • Step size is first-class, not a footnote.
  • Backprop is the chain rule with engineering discipline.
  • Calculus intuition checkpoint complete. Linear algebra gives the language of the spaces those gradients live in.

Read more

Go deeper when curiosity hits - videos, essays, and primary trails.

Check your understanding

A short learning loop - try a few, learn from misses, mark complete when you have engaged. No timer, no scoreboard.

1.A partial derivative…
2.Gradient ∇f…
3.Gradient descent steps…
4.Huge learning rate can…
5.Backprop is…

Try at least 3 core prompts, or choose I'll return later.