Dot products and projections

Vectors can point anywhere. Physics needs work (force along a displacement). Recommendation systems need a meter for alignment: how much do two directions agree? Without that meter, "similar" stays a shrug.

Hermann Grassmann (1809-1877) built a vast geometric algebra in the Ausdehnungslehre, ahead of its time and long ignored. MacTutor records how he kept developing the ideas anyway. Steal the ethic carefully: when you see a cleaner language for space, write it down even if the room is not ready. Inner products are that language's heartbeat; the dot product is the meter you meet first.

From it you get lengths, angles, and projections - the geometry behind similarity search, least squares shadows, and attention scores. For a modern rereader, this chapter is where arrows stop being lonely and start talking about agreement.

For $\mathbf{u}, \mathbf{v}$ in $\mathbb{R}^n$,

when the geometric interpretation applies. Large positive: aligned. Near zero: orthogonal. Negative: opposing.

Length comes free: $\|\mathbf{v}\| = \sqrt{\mathbf{v}\cdot\mathbf{v}}$.

The shadow of $\mathbf{v}$ onto the direction of $\mathbf{u}$ is

(for $\mathbf{u} \neq 0$). That is "how much of $\mathbf{v}$ lies along $\mathbf{u}$."

Dot product adds the products of corresponding components. Geometrically it secretly contains $\cos\theta$. Projection uses that number to drop a perpendicular shadow of one vector onto another - the core of least squares later.

If you push a crate at an angle, only the component along the motion does work. Dot product filters the aligned part. Physics called this pattern "work" long before ML called a cousin "cosine similarity."

Cosine similarity is the dot product of normalized vectors: divide by lengths so magnitude stops dominating angle. That is why recommenders often prefer cosine when vector lengths reflect popularity more than affinity.

Algebrica's inner-product-spaces page and cosine-similarity page are the formal and applied twins of this lesson. Projection is "how much of this lies along that."

Large dot product always means "similar." Without normalization, long vectors can dominate. Two long nearly-orthogonal vectors can outscore two short aligned ones. Decide whether you care about angle, magnitude, or both.

Orthogonal means "unrelated in the real world." Orthogonal means zero inner product in the chosen geometry. It is a mathematical silence, not a metaphysical claim about causation.

Projection as "delete the other parts forever." Projection extracts a component along a subspace. The residual $\mathbf{v} - \mathrm{proj}_{\mathbf{u}}\mathbf{v}$ is the leftover, orthogonal to $\mathbf{u}$ in the Euclidean case. Least squares lives on that residual story.

Attention is "just" a soft lookup. Scaled dot-product attention uses alignment scores between queries and keys, then softmax. The soft lookup is built on this chapter's meter.

Using cosine on uncentered data carelessly. Mean structure can dominate angles. Sometimes people subtract means first; know whether you did.

Thinking projection removes noise always. Projection onto a poor subspace can remove signal. The subspace choice is the plot.

Hermann Grassmann's Ausdehnungslehre built a vast geometric algebra ahead of its time and was long ignored. He kept developing the ideas anyway. MacTutor's Grassmann biography is the standard short map. The ethic is conviction: cleaner language for space is worth writing even when the room is not ready.

You do not need Grassmann's full system to use $\mathbf{u}\cdot\mathbf{v}$. You do need his courage toward structure when coordinate mess tempts you to give up.

Work in physics is force dotted with displacement. Navigation and robotics constantly decompose motion into components along useful axes. Recommendation "similarity" is often a cosine of feature vectors. Search ranking and clustering reuse the same alignment instinct.

Whenever someone says "these directions agree," hear a dot product humming - or hear a missing normalization warning.