Expectation and loss

Under risk, people need averages that respect chance: what an insurer should expect to pay, what a fair game should return, what a planner should treat as a center of mass when outcomes scatter. Probability weights outcomes. Expectation turns those weights into a single summary that training algorithms, insurers, and planners optimize against.

Andrey Kolmogorov axiomatized probability in 1933, giving the subject a measure-theoretic spine so expectation and dependence could live in one rigorous house. MacTutor is the short stop. Steal the ethic: pin down the rules so creativity does not collapse into confusion. Loss functions turn prediction mistakes into numbers; risk is expected loss. This chapter is where chance becomes an objective.

For a modern rereader, remember that expectation is a summary, not a destiny, and not always a value the random variable can take. Variance and ruin still matter.

For a discrete random variable $X$ with values $x_i$ and probabilities $p_i$,

It is the center of mass of the distribution.

Linearity: $\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]$ always (no independence needed). That convenience is almost unfairly useful.

Variance: $\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}X)^2]$ measures spread around the center.

A loss $\ell(y, \hat{y})$ scores a mistake. Risk is $\mathbb{E}[\ell(Y, f(X))]$. Machine learning estimates risk with an average over samples (empirical risk).

A lottery with tiny chance of a huge prize can have a shiny expected value and still be a bad personal plan - variance and ruin matter too. Expectation is a summary, not a destiny, and not always a value the random variable can take (the average of a fair die is $3.5$).

Casinos set rules so their expectation is positive. Insurance prices premiums around expected claims plus margin. Waiting-time models use expectation as a planning number with variance as a honesty check.

In ML, choosing a loss is choosing which mistakes hurt in the average that training will minimize. MSE, cross-entropy, hinge: different geometries of error.

Algebrica's mean-or-expected-value page develops discrete and continuous cases.

Strogatz's dating (secretary) problem shows expectation as strategy: interview candidates in random order, reject the first roughly $1/e$ as calibration, then take the next who beats all of them. Expectation is not only average payoff; it can tell you when to stop searching.

Expectation as a typical outcome. Means can sit where no probability mass lives. Modes and medians answer different questions.

Linearity requiring independence. Linearity of expectation holds without independence. Variance calculations do not get the same free pass.

Empirical risk as the true risk. The average on a sample is a stand-in. Generalization asks whether that stand-in tracks the expectation under the true distribution. Overfitting is the gap becoming a chasm.

Ignoring tails while optimizing means. Fat-tailed losses can make mean-focused training brittle. Companion metrics matter.

Optimizing average reward while ignoring ruin. A strategy with positive expectation can still bankrupt you via variance. Kelly-style thinking exists because means are incomplete.

Assuming empirical risk decreases imply true risk decreases. Overfitting is the counterexample industry.

Andrey Kolmogorov axiomatized probability in 1933, giving the subject a measure-theoretic spine so expectation and dependence could live in one rigorous house. MacTutor's Kolmogorov biography is the standard short reference. Foundations for a wild field: pin down the rules so creativity does not collapse into confusion.

You can compute discrete expectations without measure theory. You cannot appreciate why the general theory needed spine until the house gets large.

Insurance premiums, expected wait times, and average damage under risk scenarios are expectations. Project planning that quotes only means without spreads is incomplete communication.

A fair game has expected net gain near zero. Real casinos and real markets rarely offer you that courtesy.