Correlation, sampling, and calibration

Expectation summarizes a single random variable. Life immediately asks about pairs: when $X$ runs high, does $Y$ tend to follow? A second human need arrives with every dataset: your estimate is from a sample, not the oracle distribution. A third need is uniquely modern in ML dashboards: when a model says "$80\%$," does the world agree?

Francis Galton (1822-1911) helped push regression and correlation into quantitative scientific practice; MacTutor sketches the career. Karl Pearson later standardized the product-moment correlation coefficient widely used today; MacTutor's Pearson biography covers that biometric program. Steal Galton's measuring instinct carefully, and keep the moral warning loud: association is not automatic causation. (Galton's broader historical legacy is complicated; this chapter borrows the measurement ethic, not a sanitized hero poster.)

For a modern rereader, covariance/correlation, sampling uncertainty, and calibration are three instruments on one workbench.

For random variables $X,Y$ with finite means and variances,

(when $\sigma_X\sigma_Y>0$). Correlation scales covariance to a unitless linear-association score. $\pm 1$ signals perfect linear relationship almost surely; $0$ means uncorrelated, not necessarily independent.

Sampling uncertainty: an estimate $\hat\rho$ or $\bar X$ from $n$ draws wobbles. Larger $n$ typically tightens the wobble (under standard assumptions), but dependence, bias, and heavy tails can spoil naive confidence. Always ask what was sampled and how.

Calibration of probabilities: among cases where a model predicts probability near $p$, the observed frequency should be near $p$ if the model is calibrated. Reliability diagrams bucket predictions and compare forecast average to outcome rate. Sharpness (confident predictions) without calibration is stylish false precision.

Covariance is an expected product of centered deviations: same-direction wanderings push it positive; opposite wanderings push it negative. Correlation divides out scales so meters and millimeters can talk.

Sampling is the humility layer. A correlation of $0.4$ on $n=20$ noisy points is a rumor; on $n=10^5$ carefully collected points it is a stronger claim (still not causation).

Calibration is keeping promises about frequencies. Weather forecasts taught this publicly; classifiers must relearn it whenever they optimize a proxy loss that does not enforce reliability.

Algebrica's variance-and-covariance page is the formal stop for the first instrument.

Uncorrelated implies independent. False in general. Independence implies uncorrelated (when moments exist); the converse needs extra structure ( jointly Gaussian is the famous friendly case).

Correlation implies causation. The slogan is famous because the mistake is common. Confounders and selection create dances without direct arrows.

Calibration equals accuracy. A model can be calibrated and still useless (always predict the base rate). A model can be accurate on hard labels yet miscalibrated on probabilities.

Bigger sample fixes bias. Sample size fights variance; bias from bad sampling frames needs design fixes.

$|\rho|$ near 1 always means "strong for my task." Linear association can be strong while the relationship you care about is nonlinear or conditional.

Test-set metrics without sampling talk. Every reported AUC/accuracy is an estimate. Especially fragile on small or shifting populations.

Galton's program made co-relation a scientific habit; Pearson's product-moment coefficient standardized a workhorse formula; later statistics deepened sampling distributions and errors-in-variables. Calibration as a named ML concern rides on older meteorological and probabilistic forecasting ethics: stated probabilities should mean something in frequency.

Doctors reading risk scores need calibration, not only ranking. Marketers correlating ad spend with sales must hunt confounders (seasonality). Pollsters live and die by sampling frames.

A forecast office that says "$70\%$ rain" on many days should see rain on about $70\%$ of those days. That is the public face of this chapter.