Covariance and correlation
A cloud of points shows two quantities measured together. Can you read off from the picture how strongly y moves with x, as one number between −1 and 1?
That number is the correlation r, and the setup below gives its formula. Stage 1 asks you to guess r for six clouds before the page computes it.
Before you start · 3 quick checks on an expectation, a standard deviation and independent events, about 3 min
Covariance. Let \(X\) and \(Y\) be random variables on the same experiment, each taking finitely many values (discrete random variables, as in the lesson on PMF, PDF and CDF), with means \(\mu_X = E[X]\) and \(\mu_Y = E[Y]\). Write \(U = X - \mu_X\) and \(V = Y - \mu_Y\) for their deviations from the means. The covariance of \(X\) and \(Y\) is the mean product of the deviations,
\[\begin{aligned} &\operatorname{Cov}(X, Y) \\ &= E[(X - \mu_X)(Y - \mu_Y)] \\ &= E[UV]. \end{aligned}\]Each outcome contributes the product of the two deviations, weighted by its probability: a positive product when \(X\) and \(Y\) are on the same side of their means, a negative one when they are on opposite sides. With \(Y = X\) it is the variance: \(\operatorname{Cov}(X, X) = E[U^2] = \operatorname{Var}(X)\). Expanding the product and using linearity, as the lesson on expectation and variance did for the variance, gives the shortcut \(\operatorname{Cov}(X, Y) = E[XY] - \mu_X\mu_Y\).
Correlation. When \(\operatorname{SD}(X) \gt 0\) and \(\operatorname{SD}(Y) \gt 0\), the correlation is
\[\rho(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\operatorname{SD}(X)\operatorname{SD}(Y)}.\]The covariance carries the units of \(X\) times those of \(Y\); the correlation has no units, and it always lies between \(-1\) and 1 (stage 3 shows why).
A cloud of points. For \(n\) points \((x_1, y_1), \dots, (x_n, y_n)\) with means \(\bar{x} = \tfrac{1}{n}\sum x_i\) and \(\bar{y} = \tfrac{1}{n}\sum y_i\), the correlation is
\[r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}},\]defined when the \(x_i\) are not all equal and the \(y_i\) are not all equal. It is \(\rho(X, Y)\) for the pair \((X, Y)\) picked from the \(n\) points with probability \(\tfrac{1}{n}\) each: the factors \(\tfrac{1}{n}\) cancel.
Fine print: n or n − 1, and independence
Statistics software reports the sample covariance of data with \(n - 1\) in place of \(n\). The correlation \(r\) is the same either way, because the same factor appears above and below the fraction line.
Random variables \(X\) and \(Y\) with finitely many values are independent when \[\begin{aligned} &P(X = x, Y = y) \\ &= P(X = x)\,P(Y = y) \end{aligned}\] for all values \(x\) and \(y\): learning the value of one does not change the probabilities for the other. Stage 3 shows what independence does to the covariance.
Guess the correlation
Each cloud below has 80 points. Set your guess for its correlation \(r\) and check it; the page then computes \(r\) and shows where it comes from.
How the game works
Move the slider or type a number from \(-1\) to 1 (steps of 0.05), then press Check my guess. Your guess is locked, and the page shows the cloud's \(r\), how far your guess was from it, and the means and the sign of each point's product of deviations. Next cloud moves on; after the sixth cloud a table compares all your guesses with the true values. Start over clears everything.
The page made each cloud with a seeded random generator, so every visitor sees the same six clouds. The coordinates are rounded to two decimals.
| Cloud | Your guess | r | Distance |
|---|
Signs. Split a cloud by the dashed lines through \((\bar{x}, \bar{y})\). Points up and to the right of the center, or down and to the left, have \((x_i - \bar{x})(y_i - \bar{y}) \gt 0\); the other two quadrants give negative products. The sign of \(r\) is the sign of the sum, and a point contributes more the farther it lies from both dashed lines.
Size. \(r = 1\) or \(r = -1\) exactly when all the points lie on one straight line that is neither horizontal nor vertical. More generally, the least-squares line, the line \(y = a + bx\) that makes the sum of squared vertical misses smallest, leaves misses whose squares add up to \((1 - r^2)\sum (y_i - \bar{y})^2\). So \(r^2\) is the share of the spread of \(y\) that a straight line accounts for; at \(r = 0.5\) that share is only one quarter, so a cloud with \(r = 0.5\) still scatters widely around its line.
Zero. Cloud 6 has \(r = 0\) exactly, yet \(y\) follows \(x\) closely along a curve. A correlation of 0 says that the least-squares line is the horizontal line at \(\bar{y}\): no tilted line has a smaller sum of squared vertical misses. It does not say that \(x\) and \(y\) are unrelated.
The variance of a sum
Two predictions first. The explorer below them opens once both are answered.
Explorer: Var(X ± Y)
Choose the standard deviations \(\sigma_X = \operatorname{SD}(X)\) and \(\sigma_Y = \operatorname{SD}(Y)\), the sum or the difference, and the correlation \(\rho\). Then \(\operatorname{Cov}(X, Y) = \rho\sigma_X\sigma_Y\), and the bars show the three terms of the variance and their total.
Goals
For random variables \(X\) and \(Y\) on the same experiment, with \(\sigma_X = \operatorname{SD}(X)\), \(\sigma_Y = \operatorname{SD}(Y)\) and \(\operatorname{Cov}(X, Y) = \rho\sigma_X\sigma_Y\),
\[\begin{aligned} &\operatorname{Var}(X + Y) \\ &= \sigma_X^2 + \sigma_Y^2 + 2\rho\sigma_X\sigma_Y, \\ &\operatorname{Var}(X - Y) \\ &= \sigma_X^2 + \sigma_Y^2 - 2\rho\sigma_X\sigma_Y. \end{aligned}\]The covariance enters twice, once for each order of the cross product \(UV\) and \(VU\). With \(\sigma_X, \sigma_Y \gt 0\), a positive correlation makes the variance of a sum more and that of a difference less than \(\sigma_X^2 + \sigma_Y^2\); a negative correlation does the opposite. With \(\rho = -1\) and \(\sigma_X = \sigma_Y\), the sum \(X + Y\) has variance 0: it equals its mean with probability 1. Stage 3 proves the formula for the sum.
Why the covariance counts twice
Assumptions. \(X\) and \(Y\) are random variables on the same experiment, each taking finitely many values; they need not be independent, and either may be a constant. As in the setup, \(\mu_X = E[X]\), \(\mu_Y = E[Y]\), \(U = X - \mu_X\) and \(V = Y - \mu_Y\), so that \(\operatorname{Var}(X) = E[U^2]\), \(\operatorname{Var}(Y) = E[V^2]\) and \(\operatorname{Cov}(X, Y) = E[UV]\).
Conclusion.
\[\begin{aligned} &\operatorname{Var}(X + Y) \\ &= \operatorname{Var}(X) + \operatorname{Var}(Y) \\ &\quad {} + 2\operatorname{Cov}(X, Y). \end{aligned}\]Facts the proof uses.
- linearity of expectation: for numbers \(a\), \(b\), \(c\), \(d\) and random variables \(A\), \(B\), \(C\) on the same experiment, \[\begin{aligned} &E[aA + bB + cC + d] \\ &= aE[A] + bE[B] \\ &\quad {} + cE[C] + d, \end{aligned}\] with no independence needed;
- the definition of variance, \(\operatorname{Var}(Z) = E[(Z - E[Z])^2]\);
- ordinary algebra of numbers, applied to the values of the random variables in each outcome.
Each option is one step about the \(X\) and \(Y\) of the theorem. Pick the one that is true for every pair the theorem covers, constants and dependent pairs included.
- Scaled sums. For numbers \(a\) and \(b\), the deviation of \(aX + bY\) is \(aU + bV\), and the same steps give \[\begin{aligned} &\operatorname{Var}(aX + bY) \\ &= a^2\operatorname{Var}(X) + b^2\operatorname{Var}(Y) \\ &\quad {} + 2ab\operatorname{Cov}(X, Y). \end{aligned}\] With \(a = 1\) and \(b = -1\) this is the formula for \(X - Y\) from stage 2.
- Change of units. For numbers \(a\), \(b\), \(c\), \(d\), the deviations of \(aX + c\) and \(bY + d\) are \(aU\) and \(bV\), so \[\begin{aligned} &\operatorname{Cov}(aX + c, bY + d) \\ &= ab\operatorname{Cov}(X, Y). \end{aligned}\] When \(ab \gt 0\) the correlation (where it is defined) does not change, and when \(ab \lt 0\) it changes sign, because \(\operatorname{SD}(aX + c) = \lvert a \rvert\operatorname{SD}(X)\).
- Independence. If \(X\) and \(Y\) are independent, then \[\begin{aligned} &E[XY] \\ &= \sum_x \sum_y xy\,P(X = x)P(Y = y) \\ &= E[X]E[Y], \end{aligned}\] so \(\operatorname{Cov}(X, Y) = 0\) and the variances add: \[\begin{aligned} &\operatorname{Var}(X + Y) \\ &= \operatorname{Var}(X) + \operatorname{Var}(Y). \end{aligned}\] The converse fails: if \(X\) is \(-1\), 0 or 1 with probability \(\tfrac{1}{3}\) each and \(Y = X^2\), then \(E[X] = 0\) and \(E[XY] = 0\), so \(\operatorname{Cov}(X, Y) = 0\), yet \(P(Y = 0 \mid X = 0) = 1\) while \(P(Y = 0) = \tfrac{1}{3}\).
- Why \(-1 \le \rho \le 1\). Let \(\operatorname{SD}(X) = \sigma_X \gt 0\) and \(\operatorname{SD}(Y) = \sigma_Y \gt 0\). The scaled-sum formula with \(a = 1/\sigma_X\) and \(b = \pm 1/\sigma_Y\) gives \[\operatorname{Var}\Big(\frac{X}{\sigma_X} \pm \frac{Y}{\sigma_Y}\Big) = 2 \pm 2\rho,\] and a variance is never negative, so \(-1 \le \rho \le 1\). Equality \(\rho = -1\) or \(\rho = 1\) makes one of these variances 0, which happens exactly when \(Y = c + dX\) with probability 1 for numbers \(c\) and \(d \ne 0\). The same argument gives \(\lvert\operatorname{Cov}(X, Y)\rvert \le \operatorname{SD}(X)\operatorname{SD}(Y)\) whenever both standard deviations are positive; when one of them is 0, that variable is constant with probability 1 and both sides are 0. It also gives \(-1 \le r \le 1\) for every cloud.
Problems
Each problem has hints and a full solution.
Further reading
- J. K. Blitzstein and J. Hwang, Introduction to Probability, 2nd ed. (CRC Press, 2019), ch. 7, "Joint Distributions," section 7.3 on covariance and correlation.
- S. Ross, A First Course in Probability, 10th ed. (Pearson, 2019), ch. 7, "Properties of Expectation," section 7.4 on covariance, variance of sums and correlations.
- Z. Bodie, A. Kane and A. J. Marcus, Investments, 10th ed. (McGraw-Hill, 2014), ch. 7, "Optimal Risky Portfolios," on the variance of a two-asset portfolio.