6.3 Least Squares: Beat the Normal Equations

Seven measurements: $(- 2, - 2, - 2.4)$ , $open parenthesis minus 2, 1, 0.5 close parenthesis$ , $(- 1, - 1, - 0.9)$ , $open parenthesis 0, 0, 0.6 close parenthesis$ , $(1, - 1, - 0.3)$ , $open parenthesis 1, 2, 2.7 close parenthesis$ , $open parenthesis 2, 1, 2.2 close parenthesis$ . One model: a plane $z = a x + b y + c$ . Each data point would like the coefficients to satisfy $a x_{i} + b y_{i} + c = z_{i}$ — and read that way, each measurement is a linear equation in the unknowns $(a, b, c)$ : seven planes in coefficient space. Chapter 1 says three independent planes already pin down a point, so seven generically share nothing. The system is overdetermined and inconsistent; no plane passes through all seven points, and the interesting question is which plane misses them least.

"Least" needs a score. The residual of point $i$ is the vertical gap $r_{i} = z_{i} - (a x_{i} + b y_{i} + c)$ , and the score is the sum of squared residuals over all seven points — squared so that misses above and below both count against you, and big misses count hardest. Your job in the first figure: tune $a$ , $b$ , $c$ by hand and drive the score as low as you can. Then find out whether you beat the machine. (You will not. The point is understanding why.)

Seven small dark spheres scattered in 3D space around a tilted, semi-transparent plane through the origin. Two vertical dashed red segments connect two of the spheres to the plane, showing how far the plane misses them vertically. A second, better-fitting plane is included but hidden until the student reveals it. — Seven data points and the deliberately imperfect starting plane $z = 0.5 x + 0.5 y$ . The dashed red segments at $open parenthesis 1, 2 close parenthesis$ and $(- 2, - 2)$ are residuals — vertical gaps between point and plane. A least-squares plane is hidden in the scene; leave it hidden until you have a fit you are proud of.

Explore the figure

Orbit the data and read the two drawn residuals. Above $open parenthesis 1, 2 close parenthesis$ the plane sits at $z equals 1.5$ while the point is at $2.7$ — a gap of $1.2$ above. At $(- 2, - 2)$ the plane is at $minus 2$ and the point at $minus 2.4$ — a gap of $0.4$ below.
Judge all seven points: five sit above the starting plane, only two below. A plane that splits its misses is doing better than one that is systematically low — this one is too low, and too flat in the $y$ direction. Its score is $3.55$ .
Hide the two objects labeled residual at $open parenthesis 1, 2 close parenthesis$ and residual at $(- 2, - 2)$ — they are drawn for the starting plane only and will not follow your edits. Then edit the plane's expression: try $0.5 x + 0.9 y + 0.3$ first, and watch the plane move as you retype. Better or worse? How can you tell from the picture alone?
Keep tuning — treat the three coefficients like three sliders and iterate until you cannot visibly improve. Hand fits usually land near $a approximately equals 0.4$ – $0.5$ , $b approximately equals 0.9$ – $1.0$ , $c approximately equals 0.3$ – $0.5$ , scoring somewhere between $0.1$ and $0.3$ . Write your three numbers down.
Now reveal the hidden object labeled least-squares plane and orbit. How close did you come — and can you find any single coefficient change that would visibly improve on it?

The normal equations

Your tuning loop was a search of a subspace. Collect the seven heights a candidate plane produces into a vector in $ℝ to the power 7$ : it equals $a$ times the column of $x_{i}$ 's, plus $b$ times the column of $y_{i}$ 's, plus $c$ times a column of ones. As $(a, b, c)$ range over all values, these vectors sweep out the span of three columns — a three-dimensional subspace of $ℝ to the power 7$ — and you were hunting the point of it closest to the data vector $𝐳 equals open parenthesis z sub 1, and so on, z sub 7 close parenthesis$ . Section 6.1 answered that hunt in general: closest means the residual vector $open parenthesis r sub 1, and so on, r sub 7 close parenthesis$ is perpendicular to the subspace, i.e. to each of the three spanning columns. Writing out those three dot products — $∑ x sub i r sub i equals 0$ , $∑ y sub i r sub i equals 0$ , $∑ r sub i equals 0$ — and substituting the definition of $r_{i}$ gives the normal equations:

$15 a + 6 b - c = 11.5, 6 a + 12 b = 14.1, - a + 7 c = 2.4$

Three equations, three unknowns, one solution — the whole search compressed onto an index card.

The winner scores $0.0723$ — nearly fifty times better than the starting plane's $3.55$ , and better than any hand fit. Its seven residuals are $minus 0.046$ , $minus 0.065$ , $plus 0.077$ , $plus 0.199$ , $minus 0.132$ , $minus 0.051$ , $plus 0.018$ : small everywhere, mixed in sign, and summing to zero — the visible face of the perpendicularity conditions, since $∑ r sub i equals 0$ is literally one of them.

Feel the trade-off

The second figure replays the contest with the coefficients on actual sliders and the two residual segments now live — they stretch and shrink as you drag.

Seven small dark spheres scattered in 3D space around a tilted, semi-transparent plane. Three sliders named a, b, and c control the plane z = ax + by + c. Two vertical dashed red segments connect two of the spheres to the plane and stretch or shrink live as the sliders move. A second, better-fitting plane is included but hidden. — The same seven data points, with the plane $z = a x + b y + c$ driven by sliders $a$ , $b$ , $c$ and the dashed residual segments at $open parenthesis 1, 2 close parenthesis$ and $(- 2, - 2)$ updating live as you drag. The least-squares plane is hidden in this scene too.

From the starting setting $open parenthesis 0.5, 0.5, 0 close parenthesis$ , drag $b$ upward and watch the residual at $open parenthesis 1, 2 close parenthesis$ shrink — then watch what your improvement costs at $(- 2, - 2)$ . Every coefficient helps some points at other points' expense; that tension is what the score arbitrates.
Try to make both drawn residuals small at once, then check the rest of the data by orbiting. The best reachable setting on these sliders is about $open parenthesis 0.40, 0.95, 0.40 close parenthesis$ , scoring $0.0800$ — the true optimum lies between slider clicks, and no setting the sliders can reach attains $0.0723$ .
Reveal the hidden least-squares plane and set the sliders to hug it as closely as you can. At the optimum, no single-slider move improves the score: each slider direction has been made exactly perpendicular to the remaining error. That is the normal equations, felt by hand.

An original work of XYZ Homework, built around interactive XYZ 3D figures. Its chapter sequence is aligned to Interactive Linear Algebra (Margalit & Rabinoff, Georgia Tech, GNU FDL); this work is original, copies nothing from it, and is not affiliated with or endorsed by its authors. License: CC-BY-NC-SA-4.0.