Chapter 6: Orthogonality and Least Squares

Every chapter so far has asked exact questions: which points satisfy the equations, which vectors are reachable, which directions are invariant. This chapter asks the question that dominates applied mathematics: when the exact answer does not exist, what is the best one? The tool is perpendicularity. The point of a subspace closest to an outside vector is found by dropping a perpendicular — the orthogonal projection — and the leftover error vector, standing at right angles to the whole subspace, is the certificate that nothing closer exists.

Perpendicularity organizes space itself. Every subspace has an orthogonal complement — the set of all vectors perpendicular to it — and the two mesh perfectly: a line through the origin pairs with a plane, their dimensions add to three, and every vector of space splits uniquely into a piece in each. Projection is exactly that splitting, performed on one vector at a time.

The payoff is least squares. A batch of real measurements almost never lies exactly on the model surface you would like to fit through it — as a system of equations, the data is overdetermined and inconsistent. But the outputs the model can produce form a subspace, and the best fit is the projection of the data onto it. In this chapter you will drop a perpendicular onto a plane, watch a line and its complement pivot in lockstep, and then take the wheel yourself: hand-tune a plane through seven data points, try to beat the normal equations, and see exactly why you cannot.