Chapter 6: Orthogonality and Least Squares

Every chapter so far has asked exact questions: which points satisfy the equations, which vectors are reachable, which directions are invariant. This chapter asks the question that dominates applied mathematics: when the exact answer does not exist, what is the best one? The tool is perpendicularity — and the whole chapter runs on the one test Section 0.6 supplied: two vectors are orthogonal exactly when their dot product is zero. That test is what makes perpendicularity computable here rather than merely visible, and it is why every argument below survives into dimensions no figure can draw. The point of a subspace closest to an outside vector is found by dropping a perpendicular — the orthogonal projection — and the leftover error vector, standing at right angles to the whole subspace, is the certificate that nothing closer exists.

Perpendicularity organizes space itself. Every subspace has an orthogonal complement — the set of all vectors perpendicular to it — and the two mesh perfectly: a line through the origin pairs with a plane, their dimensions add to three, and every vector of space splits uniquely into a piece in each. Projection is exactly that splitting, performed on one vector at a time.

The payoff is least squares. A batch of real measurements almost never lies exactly on the model surface you would like to fit through it — as a system of equations, the data is overdetermined and inconsistent. But the outputs the model can produce form a subspace, and the best fit is the projection of the data onto it. In this chapter you will drop a perpendicular onto a plane, watch a line and its complement pivot in lockstep, and then take the wheel yourself: hand-tune a plane through seven data points, try to beat the normal equations, and see exactly why you cannot. A closing section removes the one thing that made that algebra heavy: you will straighten an oblique pair of spanning vectors into a perpendicular one, without ever disturbing the space they span.