A.5 Inner Product and Projections

Inner Product and Orthogonality

To do basic geometry, we need length, and we need angles. We have already seen the euclidean length, so let us figure out how to compute angles. Mostly, we are worried about the right angleto the power 1.

Given two (column) vectors in $ℝ^{n}$ , we define the (standard) inner product as the dot product:

$⟨ \vec{x}, \vec{y} ⟩ = \vec{x} \cdot \vec{y} = {\vec{y}}^{T} \vec{x} = x_{1} y_{1} + x_{2} y_{2} + \dots + x_{n} y_{n} = \sum_{i = 1}^{n} x_{i} y_{i} .$

Why do we seemingly give a new notation for the dot product? Because there are other possible inner products, which are not the dot product, although we will not worry about others here. An inner product can even be defined on spaces of functions as we do in Chapter 4:

$⟨ f open parenthesis t close parenthesis, g open parenthesis t close parenthesis ⟩ equals ∫ a b f open parenthesis t close parenthesis g open parenthesis t close parenthesis d t.$

But we digress.

The inner product satisfies the following rules:

$⟨ x →, x → ⟩ greater than or equal to 0$ , and $⟨ x →, x → ⟩ equals 0$ if and only if $x → equals 0$ ,
$⟨ \vec{x}, \vec{y} ⟩ = ⟨ \vec{y}, \vec{x} ⟩$ ,
$⟨ a \vec{x}, \vec{y} ⟩ = ⟨ \vec{x}, a \vec{y} ⟩ = a ⟨ \vec{x}, \vec{y} ⟩$ ,
$⟨ \vec{x} + \vec{y}, \vec{z} ⟩ = ⟨ \vec{x}, \vec{z} ⟩ + ⟨ \vec{y}, \vec{z} ⟩$ and $⟨ \vec{x}, \vec{y} + \vec{z} ⟩ = ⟨ \vec{x}, \vec{y} ⟩ + ⟨ \vec{x}, \vec{z} ⟩$ .

Anything that satisfies the properties above can be called an inner product, although in this section we are concerned with the standard inner product in $ℝ^{n}$ .

The standard inner product gives the euclidean length:

$‖ \vec{x} ‖ = \sqrt{⟨ \vec{x}, \vec{x} ⟩} = \sqrt{x_{1}^{2} + x_{2}^{2} + \dots + x_{n}^{2}} .$

How does it give angles?

You may recall from multivariable calculus, that in two or three dimensions, the standard inner product (the dot product) gives you the angle between the vectors:

$⟨ \vec{x}, \vec{y} ⟩ = ‖ \vec{x} ‖ ‖ \vec{y} ‖ \cos θ .$

That is,

θ

is the angle that

\vec{x}

and

\vec{y}

make when they are based at the same point.

In $ℝ^{n}$ (any dimension), we are simply going to say that $θ$ from the formula is what the angle is. This makes sense as any two vectors based at the origin lie in a 2-dimensional plane (subspace), and the formula works in 2 dimensions. In fact, one could even talk about angles between functions this way, and we do in Chapter 4, where we talk about orthogonal functions (functions at right angle to each other).

To compute the angle we compute

$\cos θ = \frac{⟨ \vec{x}, \vec{y} ⟩}{‖ \vec{x} ‖ ‖ \vec{y} ‖} .$

Our angles are always in radians. We are computing the cosine of the angle, which is really the best we can do. Given two vectors at an angle

θ

, we can give the angle as

- θ

2 π minus θ

, etc., see Figure

1

. Fortunately,

\cos θ = \cos (- θ) = \cos (2 π - θ)

. If we solve for

θ

using the inverse cosine

\cos^{- 1}

, we can just decree that

0 less than or equal to θ less than or equal to π

Picture of the angle between two vectors. Arc arrow from y to x with angle theta. Backwards is -theta. — Figure $1$ : Angle between vectors.

As we said, the most important angle is the right angle. A right angle is $\frac{π}{2}$ radians, and $\cos (\frac{π}{2}) = 0$ , so the formula is particularly easy in this case. We say vectors $\vec{x}$ and $\vec{y}$ are orthogonal if they are at right angles, that is if

$⟨ x →, y → ⟩ equals 0.$

The vectors

open parenthesis 1, 0, 0, 1 close parenthesis

and

open parenthesis 1, 2, 3, minus 1 close parenthesis

are orthogonal. So are

open parenthesis 1, 1 close parenthesis

and

open parenthesis 1, minus 1 close parenthesis

. However,

open parenthesis 1, 1 close parenthesis

and

open parenthesis 1, 2 close parenthesis

are not orthogonal as their inner product is

3

and not

0

Orthogonal Projection

A typical application of linear algebra is to take a difficult problem, write everything in the right basis, and in this new basis the problem becomes simple. A particularly useful basis is an orthogonal basis, that is a basis where all the basis vectors are orthogonal. When we draw a coordinate system in two or three dimensions, we almost always draw our axes as orthogonal to each other.

Generalizing this concept to functions, it is particularly useful in Chapter 4 to express a function using a particular orthogonal basis, the Fourier series.

To express one vector in terms of an orthogonal basis, we need to first project one vector onto another. Given a nonzero vector $\vec{v}$ , we define the orthogonal projection of $\vec{w}$ onto $\vec{v}$ as

${proj}_{\vec{v}} (\vec{w}) = (\frac{⟨ \vec{w}, \vec{v} ⟩}{⟨ \vec{v}, \vec{v} ⟩}) \vec{v} .$

For the geometric idea, see Figure

2

. That is, we find the "shadow of

\vec{w}

" on the line spanned by

\vec{v}

if the direction of the sun’s rays were exactly perpendicular to the line. Another way of thinking about it is that the tip of the arrow of

{proj}_{\vec{v}} (\vec{w})

is the closest point on the line spanned by

\vec{v}

to the tip of the arrow of

\vec{w}

. In terms of euclidean distance,

\vec{u} = {proj}_{\vec{v}} (\vec{w})

minimizes the distance

‖ \vec{w} - \vec{u} ‖

among all vectors

\vec{u}

that are multiples of

\vec{v}

. Because of this, this projection comes up often in applied mathematics in all sorts of contexts we cannot solve a problem exactly: We can’t always solve "Find

\vec{w}

as a multiple of

\vec{v}

" but

{proj}_{\vec{v}} (\vec{w})

is the best "solution."

Function graph showing the parametric curve (t*cos(a), t*sin(a)) for t in [-7, 7], the parametric curve (3*t, 4*t) for t in [0, 1], the parametric curve ((3*cos(a)+4*sin(a))*cos(a)*t, (3*cos(a)+4*sin(a))*sin(a)*t) for t in [0, 1] and the parametric curve ((3*cos(a)+4*sin(a))*cos(a) + (3 - (3*cos(a)+4*sin(a))*cos(a))*t, (3*cos(a)+4*sin(a))*sin(a) + (4 - (3*cos(a)+4*sin(a))*sin(a))*t) for t in [0, 1]. Adjustable parameter: Direction of v (a) = 0.79 rad. Viewing window: x from -5.91 to 7.91, y from -3.47 to 5.07. — w = (3, 4) in blue, the line spanned by v dashed green, proj_v(w) in orange, and w − proj_v(w) dashed in purple — drawn from the tip of the projection to the tip of w, so the right angle is the one you can actually see. Drag the direction a to sweep the shadow along the line: the purple segment meets the green line at a right angle at every single setting, which is the check this section runs once by hand. At a = 0.7854 (π/4) the projection is (3.5, 3.5) = (7/2)·(1, 1) and at a = −0.7854 it is (−0.5, 0.5) = (−1/2)·(1, −1) — the very coefficients this section computes for (3, 4) in the orthogonal basis (1, 1), (1, −1). Push a past −0.64, where tan a = −3/4, and the projection collapses to a point, because there v ⟂ w and the shadow is nothing at all.

A 3D scene showing the vector v = (2, 1, c) as an orange segment from the origin, its orthogonal projection (2, 1, 0) as a blue segment lying in the translucent xy-plane, and a vertical dashed-style drop line connecting the tip of v to the tip of its projection. Dragging the slider changes only the height c of v; the blue projection in the plane stays fixed, showing that projection discards exactly the component perpendicular to the plane. — The vector v = (2, 1, c) in orange, its orthogonal projection (2, 1, 0) onto the xy-plane in blue, and the perpendicular drop between them. Drag the slider for c: the projection never moves - projecting onto the plane simply discards the component orthogonal to it.

Picture of the orthogonal projection right triangle with sides w and proj_v(w), with proj_v(w) in the direction of v. — Figure $2$ : Orthogonal projection.

The formula follows from basic trigonometry. The length of ${proj}_{\vec{v}} (\vec{w})$ should be $\cos θ$ times the length of $\vec{w}$ , that is $(\cos θ) ‖ \vec{w} ‖$ . We take the unit vector in the direction of $\vec{v}$ , that is, $\frac{\vec{v}}{‖ \vec{v} ‖}$ and we multiply it by the length of the projection. In other words,

${proj}_{\vec{v}} (\vec{w}) = (\cos θ) ‖ \vec{w} ‖ \frac{\vec{v}}{‖ \vec{v} ‖} = \frac{(\cos θ) ‖ \vec{w} ‖ ‖ \vec{v} ‖}{{‖ \vec{v} ‖}^{2}} \vec{v} = \frac{⟨ \vec{w}, \vec{v} ⟩}{⟨ \vec{v}, \vec{v} ⟩} \vec{v} .$

Orthogonal Basis

As we said, a basis $v → sub 1, v → sub 2, …, v → sub n$ is an orthogonal basis if all vectors in the basis are orthogonal to each other, that is, if

$⟨ v → sub j, v → sub k ⟩ equals 0$

for all choices of

j

and

k

where

j ⧸ = k

(a nonzero vector cannot be orthogonal to itself). A basis is furthermore called an orthonormal basis if all the vectors in a basis are also unit vectors, that is, if all the vectors have magnitude 1. For example, the standard basis

{ open parenthesis 1, 0, 0 close parenthesis, open parenthesis 0, 1, 0 close parenthesis, open parenthesis 0, 0, 1 close parenthesis }

is an orthonormal basis of

ℝ^{3}

: Any pair is orthogonal, and each vector is of unit magnitude.

The reason why we are interested in orthogonal (or orthonormal) bases is that they make it really simple to represent a vector (or a projection onto a subspace) in the basis. The simple formula for the orthogonal projection onto a vector gives us the coefficients. In Chapter 4, we use the same idea by finding the correct orthogonal basis for the set of solutions of a differential equation. We are then able to find any particular solution by simply applying the orthogonal projection formula, which is just a couple of a inner products.

Let us come back to linear algebra. Suppose that we have a subspace and an orthogonal basis $v → sub 1, v → sub 2, …, v → sub n$ . We wish to express $\vec{x}$ in terms of the basis. If $\vec{x}$ is not in the span of the basis (when it is not in the given subspace), then of course it is not possible, but the following formula gives us at least the orthogonal projection onto the subspace, or in other words, the best approximation in the subspace.

First suppose that $\vec{x}$ is in the span. Then it is the sum of the orthogonal projections:

$\vec{x} = {proj}_{{\vec{v}}_{1}} (\vec{x}) + {proj}_{{\vec{v}}_{2}} (\vec{x}) + \dots + {proj}_{{\vec{v}}_{n}} (\vec{x}) = \frac{⟨ \vec{x}, {\vec{v}}_{1} ⟩}{⟨ {\vec{v}}_{1}, {\vec{v}}_{1} ⟩} {\vec{v}}_{1} + \frac{⟨ \vec{x}, {\vec{v}}_{2} ⟩}{⟨ {\vec{v}}_{2}, {\vec{v}}_{2} ⟩} {\vec{v}}_{2} + \dots + \frac{⟨ \vec{x}, {\vec{v}}_{n} ⟩}{⟨ {\vec{v}}_{n}, {\vec{v}}_{n} ⟩} {\vec{v}}_{n} .$

In other words, if we want to write

\vec{x} = a_{1} {\vec{v}}_{1} + a_{2} {\vec{v}}_{2} + \dots + a_{n} {\vec{v}}_{n}

, then

$a_{1} = \frac{⟨ \vec{x}, {\vec{v}}_{1} ⟩}{⟨ {\vec{v}}_{1}, {\vec{v}}_{1} ⟩}, a_{2} = \frac{⟨ \vec{x}, {\vec{v}}_{2} ⟩}{⟨ {\vec{v}}_{2}, {\vec{v}}_{2} ⟩}, \dots, a_{n} = \frac{⟨ \vec{x}, {\vec{v}}_{n} ⟩}{⟨ {\vec{v}}_{n}, {\vec{v}}_{n} ⟩} .$

Another way to derive this formula is to work in reverse. Suppose that

\vec{x} = a_{1} {\vec{v}}_{1} + a_{2} {\vec{v}}_{2} + \dots + a_{n} {\vec{v}}_{n}

. Take an inner product with

{\vec{v}}_{j}

, and use the properties of the inner product:

$\begin{matrix} ⟨ \vec{x}, {\vec{v}}_{j} ⟩ = ⟨ a_{1} {\vec{v}}_{1} + a_{2} {\vec{v}}_{2} + \dots + a_{n} {\vec{v}}_{n}, {\vec{v}}_{j} ⟩ \\ = a_{1} ⟨ {\vec{v}}_{1}, {\vec{v}}_{j} ⟩ + a_{2} ⟨ {\vec{v}}_{2}, {\vec{v}}_{j} ⟩ + \dots + a_{n} ⟨ {\vec{v}}_{n}, {\vec{v}}_{j} ⟩ . \end{matrix}$

As the basis is orthogonal, then

⟨ v → sub k, v → sub j ⟩ equals 0

whenever

k ⧸ = j

. That means that only one of the terms, the

j^{th}

one, on the right hand side is nonzero and we get

$⟨ \vec{x}, {\vec{v}}_{j} ⟩ = a_{j} ⟨ {\vec{v}}_{j}, {\vec{v}}_{j} ⟩ .$

Solving for

a_{j}

we find

a_{j} = \frac{⟨ \vec{x}, {\vec{v}}_{j} ⟩}{⟨ {\vec{v}}_{j}, {\vec{v}}_{j} ⟩}

as before.

If the basis is orthonormal rather than orthogonal, then all the denominators are one. It is easy to make a basis orthonormal—divide all the vectors by their size. If you want to decompose many vectors, it may be better to find an orthonormal basis. In the example above, the orthonormal basis we would thus create is

$(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}), (\frac{1}{\sqrt{2}}, \frac{- 1}{\sqrt{2}}) .$

Then the computation would have been

$\begin{matrix} (3, 4) = ⟨ (3, 4), (\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}) ⟩ (\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}) + ⟨ (3, 4), (\frac{1}{\sqrt{2}}, \frac{- 1}{\sqrt{2}}) ⟩ (\frac{1}{\sqrt{2}}, \frac{- 1}{\sqrt{2}}) \\ = \frac{7}{\sqrt{2}} (\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}) + \frac{- 1}{\sqrt{2}} (\frac{1}{\sqrt{2}}, \frac{- 1}{\sqrt{2}}) . \end{matrix}$

Maybe the example is not so awe inspiring, but given vectors in

ℝ to the power 20

rather than

ℝ^{2}

, then surely one would much rather do 20 inner products (or 40 if we did not have an orthonormal basis) rather than solving a system of twenty equations in twenty unknowns using row reduction of a

20 times 21

matrix.

As we said above, the formula still works even if $\vec{x}$ is not in the subspace, although then it does not get us the vector $\vec{x}$ but its projection. More concretely, suppose that $S$ is a subspace that is the span of $v → sub 1, v → sub 2, …, v → sub n$ and $\vec{x}$ is any vector. Let ${proj}_{S} (\vec{x})$ be the vector in $S$ that is the closest to $\vec{x}$ . Then

${proj}_{S} (\vec{x}) = \frac{⟨ \vec{x}, {\vec{v}}_{1} ⟩}{⟨ {\vec{v}}_{1}, {\vec{v}}_{1} ⟩} {\vec{v}}_{1} + \frac{⟨ \vec{x}, {\vec{v}}_{2} ⟩}{⟨ {\vec{v}}_{2}, {\vec{v}}_{2} ⟩} {\vec{v}}_{2} + \dots + \frac{⟨ \vec{x}, {\vec{v}}_{n} ⟩}{⟨ {\vec{v}}_{n}, {\vec{v}}_{n} ⟩} {\vec{v}}_{n} .$

Of course, if

\vec{x}

is in

S

, then

{proj}_{S} (\vec{x}) = \vec{x}

, as the closest vector in

S

\vec{x}

\vec{x}

itself. But true utility is obtained when

\vec{x}

is not in

S

. In much of applied mathematics, we cannot find an exact solution to a problem, but we try to find the best solution out of a small subset (subspace). The partial sums of Fourier series from Chapter 4 are one example. Another example is least square approximation to fit a curve to data. Yet another example is given by the most commonly used numerical methods to solve partial differential equations, the finite element methods.

Gram–Schmidt Process

Before leaving orthogonal bases, let us note a procedure for manufacturing them out of any old basis. It may not be difficult to come up with an orthogonal basis for a 2-dimensional subspace, but for a 20-dimensional subspace, it seems a daunting task. Fortunately, the orthogonal projection can be used to "project away" the bits of the vectors that are making them not orthogonal. It is called the Gram-Schmidt process.

We start with a basis of vectors $v → sub 1, v → sub 2, …, v → sub n$ . We construct an orthogonal basis $w → sub 1, w → sub 2, …, w → sub n$ as follows.

$\begin{matrix} {\vec{w}}_{1} = {\vec{v}}_{1}, \\ {\vec{w}}_{2} = {\vec{v}}_{2} - {proj}_{{\vec{w}}_{1}} ({\vec{v}}_{2}), \\ {\vec{w}}_{3} = {\vec{v}}_{3} - {proj}_{{\vec{w}}_{1}} ({\vec{v}}_{3}) - {proj}_{{\vec{w}}_{2}} ({\vec{v}}_{3}), \\ {\vec{w}}_{4} = {\vec{v}}_{4} - {proj}_{{\vec{w}}_{1}} ({\vec{v}}_{4}) - {proj}_{{\vec{w}}_{2}} ({\vec{v}}_{4}) - {proj}_{{\vec{w}}_{3}} ({\vec{v}}_{4}), \\ ⋮ \\ {\vec{w}}_{n} = {\vec{v}}_{n} - {proj}_{{\vec{w}}_{1}} ({\vec{v}}_{n}) - {proj}_{{\vec{w}}_{2}} ({\vec{v}}_{n}) - \dots - {proj}_{{\vec{w}}_{n - 1}} ({\vec{v}}_{n}) . \end{matrix}$

What we do is at the

k^{th}

step, we take

{\vec{v}}_{k}

and we subtract the projection of

{\vec{v}}_{k}

to the subspace spanned by

w → sub 1, w → sub 2, …, w → sub k minus 1

Footnotes

[1] When Euclid defined angles in his Elements, the only angle he ever really defined was the right angle.

Adapted from Differential Equations for Engineers by Jiří Lebl (Oklahoma State University), hosted on LibreTexts (math.libretexts.org) and licensed under CC BY-SA 4.0. Changes were made. License: CC-BY-SA-4.0.