A.2 Matrix Algebra

One-by-One Matrices

Let us motivate what we want to achieve with matrices. Real-valued linear mappings of the real line, linear functions that eat numbers and spit out numbers, are just multiplications by a number. Consider a mapping defined by multiplying by a number. Let’s call this number $\alpha$. The mapping then takes $x$ to $\alpha x$. We can add such mappings: If we have another mapping $\beta$, then

$$\alpha x+\beta x=(\alpha +\beta )x.$$

We get a new mapping $\alpha +\beta$ that multiplies $x$ by, well, $\alpha +\beta$. If $D$ is a mapping that doubles its input, $Dx=2x$, and $T$ is a mapping that triples, $Tx=3x$, then $D+T$ is a mapping that multiplies by $5$, $(D+T)x=5x$.

Similarly we can compose such mappings, that is, we could apply one and then the other. We take $x$, we run it through the first mapping $\alpha$ to get $\alpha$ times $x$, then we run $\alpha x$ through the second mapping $\beta$. In other words,

$$\beta (\alpha x)=(\beta \alpha )x.$$

We just multiply those two numbers. Using our doubling and tripling mappings, if we double and then triple, that is $T(Dx)$ then we obtain $3(2x)=6x$. The composition $TD$ is the mapping that multiplies by $6$. For larger matrices, composition also ends up being a kind of multiplication.

Matrix Addition and Scalar Multiplication

The mappings that multiply numbers by numbers are just $1\times 1$ matrices. The number $\alpha$ above could be written as a matrix $[\alpha ]$. Perhaps we would want to do to all matrices the same things that we did to those $1\times 1$ matrices at the start of this section above. First, let us add matrices. If we have a matrix $A$ and a matrix $B$ that are of the same size, say $m\times n$, then they are mappings from ${ℝ}^n$ to ${ℝ}^m$. The mapping $A+B$ should also be a mapping from ${ℝ}^n$ to ${ℝ}^m$, and it should do the following to vectors:

$$(A+B)\overrightarrow{x}=A\overrightarrow{x}+B\overrightarrow{x}.$$

It turns out you just add the matrices element-wise: If the $i{j}^{\text{th}}$ entry of $A$ is $a_{ij}$, and the $i{j}^{\text{th}}$ entry of $B$ is $b_{ij}$, then the $i{j}^{\text{th}}$ entry of $A+B$ is $a_{ij}+b_{ij}$. If

$$A=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right]\text{and}B=\left[\begin{array}{ccc}{b}_{11}& b_{12}& b_{13}\\ b_{21}& b_{22}& b_{23}\end{array}\right],$$

$$A+B=\left[\begin{array}{ccc}{a}_{11}+b_{11}& a_{12}+b_{12}& a_{13}+b_{13}\\ a_{21}+b_{21}& a_{22}+b_{22}& a_{23}+b_{23}\end{array}\right].$$

Let us illustrate on a more concrete example:

$$\left[\begin{array}{cc}1& 2\\ 3& 4\\ 5& 6\end{array}\right]+\left[\begin{array}{cc}7& 8\\ 9& 10\\ 11& -1\end{array}\right]=\left[\begin{array}{cc}1+7& 2+8\\ 3+9& 4+10\\ 5+11& 6-1\end{array}\right]=\left[\begin{array}{cc}8& 10\\ 12& 14\\ 16& 5\end{array}\right].$$

Let’s check that this does the right thing to a vector. Let’s use some of the vector algebra that we already know, and regroup things:

$$\begin{array}{c}\left[\begin{array}{cc}1& 2\\ 3& 4\\ 5& 6\end{array}\right]\left[\begin{array}{c}2\\ -1\end{array}\right]+\left[\begin{array}{cc}7& 8\\ 9& 10\\ 11& -1\end{array}\right]\left[\begin{array}{c}2\\ -1\end{array}\right]=\left(2\right[\begin{array}{c}1\\ 3\\ 5\end{array}]-[\begin{array}{c}2\\ 4\\ 6\end{array}\left]\right)+\left(2\right[\begin{array}{c}7\\ 9\\ 11\end{array}]-[\begin{array}{c}8\\ 10\\ -1\end{array}\left]\right)\\ =2\left(\right[\begin{array}{c}1\\ 3\\ 5\end{array}]+[\begin{array}{c}7\\ 9\\ 11\end{array}\left]\right)-\left(\right[\begin{array}{c}2\\ 4\\ 6\end{array}]+[\begin{array}{c}8\\ 10\\ -1\end{array}\left]\right)\\ =2\left[\begin{array}{c}1+7\\ 3+9\\ 5+11\end{array}\right]-\left[\begin{array}{c}2+8\\ 4+10\\ 6-1\end{array}\right]=2\left[\begin{array}{c}8\\ 12\\ 16\end{array}\right]-\left[\begin{array}{c}10\\ 14\\ 5\end{array}\right]\\ =\left[\begin{array}{cc}8& 10\\ 12& 14\\ 16& 5\end{array}\right]\left[\begin{array}{c}2\\ -1\end{array}\right](=[\begin{array}{c}2(8)-10\\ 2(12)-14\\ 2(16)-5\end{array}]=[\begin{array}{c}6\\ 10\\ 27\end{array}\left]\right).\end{array}$$

If we replaced the numbers by letters that would constitute a proof! You’ll notice that we didn’t really have to even compute what the result is to convince ourselves that the two expressions were equal.

If the sizes of the matrices do not match, then addition is not defined. If $A$ is $3\times 2$ and $B$ is $2\times 5$, then we cannot add these matrices. We don’t know what that could possibly mean.

It is also useful to have a matrix that when added to any other matrix does nothing. This is the zero matrix, the matrix of all zeros:

$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right].$$

We often denote the zero matrix by $0$ without specifying size. We would then just write $A+0$, where we just assume that $0$ is the zero matrix of the same size as $A$.

There are really two things we can multiply matrices by. We can multiply matrices by scalars or we can multiply by other matrices. Let us first consider multiplication by scalars. For a matrix $A$ and a scalar $\alpha$, we want $\alpha A$ to be the matrix that accomplishes

$$(\alpha A)\overrightarrow{x}=\alpha (A\overrightarrow{x}).$$

That is just scaling the result by $\alpha$. If you think about it, scaling every term in $A$ by $\alpha$ achieves just that: If

$$A=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right],\text{then}\alpha A=\left[\begin{array}{ccc}\alpha a_{11}& \alpha a_{12}& \alpha a_{13}\\ \alpha a_{21}& \alpha a_{22}& \alpha a_{23}\end{array}\right].$$

For example,

$$2\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right]=\left[\begin{array}{ccc}2& 4& 6\\ 8& 10& 12\end{array}\right].$$

Let us list some properties of matrix addition and scalar multiplication. Denote by $0$ the zero matrix, by $\alpha$, $\beta$ scalars, and by $A$, $B$, $C$ matrices. Then:

$$\begin{array}{c}A+0=A=0+A,\\ A+B=B+A,\\ (A+B)+C=A+(B+C),\\ \alpha (A+B)=\alpha A+\alpha B,\\ (\alpha +\beta )A=\alpha A+\beta A.\end{array}$$

Matrix Multiplication

As we mentioned above, composition of linear mappings is also a multiplication of matrices. Suppose $A$ is an $m\times n$ matrix, that is, $A$ takes ${\mathbb{R}}^n$ to ${\mathbb{R}}^m$, and $B$ is an $n\times p$ matrix, that is, $B$ takes ${\mathbb{R}}^p$ to ${\mathbb{R}}^n$. The composition $AB$ should work as follows

$$AB\overrightarrow{x}=A(B\overrightarrow{x}).$$

First, a vector $\overrightarrow{x}$ in ${\mathbb{R}}^p$ gets taken to the vector $B\overrightarrow{x}$ in ${\mathbb{R}}^n$. Then the mapping $A$ takes it to the vector $A(B\overrightarrow{x})$ in ${\mathbb{R}}^m$. In other words, the composition $AB$ should be an $m\times p$ matrix. In terms of sizes we should have

Notice how the middle size must match.

OK, now we know what sizes of matrices we should be able to multiply, and what the product should be. Let us see how to actually compute matrix multiplication. We start with the so-called dot product (or inner product) of two vectors. Usually this is a row vector multiplied with a column vector of the same size. Dot product multiplies each pair of entries from the first and the second vector and sums these products. The result is a single number. For example,

$$\left[\begin{array}{ccc}{a}_1& a_2& a_3\end{array}\right]·\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\end{array}\right]=a_1{b}_1+a_2{b}_2+a_3{b}_3.$$

Armed with the dot product we define the product of matrices. We denote by ${row}_i(A)$ the $i^{\text{th}}$ row of $A$ and by ${column}_j(A)$ the $j^{\text{th}}$ column of $A$. For an $m\times n$ matrix $A$ and an $n\times p$ matrix $B$ we can compute the product $AB$: The matrix $AB$ is an $m\times p$ matrix whose $i{j}^{\text{th}}$ entry is the dot product

$${row}_i(A)·{column}_j(B).$$

For example, given a $2\times 3$ and a $3\times 2$ matrix we should end up with a $2\times 2$ matrix:

$$\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right]\left[\begin{array}{cc}-1& 2\\ -7& 0\\ 1& -1\end{array}\right]=\left[\begin{array}{ccc}1·(-1)+2·(-7)+3·1& & 1·2+2·0+3·(-1)\\ 4·(-1)+5·(-7)+6·1& & 4·2+5·0+6·(-1)\end{array}\right]=\left[\begin{array}{cc}-12& -1\\ -33& 2\end{array}\right].$$

A useful consequence of the definition is that the evaluation $A\overrightarrow{x}$ for a matrix $A$ and a (column) vector $\overrightarrow{x}$ is also matrix multiplication. That is really why we think of vectors as column vectors, or $n\times 1$ matrices. For example,

$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]\left[\begin{array}{c}2\\ -1\end{array}\right]=\left[\begin{array}{c}1·2+2·(-1)\\ 3·2+4·(-1)\end{array}\right]=\left[\begin{array}{c}0\\ 2\end{array}\right].$$

You should stare at the computation of multiplication of matrices $AB$ and the previous definition of $A\overrightarrow{y}$ as a mapping for a moment. What we are doing with matrix multiplication is applying the mapping $A$ to the columns of $B$. This is usually written as follows. Suppose we write the $n\times p$ matrix $B=[{\overrightarrow{b}}_1{\overrightarrow{b}}_2\cdots {\overrightarrow{b}}_p]$, where ${\overrightarrow{b}}_1,{\overrightarrow{b}}_2,\dots ,{\overrightarrow{b}}_p$ are the columns of $B$. Then for an $m\times n$ matrix $A$,

$$AB=A[{\overrightarrow{b}}_1{\overrightarrow{b}}_2\cdots {\overrightarrow{b}}_p]=[A{\overrightarrow{b}}_{1}A{\overrightarrow{b}}_2\cdots A{\overrightarrow{b}}_p].$$

The columns of the $m\times p$ matrix $AB$ are the vectors $A{\overrightarrow{b}}_1,A{\overrightarrow{b}}_2,\dots ,A{\overrightarrow{b}}_p$. For example, in (9.2.1), the columns of

$$\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right]\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\\ b_{31}& b_{32}\end{array}\right]$$

$$\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right]\left[\begin{array}{c}{b}_{11}\\ b_{21}\\ b_{31}\end{array}\right]\text{and}\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right]\left[\begin{array}{c}{b}_{12}\\ b_{22}\\ b_{32}\end{array}\right].$$

Rules of Matrix Algebra

For multiplication we want an analogue of a 1. That is, we desire a matrix that just leaves everything as it found it. This analogue is the so-called identity matrix. The identity matrix is a square matrix with 1s on the main diagonal and zeros everywhere else. It is usually denoted by $I$. For each size we have a different identity matrix and so sometimes we may denote the size as a subscript. For example, $I_3$ is the $3\times 3$ identity matrix

$$I=I_3=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right].$$

$$\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]=\left[\begin{array}{ccc}{a}_{11}·1+a_{12}·0& & a_{11}·0+a_{12}·1\\ a_{21}·1+a_{22}·0& & a_{21}·0+a_{22}·1\end{array}\right]=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right].$$

We have the following rules for matrix multiplication. Suppose that $A$, $B$, $C$ are matrices of the correct sizes so that the following make sense. Let $\alpha$ denote a scalar (number). Then

$$\begin{array}{c}A(BC)=(AB)C\text{(associative law)},\\ A(B+C)=AB+AC\text{(distributive law)},\\ (B+C)A=BA+CA\text{(distributive law)},\\ \alpha (AB)=(\alpha A)B=A(\alpha B),\\ IA=A=AI\text{(identity)}.\end{array}$$

A multiplication rule, one you have used since primary school on numbers, is quite conspicuously missing for matrices. That is, matrix multiplication is not commutative. Firstly, just because $AB$ makes sense, it may be that $BA$ is not even defined. For example, if $A$ is $2\times 3$, and $B$ is $3\times 4$, the we can multiply $AB$ but not $BA$.

Even if $AB$ and $BA$ are both defined, does not mean that they are equal. For example, take $A=\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]$ and $B=\left[\begin{array}{cc}1& 0\\ 0& 2\end{array}\right]$:

$$AB=\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]\left[\begin{array}{cc}1& 0\\ 0& 2\end{array}\right]=\left[\begin{array}{cc}1& 2\\ 1& 2\end{array}\right]\text{⧸}=\left[\begin{array}{cc}1& 1\\ 2& 2\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 2\end{array}\right]\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]=BA.$$

Inverse

A couple of other algebra rules you know for numbers do not quite work on matrices:

1. $AB=AC$ does not necessarily imply $B=C$, even if $A$ is not 0.
2. $AB=0$ does not necessarily mean that $A=0$ or $B=0$.

For example:

$$\left[\begin{array}{cc}0& 1\\ 0& 0\end{array}\right]\left[\begin{array}{cc}0& 1\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}0& 0\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}0& 1\\ 0& 0\end{array}\right]\left[\begin{array}{cc}0& 2\\ 0& 0\end{array}\right].$$

To make these rules hold, we do not just need one of the matrices to not be zero, we would need to by a matrix. This is where the matrix inverse comes in. Suppose that $A$ and $B$ are $n\times n$ matrices such that

$$AB=I=BA.$$

Then we call $B$ the inverse of $A$ and we denote $B$ by $A^{-1}$. Perhaps not surprisingly, ${(A^{-1})}^{-1}=A$, since if the inverse of $A$ is $B$, then the inverse of $B$ is $A$. If the inverse of $A$ exists, then we say $A$ is invertible. If $A$ is not invertible, we say $A$ is singular.

If $A=[a]$ is a $1\times 1$ matrix, then $A^{-1}$ is $a^{-1}=\frac{1}{a}$. That is where the notation comes from. The computation is not nearly as simple when $A$ is larger.

The proper formulation of the cancellation rule is:

If $A$ is invertible, then $AB=AC$ implies $B=C$.

The computation is what you would do in regular algebra with numbers, but you have to be careful never to commute matrices:

$$\begin{array}{c}AB=AC,\\ A^{-1}AB=A^{-1}AC,\\ IB=IC,\\ B=C.\end{array}$$

If $A$ is invertible, then $BA=CA$ implies $B=C$.

The rule says, among other things, that the inverse of a matrix is unique if it exists: If $AB=I=AC$, then $A$ is invertible and $B=C$.

We will see later how to compute an inverse of a matrix in general. For now, let us note that there is a simple formula for the inverse of a $2\times 2$ matrix

$${\left[}^{\begin{array}{cc}a& b\\ c& d\end{array}}=\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right].$$

$${\left[}^{\begin{array}{cc}1& 1\\ 2& 4\end{array}}=\frac{1}{1·4-1·2}\left[\begin{array}{cc}4& -1\\ -2& 1\end{array}\right]=\left[\begin{array}{cc}2& \frac{-1}{2}\\ -1& \frac{1}{2}\end{array}\right].$$

$$\left[\begin{array}{cc}1& 1\\ 2& 4\end{array}\right]\left[\begin{array}{cc}2& \frac{-1}{2}\\ -1& \frac{1}{2}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\text{and}\left[\begin{array}{cc}2& \frac{-1}{2}\\ -1& \frac{1}{2}\end{array}\right]\left[\begin{array}{cc}1& 1\\ 2& 4\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right].$$

$$\left[\begin{array}{cc}1& 1\\ 2& 2\end{array}\right]$$

$${\left[}^{\begin{array}{cc}1& 1\\ 2& 2\end{array}}=\frac{1}{1·2-1·2}\left[\begin{array}{cc}2& -1\\ -2& 1\end{array}\right]=?$$

So a $2\times 2$ matrix $A$ is invertible whenever

$$ad-bc\text{⧸}=0$$

Diagonal Matrices

A simple (and surprisingly useful) type of a square matrix is a so-called diagonal matrix. It is a matrix whose entries are all zero except those on the main diagonal from top left to bottom right. For example a $4\times 4$ diagonal matrix is of the form

$$\left[\begin{array}{cccc}{d}_1& 0& 0& 0\\ 0& d_2& 0& 0\\ 0& 0& d_3& 0\\ 0& 0& 0& d_4\end{array}\right].$$

$$\left[\begin{array}{ccc}1& 0& 0\\ 0& 2& 0\\ 0& 0& 3\end{array}\right]\left[\begin{array}{c}4\\ 5\\ 6\end{array}\right]=\left[\begin{array}{c}1·4\\ 2·5\\ 3·6\end{array}\right]=\left[\begin{array}{c}4\\ 10\\ 18\end{array}\right].$$

$$\left[\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& -1\end{array}\right]\left[\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right]=\left[\begin{array}{ccc}2& 2& 2\\ 3& 3& 3\\ -1& -1& -1\end{array}\right].$$

$$\left[\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right]\left[\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& -1\end{array}\right]=\left[\begin{array}{ccc}2& 3& -1\\ 2& 3& -1\\ 2& 3& -1\end{array}\right].$$

$$\left[\begin{array}{ccc}1& 0& 0\\ 0& 2& 0\\ 0& 0& 3\end{array}\right]\left[\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& -1\end{array}\right]=\left[\begin{array}{ccc}1·2& 0& 0\\ 0& 2·3& 0\\ 0& 0& 3·(-1)\end{array}\right]=\left[\begin{array}{ccc}2& 0& 0\\ 0& 6& 0\\ 0& 0& -3\end{array}\right].$$

$${\left[}^{\begin{array}{ccc}{d}_1& 0& 0\\ 0& d_2& 0\\ 0& 0& d_3\end{array}}=\left[\begin{array}{ccc}{d}_1^{-1}& 0& 0\\ 0& d_2^{-1}& 0\\ 0& 0& d_3^{-1}\end{array}\right].$$

$$\underset{A^{-1}}{\underbrace{{\left[}^{\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& 4\end{array}}}}\underset{A}{\underbrace{\left[\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& 4\end{array}\right]}}=\underset{A^{-1}}{\underbrace{\left[\begin{array}{ccc}\frac{1}{2}& 0& 0\\ 0& \frac{1}{3}& 0\\ 0& 0& \frac{1}{4}\end{array}\right]}}\underset{A}{\underbrace{\left[\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& 4\end{array}\right]}}=\underset{I}{\underbrace{\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]}}.$$

Transpose

Vectors do not always have to be column vectors, that is just a convention. Swapping rows and columns is from time to time needed. The operation that swaps rows and columns is the so-called transpose. The transpose of $A$ is denoted by $A^T$. Example:

$${\left[}^{\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}}=\left[\begin{array}{cc}1& 4\\ 2& 5\\ 3& 6\end{array}\right].$$

A key feature of the transpose is that if the product $AB$ makes sense, then $B^T{A}^T$ also makes sense, at least from the point of view of sizes. In fact, we get precisely the transpose of $AB$. That is:

$${(AB)}^T=B^T{A}^T.$$

$${\left(\right[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\left]\right[\begin{array}{cc}0& 1\\ 1& 0\\ 2& -2\end{array}\left]\right)}^T=\left[\begin{array}{ccc}0& 1& 2\\ 1& 0& -2\end{array}\right]\left[\begin{array}{cc}1& 4\\ 2& 5\\ 3& 6\end{array}\right].$$

If we have a column vector $\overrightarrow{x}$ to which we apply a matrix $A$ and we transpose the result, then the row vector ${\overrightarrow{x}}^T$ applies to $A^T$ from the left:

$${(A\overrightarrow{x})}^T={\overrightarrow{x}}^T{A}^T.$$

$$\overrightarrow{x}·\overrightarrow{y}={\overrightarrow{y}}^T\overrightarrow{x}.$$

We say a matrix $A$ is symmetric if $A=A^T$. For example,

$$\left[\begin{array}{ccc}1& 2& 3\\ 2& 4& 5\\ 3& 5& 6\end{array}\right]$$

Footnotes

[1] As a side note, mathematicians write ${\overrightarrow{y}}^T\overrightarrow{x}$ and physicists write ${\overrightarrow{x}}^T\overrightarrow{y}$. Shhh…don’t tell anyone, but the physicists are probably right on this.

[2] Although so far we have not learned enough about matrices to really appreciate them.

Adapted from Differential Equations for Engineers by Jiří Lebl (Oklahoma State University), hosted on LibreTexts (math.libretexts.org) and licensed under CC BY-SA 4.0. Changes were made. License: CC-BY-SA-4.0.