Eigenvectors and Diagonalizing Matrices

E. L. Lady

Let A be an n × n matrix and suppose there exists a basis v_{1}, …, v_{n} for ℝ^{n} such that for each i, Av_{i} = λ_{i}v_{i} for some scalar λ. (I. e. v_{i} is an eigenvector for A corresponding to the eigenvalue λ_{i}.)

Let P be the n × n matrix whose columns are the basis vectors v_{1}, …, v_{n}, i. e. P = [v_{1} v_{2} … v_{n}]. The fact that the columns of P are a basis for ℝ^{n} implies that the column space of P is all of ℝ^{n}, so rank P = n and therefore P is non-singular.

Now notice that P^{−1}v_{i} is the ith column of P^{−1}P, i. e. P^{−1}v_{i} = e_{i}, where e_{i} is the ith column of the identity matrix I_{n}.

Theorem. P^{−1}AP is the diagonal matrix with λ_{1}, …, λ_{n} on the diagonal.

proof:

P^{−1}AP = P^{−1}A [ v_{1} … v_{n}]

= P^{−1} [ Av_{1} … Av_{n}]

= P^{−1} [ λ_{1}v_{1} … λ_{n}v_{n}]

= [ λ_{1}P^{−1}v_{1} … λ_{n}P^{−1}v_{n}]

= [ λ_{1}e_{1} … λ_{n}e_{n}]

= [ \begin{matrix}λ_{1} & 0 & … & 0 \\ 0 & λ_{2} & … & 0 \\ ⋮ & ⋮ & & ⋮ \\ 0 & 0 & … & λ_{n} \end{matrix} ] .

Example. A = [ \begin{matrix}19 & 20 & −16 \\ 20 & 13 & 4 \\ −16 & 4 & 31 \end{matrix} ], and the v_{i} are [ \begin{matrix}1 \\ 2 \\ 2 \end{matrix} ], [ \begin{matrix}−2 \\ −1 \\ 2 \end{matrix} ], [ \begin{matrix}2 \\ −2 \\ 1 \end{matrix} ]. Then a

calculation shows that Av_{i} = λ_{i}v_{i}, where λ_{1} = 27, λ_{2} = 45, and λ_{3} = −9. Now let

P = [ \begin{matrix}1 & −2 & 2 \\ 2 & −1 & −2 \\ 2 & 2 & 1 \end{matrix} ]. Then P^{−1} [ \begin{matrix}1 & −2 & 2 \\ 2 & −1 & −2 \\ 2 & 2 & 1 \end{matrix} ] = P^{−1}P = [ \begin{matrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} ]. Thus

P^{−1}AP = P^{−1}A [ \begin{matrix}1 & −2 & 2 \\ 2 & −1 & −2 \\ 2 & 2 & 1 \end{matrix} ] = P^{−1} [ \begin{matrix}λ_{1} & −2λ_{2} & 2λ_{3} \\ 2λ_{1} & −λ_{2} & −2λ_{3} \\ 2λ_{1} & 2λ_{2} & λ_{3} \end{matrix} ] = [ \begin{matrix}λ_{1} & 0 & 0 \\ 0 & λ_{2} & 0 \\ 0 & 0 & λ_{3} \end{matrix} ].

(For the record, P^{−1} = \frac{1}{9} [ \begin{matrix}1 & 2 & 2 \\ −2 & −1 & 2 \\ 2 & −2 & 1 \end{matrix} ] . If you need to use this to follow the

calculation, then you’ve missed the point.)

Another interesting thing about the eigenvectors given above is that they are mutually orthogonal (perpendicular) to each other, as you can easily verify by computing the dot products. This is not something that is universally true for eignvectors, but it is also not an accident in this case.

The matrix

B = [ \begin{matrix}23 & 6 & 25 \\ 5 & 4 & 5 \\ −20 & −6 & −22 \end{matrix} ]

has three distinct eigenvalues, namely 3, 4, and −2. A set of eigenvectors for these is

[ \begin{matrix}1 \\ 5 \\ −2 \end{matrix} ] ,  [ \begin{matrix}1 \\ 1 \\ −1 \end{matrix} ] ,  [ \begin{matrix}−1 \\ 0 \\ 1 \end{matrix} ] .

Taking dot products shows that these three vectors are not orthogonal to each other. However, they are linearly independent.

The rule is as follows:

Theorem. (1) If v_{1}, …, v_{r} are eigenvectors for a matrix A and if the corresponding eigenvalues are all different, then v_{1}, …, v_{r} must be linearly independent.

(2) If the n × n matrix A is symmetric then eigenvectors corresponding to different eigenvalues must be orthogonal to each other. Furthermore, in this case there will exist n linearly independent eigenvectors for A, so that A will be diagonalizable.

(3) In the case of a symmetric matrix, the n different eigenvectors will not necessarily all correspond to different eigenvalues, so they may not automatically be orthogonal to each other. However (if the entries in A are all real numbers, as is always the case in this course), it’s always possible to find some set of n eigenvectors which are mutually orthogonal.

The reason why eigenvectors corresponding to distinct eigenvalues of a symmetric matrix must be orthogonal is actually quite simple. In fact, it is a special case of the following fact:

Proposition. Let A be any n × n matrix. If v is an eigenvector for A^{T} and if w is an eigenvector for A, and if the corresponding eigenvalues are different, then v and w must be orthogonal.

Of course in the case of a symmetric matrix, A^{T} = A, so this says that eigenvectors for A corresponding to different eigenvalues must be orthogonal.

This proposition is the result of a Lemma which is an easy exercise in summation notation. (There’s also a very fast slick proof.)

Lemma. If A is an n × n matrix and v and w are vectors in ℝ^{n}, then the dot product of v and Aw is the same as the dot product of A^{T} v and w.

Now, taking this Lemma for granted, suppose that v is an eigenvector for A^{T} and w is an eigenvector for A, and suppose that the eigenvalues are different. To be specific, let’s suppose that the two eigenvalues are 2 and 5, respectively. Then from the Lemma we get

2v · w = (A^{T} v) · w = v · (Aw) = v · 5w = 5v·w.

But since 2 and 5 are scalars (and, for that matter, so is v · w), the only way that 2v · w = 5v · w can be possible is for v · w to be 0.

Returning to the original example (at the top of p. 2), the fact that the eigenvectors of A are orthogonal to each other means that the columns of the matrix P are orthogonal to each other. And it’s very easy to see that a consequence of this is that the product P^{T} P is a diagonal matrix. In fact,

P^{T} P == [ \begin{matrix}1 & 2 & 2 \\ −2 & −1 & 2 \\ 2 & −2 & 1 \end{matrix} ] [ \begin{matrix}1 & −2 & 2 \\ 2 & −1 & −2 \\ 2 & 2 & 1 \end{matrix} ] = [ \begin{matrix}9 & 0 & 0 \\ 0 & 9 & 0 \\ 0 & 0 & 9 \end{matrix} ] .

(The entries on the diagonal are the squares of the magnitudes of the columns

— in this case, all 9.) This is more than just a curiosity, for one can also write \frac{1}{9}P^{T} P = I_{3} , showing that \frac{1}{9}P^{T} = P^{−1} . In general,

If the columns of a matrix are orthogonal to each other and non-zero, then the inverse can be obtained from the transpose by dividing each row of the transpose by the square of its magnitude.

Since it’s usually a lot easy to take the transpose of a matrix than to compute the inverse, this is certainly a useful fact. But one can do even better.

Remember that when it comes to eigenvectors, the magnitude is not really that relevant. What counts is the direction of the vector. So it’s permissible to “unitize” a set of eigenvectors and get new eigenvectors all having magnitude 1.

For instance, in the original example above, all the eigenvectors originally given have magnitude 3 (as one can easily check). One can get a new set of eigenvectors

v′_{1} = [ \begin{matrix}1/3 \\ 2/3 \\ 2/3 \end{matrix} ] ,  v′_{2} = [ \begin{matrix}−2/3 \\ −1/3 \\ 2/3 \end{matrix} ] ,  v′_{3} = [ \begin{matrix}2/3 \\ −2/3 \\ 1/3 \end{matrix} ]

all with magnitude 1. If one makes these vectors into the columns of a matrix, one gets

P_{1} = [ \begin{matrix}1/3 & −2/3 & 2/3 \\ 2/3 & −1/3 & −2/3 \\ 2/3 & 2/3 & 1/3 \end{matrix} ]

with the property that P_{1}^{T} P_{1} = I, which shows that P_{1}^{T} = P_{1}^{−1} .

Definition. A n × n matrix whose inverse is the same as its transpose is called an orthogonal matrix.

In order to be orthogonal, it is necessary that the columns of a matrix be orthogonal to each other. But it is also necessary that all the columns have magnitude 1.

Exercise (Easy!). If P is an orthogonal matrix, then the rows of P are also orthogonal to each other and all have magnitude 1.

Putting all these pieces together (including some parts that were not actually proved), we get the following:

Theorem. If A is a symmetric n × n matrix whose entries are all real numbers, then there exists an orthogonal matrix P such that P^{T} AP is a diagonal matrix.