Computing Eigenvectors
How do we find the eigenvectors of a matrix?
Eigenspace
Once we have the eigenvalues of a matrix, the next logical step is to find the eigenvectors for each of these eigenvalues. Recall that the set of all eigenvectors for a given eigenvalue \(\lambda\) of a matrix \(\mathbf{A}\) belongs to a subspace, \(E(\lambda)\), called the eigenspace of \(\mathbf{A}\) associated with the eigenvalue \(\lambda\):
\[ E(\lambda) = \{\mathbf{x}\ :\ \mathbf{A x} = \lambda \mathbf{x},\ \mathbf{x} \in \mathbb{R}^{n}\} \]
We also know that \(\mathbf{A x} = \lambda \mathbf{x}\) if and only if \((\mathbf{A} - \lambda \mathbf{I}) \mathbf{x} = \mathbf{0}\). From this, it becomes clear that:
\[ E(\lambda) = \mathcal{N}(\mathbf{A} - \lambda \mathbf{I}) \]
Thus, finding the eigenspace is equivalent to finding the nullspace of \(\mathbf{A} - \lambda \mathbf{I}\), which is in turn equivalent to solving a set of homegenous linear equations. Once we find a basis for the nullspace, we can represent \(E(\lambda)\) as the span of this basis.
Examples
Let us continue with the examples that we have been working with so far to demonstrate all this.
Example-1
\[ \mathbf{A} =\begin{bmatrix} 3 & 1\\ 0 & 2 \end{bmatrix} \]
We already know that the eigenvalues are \(2\) and \(3\). What are the eigenvectors corresponding to \(\lambda =2\)? First, we note that if \(\mathbf{x}\) is an eigenvector corresponding to \(\lambda =2\), then:
\[ \begin{aligned} (\mathbf{A} -2\mathbf{I} )\mathbf{x} & =0\\ & \\ \begin{bmatrix} \mathbf{1} & 1\\ 0 & 0 \end{bmatrix}\begin{bmatrix} x_{1}\\ x_{2} \end{bmatrix} & =\begin{bmatrix} 0\\ 0 \end{bmatrix} \end{aligned} \]
From this, we have \(x_{1} +x_{2} =0\). \(x_{2}\) is a free variable and \(x_{1}\) is a pivot variable variable. The rank of this matrix is \(1\), so every eigenvector is of the form:
\[ k\begin{bmatrix} -1\\ 1 \end{bmatrix} \]
Therefore, the eigenspace corresponding to the eigenvalue \(\lambda =2\) is given by:
\[ E( 2) =\text{span}\left\{\begin{bmatrix} -1\\ 1 \end{bmatrix}\right\} \]
If we repeat the same process for \(\lambda =3\), the eigenspace turns out to be:
\[ E( 3) =\text{span}\left\{\begin{bmatrix} 1\\ 0 \end{bmatrix}\right\} \]
What have we done so far? We have just computed the nullspace of \(\mathbf{A} -\lambda \mathbf{I}\). This is the recipe to follow: compute a basis for the nullspace of the matrix \(\mathbf{A} -\lambda \mathbf{I}\). The span of this basis (excluding the zero vector) is the set of all eigenvectors of the matrix \(\mathbf{A}\) corresponding to the eigenvalue \(\lambda\). We have already discussed an algorithm to compute the basis for the nullspace of a matrix.
For any matrix \(\mathbf{A}\), the nullspace of \(\mathbf{A}\) (excluding the vector \(\mathbf{0}\)) has all the eigenvectors of \(\mathbf{A}\) with eigenvalue \(0\). This is because \(\mathbf{Ax} =\mathbf{0} =0\mathbf{x}\) if \(\mathbf{x} \in \mathcal{N} (\mathbf{A} )\).
Example-2
Let us take one more example from the previous document.
\[ \mathbf{A} =\begin{bmatrix} 1 & 3 & 3\\ 3 & 1 & -1\\ 0 & 0 & 2 \end{bmatrix} \]
We have already computed the eigenvalues to be \(\displaystyle -2,2,4\). Let us now compute the eigenvectors corresponding to \(\displaystyle \lambda =-2\). We have to solve \(\displaystyle (\mathbf{A} +2\mathbf{I}) =\mathbf{0}\). This is the system:
\[ \begin{bmatrix} 3 & 3 & 3\\ 3 & 3 & -1\\ 0 & 0 & 4 \end{bmatrix}\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3} \end{bmatrix} =\begin{bmatrix} 0\\ 0\\ 0 \end{bmatrix} \]
Now, we go for row reduction:
\[ \begin{bmatrix} 3 & 3 & 3\\ 3 & 3 & -1\\ 0 & 0 & 4 \end{bmatrix}\xrightarrow{R_{2}\rightarrow R_{2} -R_{1}}\begin{bmatrix} 3 & 3 & 3\\ 0 & 0 & -4\\ 0 & 0 & 4 \end{bmatrix} \]
\[ \begin{bmatrix} 3 & 3 & 3\\ 0 & 0 & -4\\ 0 & 0 & 4 \end{bmatrix}\xrightarrow{R_{3}\rightarrow R_{3} +R_{2}}\begin{bmatrix} 3 & 3 & 3\\ 0 & 0 & -4\\ 0 & 0 & 0 \end{bmatrix} \]
\[ \begin{bmatrix} 3 & 3 & 3\\ 0 & 0 & -4\\ 0 & 0 & 0 \end{bmatrix}\xrightarrow{R_{2}\rightarrow \frac{-R_{2}}{4}}\begin{bmatrix} 3 & 3 & 3\\ 0 & 0 & 1\\ 0 & 0 & 0 \end{bmatrix} \]
\[ \begin{bmatrix} 3 & 3 & 3\\ 0 & 0 & -4\\ 0 & 0 & 0 \end{bmatrix}\xrightarrow{R_{1}\rightarrow \frac{R_{1}}{3}}\begin{bmatrix} 1 & 1 & 1\\ 0 & 0 & 1\\ 0 & 0 & 0 \end{bmatrix} \]
\[ \begin{bmatrix} 1 & 1 & 1\\ 0 & 0 & 1\\ 0 & 0 & 0 \end{bmatrix}\xrightarrow{R_{1}\rightarrow R_{2} -R_{2}}\begin{bmatrix} \mathbf{1} & 1 & 0\\ 0 & 0 & \mathbf{1}\\ 0 & 0 & 0 \end{bmatrix} \]
Columns \(\displaystyle 1\) and \(\displaystyle 3\) are pivot columns. \(\displaystyle x_{1}\) and \(\displaystyle x_{3}\) are pivot variables while \(\displaystyle x_{2}\) is a free variable. Therefore, setting \(\displaystyle x_{2} =1\), we get \(\displaystyle x_{1} =-1,x_{3} =0\). This gives us \(\displaystyle \begin{bmatrix} -1\\ 1\\ 0 \end{bmatrix}\) as an eigenvector corresponding to eigenvalue \(\displaystyle -2\). Therefore, the eigenspace is given by:
\[ E( -2) =\text{span}\left\{\begin{bmatrix} -1\\ 1\\ 0 \end{bmatrix}\right\} \]
In both the examples that we have discussed, the eigenspace turned out to be one-dimensional. This is not necessarily the case always. The eigenspace could have any dimension between \(0\) and \(n\) and it depends on the matrix.
Special examples
Let us take up some special examples and see what the eigenvectors and eigenvalues are for these matrices.
Diagonal matrix
If \(\mathbf{D}\) is a diagonal matrix, then the eigenvalues are the diagonal entries of the matrix. If we express \(\mathbf{D}\) as:
\[ \mathbf{D} =\text{diag} (\lambda _{1} ,\cdots ,\lambda _{n} ) \]
Then the characteristic polynomial is \((\lambda _{1} - \lambda )\cdots (\lambda_{n} -\lambda )\). The eigenvectors of \(\mathbf{D}\) are going to be elements of the standard basis \(\{\mathbf{e}_{1} ,\cdots ,\mathbf{e}_{n} \}\). To see why, observe that:
\[ \mathbf{D e}_{i} = \lambda_i \mathbf{e}_i \]
Diagonal matrices are the most interesting when it comes to the study of eigenvalues and eigenvectors because of the ease with which we can directly read off the eigenvalues and the eigenvectors. We will have more to say about diagonal matrices in the upcoming documents.
Upper Triangular matrix
Recall that an upper triangular matrix is one in which all entries below the main diagonal are zero. As an example:
\[ \mathbf{A} =\begin{bmatrix} 1 & 3 & 2\\ 0 & 2 & 1\\ 0 & 0 & 3 \end{bmatrix} \]
The eigenvalues of an upper triangular matrix can also be directly read off from the diagonal. To see why this is true, expanding the determinants along the first column would give us:
\[ ( 1-\lambda )( 2-\lambda )( 3-\lambda ) =0 \]
The eigenvector corresponding to the first eigenvalue is easy enough to compute: it is \(\mathbf{e}_1\). However, the remaining eigenvectors are not so easy to compute. We would have to go back and solve \(\displaystyle (\mathbf{A} -\lambda \mathbf{I})\mathbf{x} =\mathbf{0}\).
Identity matrix
The characteristic polynomial for the identity matrix, \(\mathbf{I}\) of shape \(n \times n\), is of the form \((1-\lambda )^{n}\). There is exactly one eigenvalue that is repeated \(n\) times. This is en extreme case of the diagonal matrix. Every vector in \(\mathbb{R}^{n}\) is an eigenvector with eigenvalue \(1\). The eigenspace is \(\mathbb{R}^{n}\). To see why, observe that for all \(\mathbf{x} \in \mathbb{R}^{n}\):
\[ \mathbf{I x} = \mathbf{x} \implies \mathbf{I x} = 1 \cdot \mathbf{x} \]
Projection matrix
If \(\mathbf{P}\) is a projection matrix that projects vectors in \(\mathbb{R}^{n}\) onto a subspace \(S\), then every vector in this subspace is an eigenvector with eigenvalue \(1\). Recall that vectors in the subspace are left undistrubed by the projection matrix. If \(\mathbf{x} \in S\), then:
\[ \mathbf{P x} = \mathbf{x} \implies \mathbf{P x} = 1 \cdot \mathbf{x} \]
Also, every vector that is orthogonal to the subspace \(S\) is an eigenvector of \(\mathbf{P}\) with eigenvalue \(0\). This is because if \(\mathbf{x} \in S^{\perp}\), then:
\[ \mathbf{Px} =\mathbf{0} \implies \mathbf{Px} = 0 \cdot \mathbf{x} \]
where, \(S^{\perp}\) is the set of all vectors perpendicular to \(S\). In fact, one can show that \(0\) and \(1\) are the only eigenvalues of a projection matrix.
Summary
To find the eigenvectors corresponding to eigenvalue \(\lambda\), we solve the homogenous system \((\mathbf{A} - \lambda \mathbf{I}) \mathbf{x} = \mathbf{0}\). From this, we can get a basis for the nullspace, \(\mathcal{N}(\mathbf{A} - \lambda \mathbf{I})\), and the span of this basis is the eigenspace \(E(\lambda)\).