Eigenvalues and Eigenvectors
Question: What are eigenvalues and eigenvectors?
Introduction
Let us consider the following linear transformation from \(\mathbb{R}^{2}\) to itself:
\[ \mathbf{T} =\begin{bmatrix} 3 & 1\\ 0 & 2 \end{bmatrix} \]
We will look at how this linear transformation operates on vectors from the geometric point of view. Specifically, we will record these two quantities:
direction of the vector before the transformation
direction of the vector after the transformation
Case-1
The vector \(\mathbf{u} =\begin{bmatrix} 1\\ 1 \end{bmatrix}\) is transformed into \(\mathbf{Tu} =\begin{bmatrix} 4\\ 2 \end{bmatrix}\).
The vector \(\mathbf{u}\) was initially pointing in a particular direction. After the transformation, it points in a different direction. This case corresponds to vectors whose direction changes after the linear transformation.
Case-2
The vector \(\mathbf{u} =\begin{bmatrix} -1\\ 1 \end{bmatrix}\) is transformed into \(\mathbf{Tu} =\begin{bmatrix} -2\\ 2 \end{bmatrix}\).
The direction of the vector \(\mathbf{Tu}\) is the same as the direction of the vector \(\mathbf{u}\). In other words, the linear transformation has preserved the direction of this vector. However, its magnitude has changed. In this case, the vector \(\mathbf{u}\) has been stretched by a factor of \(2\).
\[ \mathbf{Tu} =\begin{bmatrix} -2\\ 2 \end{bmatrix} =2\mathbf{u} \]
Another example for this case is the basis vector \(\begin{bmatrix} 1\\ 0 \end{bmatrix}\). For this vector:
\[ \mathbf{Tu} =\begin{bmatrix} 3\\ 0 \end{bmatrix} =3\mathbf{u} \]
Visually:
This case corresponds to vectors whose direction remains unchanged after the linear transformation.
What we mean by direction here is a line passing through the origin. Each vector \(\displaystyle \mathbf{u}\) specifies a direction. Every scalar multiple of this vector, \(\displaystyle k\mathbf{u}\), can be said to be in the same direction as \(\displaystyle \mathbf{u}\). Note that this holds even if \(\displaystyle k\) is negative or zero.
Eigenvectors and Eigenvalues
A non-zero vector which points in the same direction before and after the linear transformation is called an eigenvector. Since the direction of an eigenvector is unchanged by the transformation, it makes sense to look at the amount by which it is scaled after the transformation. This scalar value is called the eigenvalue. Let us now get back to the matrix that we have been working with:
\[ \mathbf{A} = \begin{bmatrix} 3 & 1\\ 0 & 2 \end{bmatrix} \]
We can state the following:
\(\begin{bmatrix}-1\\1\end{bmatrix}\) is an eigenvector of \(\mathbf{A}\) with eigenvalue \(2\)
\(\begin{bmatrix}1\\0\end{bmatrix}\) is an eigenvector of \(\mathbf{A}\) with eigenvalue \(3\)
The eigenvector and the corresponding eigenvalue make up an eigenpair. \(\left( 2,\begin{bmatrix} -1\\ 1 \end{bmatrix}\right)\) and \(\left( 3,\begin{bmatrix} 1\\ 0 \end{bmatrix}\right)\) are two eigenpairs of \(\mathbf{A}\). Let us now state our observations more formally:
Consider a matrix \(\mathbf{A}\) of shape \(n \times n\). A non-zero vector \(\mathbf{v}\) is called an eigenvector of \(\mathbf{A}\) with eigenvalue \(\lambda\) if \(\mathbf{A v} = \lambda\mathbf{v}\).
A few important points to note:
- Eigenvectors are non-zero vectors. \(\mathbf{0}\) can never be an eigenvector. This is because if \(\mathbf{A0} =\lambda\mathbf{0}\) then there is no fixed \(\lambda\) that we can associate with \(\mathbf{0}\).
- Eigenvectors are only defined for square matrices. Note that we are looking at vectors in \(\mathbb{R}^{n}\) whose direction is preserved under a linear transformation. This only makes sense if we look at maps that take a vector from a space and put it back in the same space.
- Eigenvectors are not unique. For the same eigenvalue, we have infinitely many eigenvectors. This will become apparent when we discuss eigenspaces next.
Finally, we shall look at one more way of understanding eigenvalues and eigenvectors. In the case of a system of linear equations, we were concerned with the system:
\[ \mathbf{Ax} = \mathbf{b} \]
In the study of eigenvalues and eigenvectors, our concern will be the following system:
\[ \mathbf{Ax} = \lambda\mathbf{x} \]
where we must always remember that \(\mathbf{A}\) is a square matrix. In studying \(\mathbf{Ax} = \mathbf{b}\), our goal was to solve for \(\mathbf{x}\). In studying \(\mathbf{Ax} = \lambda\mathbf{x}\), our goal is to identify the set of all eigenpairs that satisfy this equation.
Eigenspace
Consider an arbitrary \(n\times n\) matrix \(\mathbf{A}\). How big is the space of eigenvectors? If \(\mathbf{u}\) is an eigenvector of \(\mathbf{A}\) with eigenvalue \(\lambda\), what can we say about the vector \(2\mathbf{u}\)?
\[ \mathbf{A} (2\mathbf{u} )=2\mathbf{Au} =2\lambda\mathbf{u} =\lambda(2\mathbf{u} ) \]
We see that \(2\mathbf{u}\) is also an eigenvector with eigenvalue \(\lambda\). In fact, \(k\mathbf{u}\) is an eigenvector for every non-zero \(k\). Coming from another direction, let \(\mathbf{u}\) and \(\mathbf{v}\) be two eigenvectors for the same eigenvalue \(\lambda\). Then:
\[ \mathbf{A} (\mathbf{u} +\mathbf{v} )=\mathbf{Au} +\mathbf{Av} =\lambda\mathbf{u} +\lambda\mathbf{v} =\lambda(\mathbf{u} +\mathbf{v} ) \]
Therefore, \(\mathbf{u} +\mathbf{v}\) is also an eigenvector of \(\mathbf{A}\) with eigenvalue of \(\lambda\). From these two observations, we see that the set of all eigenvectors with eigenvalue \(\lambda\), along with the zero vector, is a subspace of \(\mathbb{R}^{n}\). Such a subspace is called an eigenspace. Formally:
Given a square matrix \(\mathbf{A}\) with eigenvalue \(\lambda\), the eigenspace \(E(\lambda)\) is defined as:
\[ E(\lambda) = \{\mathbf{x}\ :\ \mathbf{Ax} = \lambda \mathbf{x},\ \mathbf{x} \in \mathbb{R}^{n}\} \]
Each non-zero vector of \(E(\lambda)\) is an eigenvector of \(\mathbf{A}\) with eigenvalue \(\lambda\). Additionally, \(E(\lambda)\) is a subspace of \(\mathbb{R}^{n}\).
Eigenspaces are also called invariant subspaces under the transformation \(\mathbf{A}\). That is, these subspaces are not distrubed by the transforamtion since \(\mathbf{A}(E(\lambda)) = E(\lambda)\).
Summary
For a matrix \(\mathbf{A}\), a non-zero vector \(\mathbf{x}\) is an eigenvector with eigenvalue \(\lambda\) if \(\mathbf{Ax} =\lambda\mathbf{x}\). Eigenvectors are those directions of \(\displaystyle \mathbb{R}^{n}\) that are preserved under the linear transformation. The eigenspace \(E(\lambda)\) is the set of all eigenvectors corresponding to \(\lambda\), along with the zero vector, and is a subspace of \(\mathbb{R}^{n}\).