Pre-multiplying a vector by a matrix stretches and rotates the vector. When considering a 2-by-2 matrix (\(\mathbf{A}\)) multiplied with a 2-by-1 vector (\(\mathbf{x}\)), a transformation of the vector (\(\mathbf{y}\)) is produced that can be displayed on a 2D plane (figure).
\[\mathbf{A}\,\mathbf{x} = \mathbf{y}\]
How the matrix \(\mathbf{A}\) transforms the vector \(\mathbf{x}\) to \(\mathbf{y}\) by stretching and rotating it.
When repeated \(\mathbf{p}\) vectors are calculated and plotted, asymptotic behaviour starts to emerge. Consider the \(\mathbf{p}_i\) vectors, where \(i = \{0, 1, \ldots, 10\}\), drawn in figure with the following equations.
\[\mathbf{y}_i = \mathbf{A}\,\mathbf{x}_i \qquad \mathbf{p}_i = \mathbf{y}_i - \mathbf{x}_i \qquad \mathbf{x}_{i+1} = \mathbf{y}_i\]
The asymptotic behaviour of the \(\mathbf{p}_i\) vectors. Drag \(\mathbf{x}_0\) to see how it behaves for different starting points.
The asymptote in figure is an eigenvector. Eigenvectors pass through the origin and when the \(\mathbf{p}\) vector comes close to the eigenvector it aligns with the origin. That's why eigenvectors are known as the directions where matrices only perform scaling transformations to vectors.
This is a powerful property that is leveraged in applications like principal component analysis, Google's original PageRank, Markov chains, buckling analysis, system state-space evolution in control systems and image compression. This article will build intuition for the reader on why Eigenvalues and Eigenvectors does what it does and how to use it generally.
The Eigen Equation
Let \(\mathbf{A}\) be a square matrix. We are hunting for a non-zero vector \(\mathbf{v}\) and a scalar \(\lambda\) such that transforming \(\mathbf{v}\) by \(\mathbf{A}\) gives back the same vector, only rescaled:
\[\mathbf{A}\mathbf{v} = \lambda \mathbf{v}\]
The left-hand side, \(\mathbf{A}\mathbf{v}\), is the full transformation — it can rotate, shear and stretch. The right-hand side, \(\lambda\mathbf{v}\), is just the original vector multiplied by a number. The equation demands that, for this particular \(\mathbf{v}\), the two are identical. In other words, \(\mathbf{A}\) acting on \(\mathbf{v}\) is pure scaling: no rotation, no shear, just a change of length by the factor \(\lambda\).
To find the special directions, rewrite the eigen equation with everything on one side:
\[\mathbf{A}\mathbf{v} - \lambda\mathbf{v} = \mathbf{0} \iff (\mathbf{A} - \lambda\mathbf{I})\mathbf{v} = \mathbf{0}\]
where \(\mathbf{I}\) is the identity matrix. We need a non-zero \(\mathbf{v}\) that the matrix \((\mathbf{A} - \lambda\mathbf{I})\) crushes to zero. That can only happen if the matrix is singular — if it collapses the plane — which means its determinant vanishes:
\[\det(\mathbf{A} - \lambda\mathbf{I}) = 0\]
Applications
Eigenvectors are analogous to the orthogonal directions of a coordinate system. They are the directions in which the matrix acts independently. Figure showed one matrix \(\mathbf{A}\) transformation path. A grid of points will next be used to reveal more eigen phenomena in figure. The matrix \(\mathbf{A}\) is used to transform each location and arrows point from the start to where the matrix sends it.
The matrix \(\mathbf{A}\) acting on a grid of points. Each arrow carries a point to its image \(\mathbf{A}\mathbf{v}\). The two highlighted lines are the eigen-directions; the arrows that begin on them stay collinear with the line — pure scaling. The faint dashed loop is the unit circle and the heavier loop is its image, an ellipse, showing how the whole plane is squashed and stretched.
The vectors close to the eigenvectors align asymptotically with the eigenvector lines, pointing inward (to the origin, shrinking) or outward (stretching). The eigenvalues can also be seen as the \(\lambda\) values. Eigenvalues smaller than 1 result in shrinking eigenvector directions while \(\lambda > 1\) result in stretching directions.
Each eigenvector direction encodes unique, independent, behaviour. Extracting them unlocks major advantages. First, eigenpairs describe the contribution of each behaviour and how it acts on each dimension of the system.
All eigenpairs can fully describe a system and one can convert back and forth between system dynamics and its eigen representation. What's more, one can leave out behaviours with poor contributions when converting back to the actual system. This is how image and video compression is done.
Mode Shapes
Consider a spring system with two nodes, \(x_1\) and \(x_2\) on which forces \(f_1\) and \(f_2\) can be applied.
Spring system with two nodes connected between 3 springs, constrained between fixed anchors.
The stiffness matrix is:
The eigenvalues and eigenvectors are:
mode 1
eigenvalue \(\lambda_1\):
eigenvector \(\mathbf{v}_1\):
mode 2
eigenvalue \(\lambda_2\):
eigenvector \(\mathbf{v}_2\):
The matrix \(\mathbf{K}\) acting on a grid of points.
The system has two unknowns (\(x_1\) and \(x_2\)), two dimensions it can move in, results in a 2-by-2 matrix and has two eigenpairs. The two eigenvectors are the two mode shapes the system has and can be seen in figure.
The mode shapes.
The interesting observation is that every deformation of the system can be encoded by adding a portion of each mode shape.
\[\mathbf{x} = c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2\]
A last note is that, when looking at strain energy:
\[U = \tfrac{1}{2}\,\mathbf{x}^{\mathsf{T}} \mathbf{K}\, \mathbf{x}\]
It is always the smallest when \(\mathbf{x}\) lies on \(\mathbf{v}_1\) and the largest when on \(\mathbf{v}_2\) for the same length \(\mathbf{x}\) (\(\lVert \mathbf{x} \rVert\)). The eigenvector associated with the smallest eigenvalue gives the minimum strain energy while the largest eigenvalue gives the maximum strain energy. Therefore, for a perturbation or a stretch-and-release action on a system, the high energy demand (high eigenvalue) mode shape has the lowest contribution to how the system reacts. Ignoring them can still provide good approximations of system behaviour.
Primary Component Analysis
Consider a data set of 20 countries and we are looking at:
- GDP per capita (USD/people)
- Population density (people/km²)
- Urbanization (%)
- Distance from equator (km)
- HDI (index from 0 to 1)
Principal Component Analysis, or PCA, is a method for finding the main independent directions of variation in a dataset. It illuminates new axes called principal components. These axes are weighted combinations of the original criteria. PC1 is the direction where the data varies the most. PC2 is the next strongest independent direction. Together, PC1 and PC2 often give a useful map of the design space.
PCA usually starts with a z-scoring step to normalise numbers with different scales and units. If data is normalised, a covariance matrix is created with
\[\mathrm{Cov}(x,y) = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})\]
Vectors G, P, U, D and H are the columns of the data (the criteria). Computing the eigenvalues and eigenvectors of the covariance matrix reveals the dominance of PCs and their directions. The percentage of an eigenvalue to the sum of all eigenvalues results in a contribution percentage of each PC. For this example the percentages are PC1: 60.4%, PC2: 20.7%, PC3: 10.2%, PC4: 6.6%, PC5: 2.0%. Looking at only the first two components explain the system with 81.1% accuracy, a major simplification for relatively good accuracy.
The eigenvectors are referred to as the PC loadings. They are the significant independent directions in a multidimensional cloud of data and show how moving in this direction affects each criteria. The loading weights for the example are:
| Variable | PC1 | PC2 | PC3 | PC4 | PC5 |
|---|---|---|---|---|---|
| GDP per capita | 0.492 | 0.234 | −0.353 | −0.631 | −0.426 |
| Population density | −0.248 | 0.823 | −0.380 | 0.340 | −0.023 |
| Urbanization % | 0.476 | −0.301 | −0.339 | 0.675 | −0.334 |
| Distance from equator | 0.412 | 0.402 | 0.772 | 0.175 | −0.203 |
| HDI | 0.548 | 0.122 | −0.142 | 0.001 | 0.815 |
PC loadings (eigenvectors)
PC1 shows that there is a good correlation between GDP per capita, urbanization %, HDI and the distance from the equator while there is an inverse correlation of population density. It is a primary vector through a complex multidimensional data set that separates highly developed, wealthy, urbanized countries from lower-income / lower-HDI countries. If this analysis was used to test a hypothesis that the country's development level correlates with its distance from the equator, this is a good substantiation.
PC2 separates dense countries from low-density countries, with some contribution from latitude. It is dominated by population density, distance from the equator and urbanization %, showing that more populated countries are not necessarily more urbanised. While PC1 is a country development axis, PC2 is a population-distribution axis. The benefit of having only two axes describing 81.1% of the data is that a simple scatter plot can be used to communicate most of the information of a very complex dataset.
Country PCA scores.
Computing Eigenpairs
Continuing from the equation
\[\det(\mathbf{A} - \lambda\mathbf{I}) = 0\]
For a \(2\times2\) matrix this expands into a tidy quadratic, the characteristic equation:
\[ \begin{align*} \det\begin{bmatrix} a-\lambda & b \\ c & d-\lambda \end{bmatrix} &= (a-\lambda)(d-\lambda) - bc \\ &= \lambda^2 - (a+d)\,\lambda + (ad - bc) \\ &= 0 \end{align*}\]
The coefficients are two familiar quantities: the trace \(T = a + d\) (the sum down the diagonal) and the determinant \(D = ad - bc\). So the eigenvalues are simply the roots of
\[\lambda^2 - T\lambda + D = 0 \quad\Longrightarrow\quad \lambda = \frac{T \pm \sqrt{T^2 - 4D}}{2}\]
trace \(T = a + d\):
determinant \(D = ad - bc\):
discriminant \(T^2 - 4D\):
eigenvalue \(\lambda_1\):
eigenvalue \(\lambda_2\):
With each \(\lambda\) in hand, the matching eigenvector comes from the top row of \((\mathbf{A} - \lambda\mathbf{I})\mathbf{v} = \mathbf{0}\), namely \((a-\lambda)v_x + b\,v_y = 0\). One solution is \(\mathbf{v} = (b,\ \lambda - a)\), which we then normalise to unit length:
eigenvector \(\mathbf{v}_1\):
eigenvector \(\mathbf{v}_2\):
When the discriminant \(T^2 - 4D\) is negative, the square root is imaginary and the eigenvalues are complex. Geometrically the matrix is rotating the plane, so no real direction is left pointing the same way — there are no real eigen-directions to draw.
Conclusion
Types of matrices deserve a re/visit in the context of eigenvalues and eigenvectors. The identity matrix is an extreme case since it does nothing to a vector, thus \(\mathbf{A}\mathbf{v} = \mathbf{I}\mathbf{v} = 1\mathbf{v}\). Every vector \(\mathbf{v}\) is an eigenvector but there is only one eigenvalue \(\lambda = 1\). Similarly, a matrix with 2 on the diagonal and zero everywhere else (\(\mathbf{A} = 2\mathbf{I}\)) purely scales everything by 2. Every vector is still an eigenvector and there is only one eigenvalue \(\lambda = 2\).
If the matrix is diagonal (just like \(\mathbf{I}\) and \(2\mathbf{I}\)) but have different values on the diagonal, \(n\) eigenpairs suddenly emerge for an \(n\)-by-\(n\) matrix. The eigenvalues are the values on the diagonal and the eigenvectors point in the directions of the matrix dimensions. When the matrix has off-diagonal entries but is still symmetric \(\mathbf{A} = \mathbf{A}^{\mathsf{T}}\), it has \(n\) eigenvectors that are perpendicular. If a matrix is singular, \(\det(\mathbf{A}) = 0\), at least one eigenvalue is zero.
Eigenvalues and eigenvectors strip a matrix down to its essence. Buried inside all the rotation and shear are a few skeleton directions along which the matrix is nothing more than a stretch, and a set of numbers saying by how much. Once those are known, the matrix becomes easy to reason about — repeated application, stability, resonance and principal axes all fall out of the eigenpairs. The humble equation \(\mathbf{A}\mathbf{v} = \lambda\mathbf{v}\) turns out to be one of the most useful sentences in all of applied mathematics.