Markov Chains

State-based stochastic processes governed by transition matrices, stationary distributions, and long-run convergence.

A two-state Markov chain: tomorrow depends on today

Definition

A Markov chain is a process that moves between states, where the next state depends only on the current state:

$P(X_{t+1}=j \mid X_t=i, X_{t-1},\ldots,X_0) = P(X_{t+1}=j \mid X_t=i).$

This is the Markov property. The chain remembers where it is, but not how it got there.

For example, a weather model might use two states, Sunny and Rainy. If today is Sunny, tomorrow might be Sunny with probability $0.8$ and Rainy with probability $0.2$ . If today is Rainy, tomorrow might be Sunny with probability $0.4$ and Rainy with probability $0.6$ .

A two-state chain

The weather model can be written as the transition matrix

P = \begin{pmatrix} 0.8 & 0.2 \\ 0.4 & 0.6 \end{pmatrix},

where rows are today's state and columns are tomorrow's state. If the state order is $(\text{Sunny}, \text{Rainy})$ , then the first row says Sunny $\to$ Sunny with probability $0.8$ and Sunny $\to$ Rainy with probability $0.2$ .

Try it

Why must each row of a transition matrix sum to $1$ ?

Solution

From any current state, the chain must go somewhere next. The row lists all possible next-state probabilities from that current state, so those probabilities must add to $1$ .

Related concepts

Statistics· Stochastic Processes

Random WalksProcesses built from repeated random steps, modeled with transition matrices, absorbing boundaries, and diffusion-like spread.

Statistics· Bayesian Statistics

Markov Chain Monte CarloSampling from a posterior distribution you can't write down in closed form, by building a random walk whose long-run behavior matches that distribution.

Algebra· Linear Algebra

Eigenvalues and EigenvectorsThe special directions a linear transformation merely scales — central to PCA, differential equations, and Google's PageRank.

Statistics· Probability

Law of Large NumbersAs the number of trials grows, the sample mean converges to the true population mean — the mathematical foundation of why probability works.