Editing Markov chain (section)

=== Embedded Markov chain <!-- Embedded Markov chain redirects here --> ===

One method of finding the [[stationary probability distribution]], {{pi}}, of an [[ergodic]] continuous-time Markov chain, ''Q'', is by first finding its '''embedded Markov chain (EMC)'''. Strictly speaking, the EMC is a regular discrete-time Markov chain, sometimes referred to as a '''[[jump process]]'''. Each element of the one-step transition probability matrix of the EMC, ''S'', is denoted by ''s''<sub>''ij''</sub>, and represents the [[conditional probability]] of transitioning from state ''i'' into state ''j''. These conditional probabilities may be found by

:<math>
s_{ij} = \begin{cases}
\frac{q_{ij}}{\sum_{k \neq i} q_{ik}} & \text{if } i \neq j \\
0 & \text{otherwise}.
\end{cases}
</math>

From this, ''S'' may be written as
:<math>S = I - \left( \operatorname{diag}(Q) \right)^{-1} Q</math>
where ''I'' is the [[identity matrix]] and diag(''Q'') is the [[diagonal matrix]] formed by selecting the [[main diagonal]] from the matrix ''Q'' and setting all other elements to zero.

To find the stationary probability distribution vector, we must next find <math>\varphi</math> such that
:<math>\varphi S = \varphi, </math>

with <math>\varphi</math> being a row vector, such that all elements in <math>\varphi</math> are greater than 0 and [[norm (mathematics)|<math>\|\varphi\|_1</math>]] = 1. From this, {{pi}} may be found as
:<math>\pi = {-\varphi (\operatorname{diag}(Q))^{-1} \over \left\| \varphi (\operatorname{diag}(Q))^{-1} \right\|_1}.</math>

(''S'' may be periodic, even if ''Q'' is not. Once {{pi}} is found, it must be normalized to a [[unit vector]].)

Another discrete-time process that may be derived from a continuous-time Markov chain is a δ-skeleton&mdash;the (discrete-time) Markov chain formed by observing ''X''(''t'') at intervals of δ units of time. The random variables ''X''(0),&nbsp;''X''(δ),&nbsp;''X''(2δ),&nbsp;... give the sequence of states visited by the δ-skeleton.