A general model of a system which moves from one state to state is described and applied to concrete problem. It is shown that such systems tend to a steady-state eventually.

Definition

The **transition probability**
is the probability that if the system is in state
at any one observation, it will be in state
at the next observation.

Definition

A **transition matrix**
is any square matrix with nonnegative entries, all of whose column sums are
one.

Definition

The probability vectors (column vectors of a transition matrix) for are said to be the state vectors of a Markov process if the component of is the probability that the system is in the state at the observation.

Theorem

If is the transition matrix of a Markov process and is the state vector at the observation, then

Example

A car rental agency has three rental locations, 1, 2, and 3. A customer may rent a car from any of the three locations and return the car to any of the three locations. The manager finds that the customers return the cars to the various locations according to the following probabilities:

where
stands for the probability of renting a car from location
and return it to location
Suppose a car is initially rented from location number 2.

(1) Find the state vector

(2) Predict .

We define , and

which is designed to compute

Then

So all state vectors are equal to to three decimal places.

What if we set ? We obtain that

and

Definition

A transition matrix is **regular **if some integer power of it
has all positive entries.

Theorem

If is a regular transition matrix, then as

where the
are positive numbers such that

Theorem

If is a regular transition matrix and is any probability vector, then as

where
is a fixed probability vector independent of
.

Remark

Note that if is regular, then as then for some Thus which is a fixed vector and we set it to be

Definition

Let
be an transition matrix of a Markov process. State vector
is called a stable state or steady state of the Markov process if

Remark

Example

The transition matrix . Find the steady-state vector .

**Method 1**: We compute
as we did in the previous example.

**Method 2**: If
,
then
,
which is equivalent to solve a homogeneous linear system. (We build Identity
matrix with Scientific Workplace by using ''Matrices + Fill Matrix +
Identity''.) We set
as follows:

. We solve

and
the "Solution is :