L24.4 Discrete-Time Finite-State Markov Chains
Let us now abstract from our previous example and provide a general definition of what a discrete time, finite state Markov chain is. First, central in the description of a Markov process is the concept of a state, which describes the current situation of a system we are interested in. For example, in the case of the checkout counter example, the number of customers in the queue provided the right level of information needed to define a useful state. Time is assumed to be discrete, that is, divided in discrete time steps. The system starts at time 0 in an initial state, and at each successive time step, the system goes from its current state to a next one chosen with some randomness. As a result, after n such transitions, the state of the system will be random, and so we can think of it as a random variable. Let Xn be this random variable. That is, Xn represents the state in which the system is after n transitions from an initial state in which it started to operate. As a...