L24.4 Discrete-Time Finite-State Markov Chains

Let us now abstract from our previous example and provide a general definition of what a discrete time, finite state Markov chain is. First, central in the description of a Markov process is the concept of a state, which describes the current situation of a system we are interested in. For example, in the case of the checkout counter example, the number of customers in the queue provided the right level of information needed to define a useful state. Time is assumed to be discrete, that is, divided in discrete time steps. The system starts at time 0 in an initial state, and at each successive time step, the system goes from its current state to a next one chosen with some randomness. As a result, after n such transitions, the state of the system will be random, and so we can think of it as a random variable. Let Xn be this random variable. That is, Xn represents the state in which the system is after n transitions from an initial state in which it started to operate. As a shortcut, we may often say that Xn is the state of the system at time n. We suppose that there is a finite number of possible states for the system to be in. Here, we have drawn a portion of a finite state space with m possible states labeled 1 to m, using i and j as generic labels. Of course, we could think of systems with an infinite number of states, either discrete or continuous, but this is a bit more complicated, and so in this course, we restrict ourselves to a finite state space. Note that the initial state could itself be fixed or chosen randomly. Assume now that the system started in state three. What will happen next? The system will evolve according to one of the possible transitions out of state three, for example, one of these arcs. Note here that we don't have an arc from three to four. As a convention, we only include arcs for transitions that can happen. Remember the checkout counter example. Because of our assumptions that no more than one person can join the queue at any time, we didn't have arcs of the type going from one to three or from two to ten. Also, because of the customers being served one at a time, departures were limited to one person at a time, and so no arcs of the type going from two to zero or from nine to two. So the next transition out of three can be thought of a random jump where, from state three, the system will jump to either state one, state two, state j, or jump unto itself. These will be the only possibilities. We want to describe the statistics of these jumps, and we will use conditional probabilities. Given that at time zero, the system is in state three, what is the probability that it will be in state j next? These will be called transition probabilities. For example, the probability of going from three to one will be p31. Here, p32, here, p33, and here, p3j. Note that these are the only possibilities. As a result, you have p31 plus p32 plus p33 plus p3j will be 1. Assume that the system continued to evolve, and after various different steps, come back in three at time n. Again, what will happen next? Well, this property here says that the probability of going from state three to one is again p31, the same as before. In other words, here, we will say that the chain is time homogeneous. That is, these transition probabilities will be the same irrespective of the time. So this is true for all n. And the summation that we have written here for the special case is of course general. What we have is that the probability of i to j, If you sum all of these probabilities for all possible j's, you will have one. Now, in order for this probabilistic specification to make sense and be coherent, we need to make a big assumption about the evolution of the state of the system. This assumption, the so-called Markov property, given in words here and in mathematical statement here, is in fact, the defining nature of what a Markov process is. In words, what it says is that every time the system finds itself in state three, the transition probability of going to state one will always be p31, no matter how the system evolved in the past up to being in state three. In other words, no matter what path the system has followed up to the current state, the next state transition probability will be the same, independent of that past. Mathematically, conditionally on knowing your current state, having more information about the past state variables does not change the transition probability to your next state. In other words, the probability distribution of the next state, X n+1, depends on the past only through the value of the present state, Xn. So you can see that as the definition of the transition probability and that property, that equality from here to here, being the Markov property. For this property to hold in any modeling application, you need to choose your state carefully. You want to ensure that the information contained in the description of your state captures all the relevant information to make predictions about the future evolution. Now, given a system, how to properly define the state variables which will allow us to model its evolution as a Markov process is somewhat of an art, and there are no cookbook recipes to do it. However, with a little bit of experience and practice, one quickly gets the required intuition to do this properly. You will be able to do so in that class.