L26.8 Mean First Passage Time

In this video, we continue our exploration of quantities of interest associated with the short-term behavior of Markov chain. This time, we suppose that we have a Markov chain composed of a single recurrent class, such as this one. Here, we have our recurrent class. And these are transient states. We also assume that we're interested in a specific recurrent state-- let's say 9. So this is my state s. And we will assume that the Markov chain starts from a given initial state, state i, and assume that i is 1. Now the question we ask is, given that you started in 1, how long is it going to take to reach 9 for the first time? We know this is a recurrent class, so we know that the Markov chain eventually will come to that class and circulate here. So you know that the Markov chain will get to that state 9. You are interested in knowing when it does so for the first time. Of course, we don't know for sure. This is random. It is a random variable. And let's try to calculate the expected value of this random variable. More precisely, this is what we would like to do. We would like to find the mean first passage time from i to s, from any starting state i. Here, we're going to illustrate that with 1. And mathematically, this is what we have here. So here again, what you have is that the Markov chain started in state i, at time 0. And now you are looking at this set here. And the set records all the time n such that your Markov chain, Xn, is in the state s of interest. So out of this set, you're looking at the minimum, n, that verifies that. So this is going to be a random variable. And you take the expected value of that. And this is what we call the mean first passage time from i to s. Again, this is the expected number of steps in order to get from i to s for the first time. So how do we go about calculating such a quantity? Well, let us think a little bit. We are looking at the event that you will visit 9 from 1 for the first time. What happened after visiting 9 is of no relevance in our calculation. In other words, our calculation will never involve this arc and will not involve this arc either. Because in order for the Markov chain to traverse this arc or this one, it would have to visit 9 first. So what it means is that we can forget about this arc, and we can forget about this arc-- in a sense, that they don't matter in the calculation of the mean first passage time to s. Now, removing these arcs entirely means that you would have to increase the probability of that self transition from 0.2 to 1. So what do you get? Well, you get this following graph. We have removed these arcs, and we have increased the probability 0.2 here to 1. And again, we have argued that calculating the mean first passage time from i to s in this graph is the same as doing the same thing for this graph. But here, we have a very special structure. We have only one recurrent state left, which is state 9. And that state is an absorbing state. All the other state-- this one was transient, transient, transient. This one becomes a transient state. And this one is also a transient state in this new transition probability diagram. So we get a situation where we have one single absorbing state, and we are interested in calculating the probability starting from i to reach that absorbing state and also the number of steps it takes to do that, which is what we did in the previous video. So let us repeat what we had seen before. So this is going to be the unique solution to this system. t of s equals 0, of course. Since you start from s, you are in s. And so the amount of time it takes to get to s is 0. And otherwise, for all the other states that are not s, this is the resulting system of equation that we have. And the unique solution of this system gives you the result of finding the expected amount of time to go to the absorbing state s. So using this simple trick, we've been able to use the previous calculation to calculate something else. Let us consider a related question, which we called the mean recurrence time of s. Here, let's go back to the original graph. And this is our state s. And now the question is the following-- given that you are in s, what is the amount of time it will take for the Markov chain to return to s for the first time? So for this question, the Markov chain is currently in 9. And you ask yourself, how long is it going to take, once you leave 9, to return to 9? Here again, we don't know for sure. It's a random variable. And we are interested in the expectation of that random variable-- in other words, in the expected number of steps it takes in order to get back to 9 once you are in 9. And this is what we mean by the mean recurrence time of s. And mathematically, this is what it is. It is almost the same formula as the one that you had here, except that here you have n greater than/equal to 1 as opposed to n greater than/equal to 0. It simply means that you're not interested in the first time that you're in s, because you start from s. So you want to have n greater than or equal to 1. n equals 0 would not work. ts star would have been 0. So how would you solve that problem? Well, here again, you use the same trick that we used before. And think in terms of tree. Let's look at the Markov chain in state 9. What can happen next? From 9, it can go to 3 or it can go to 5. Or it can jump on itself. So this is with a probability 0.2. This is with a probability 0.2. And this is with a probability of 0.6. And now, after you make one transmission-- so let's-- you are in 3 now-- what is the time to get to 9? Well, it's exactly t3-- where the t is, the solution here of that previous system. What about from 5? t5. And what about from 9? Well, t9. And t9 was 0. So from that tree, what you have is t9 star will be 0.2 times t3 plus 0.6 times t5 plus 0.2 times t9 plus, of course, 1, because you have done one transition. Where, again, this value of t3, t5, and t9 are the ones corresponding to this solution here. So of course, t9 is 0. So what we have shown here, that in general, this is actually what you would have. And this is exactly what we have written here. ts star is 1 plus the summation of psj of t of j, where t of j is the solution to this system. So again, you started from 9. You have to do a transition first. You can do a transition unto yourself. You can go to 3 or you can go to 6. After that transition, which stands for the number 1 here, what happens is you're trying to find the solution to this system, which is the mean first passage time, from the current state where you are, to 9. And then when you put these two things together, you get the mean recurrence time of 9.