L26.7 Expected Time to Absorption

In this video, let us look at a second quantity of interest that has to do with absorbing states. Now that we know how to calculate the probability of getting to a given absorbing state, we would like to know how long it would take to get to it. Let us first deal with that question when we have only one absorbing state. Let us consider the following Markov chain, which is a little simpler than the one that we had in the previous video. We have transient states. One, two, and three are transient states. And we have one recurrent state, four. And that recurrent state four is an absorbing state, because once you get to it you stay there forever. So in this simple example, the absorption probability to four is trivially one. No matter where you start, with probability one you're guaranteed that eventually you will reach four. But the question of interest is to know how long would it take to get to four. In other words, how many transitions would you have to do until you reach four? Of course we don't know. It is a random variable. In fact, it's more than one random variable. It would depend probably on where you started. Starting from two, one, or three, or four, would lead to different random variables. We are going to be interested in looking at their expectation, or the expected value of these random variables. More precisely, let us define exactly what we want to do, which is to find the expected number of transitions-- that we're going to call him Mu of i-- until reaching four, which is our absorbing state, given that the initial state is i, one of these four states. First as a warm up, let's do some quick calculation. If you didn't have this part here, and instead we were looking just at this portion here. Make this one disappear like this, and replace it by a loop of probability 0.8. So now if you start from state two, with probability 0.2 you will go to four, or with probability 0.8 you will remain in two. And now you ask yourself, what is the number of trials you have to do until you reach four? Well we know what it is. This is a geometric random variable with the probability of success, which is a success being going to four, of 0.2. So the expected number of trials, starting from two, that you will have to go through until you reach four, will be 1 over this probability. So 1/0.2, which is 5. Now that we have done this quick calculation, it should be clear that if we go back to the previous Markov chain that we had here, the expected time, Mu 2, would probably be bigger than 5. Since from two, not only you have the probability of going to four, but you might have some chance of traveling there. So probably the number of times until you reach four would be bigger than 5. Let's see. Well, first of all, if you start at four, the expected number of transitions until reaching four would obviously be zero. So here, for i equals 4, you indeed get zero. What about for the others? Well again, this is what we would like to calculate. How are we going to do that? Well, the argument is going to be of the same nature as the one that we used before. We are going to think in terms of tree, and consider possible options starting from a given state. So let's do this calculation from two. So you are in two, and we're going to build that tree here. So you are in state two. What could happen next? Well, you can either transition to four with the probability 0.2. Or with probability 0.8, you end up in one. And you have done one transition here. So plus one transition. So you are interested in calculating Mu 2. After one transition you either end up in four, in that case you stop, you're done. In other words the resulting value here would be Mu 4, which we know is 0. Or you are in one. And now, given that you are in one, you want to find the expected number of transitions until reaching four, which is exactly defined here. So here what you have is Mu 1. And now you can use the total expectation theorem to put all of these things together. What it means is that Mu 2 will be 1, and you have one transition. And then after you do that transition with probability 0.2, you know that you're going to be in four. And the expected value then will be Mu 4 plus 0.8, which is the probability that end up in state one. And conditional on that, the remaining expected iterations until reaching four will be Mu 1. Now this one is of course 0, so what you end up with is 1 plus 0.8 Mu 1. So you get a relation between Mu 2 and Mu 1. Now you can do the same thing if you start from one or start from three. So let's do it again from one. So you're interested in one. After one transition, so plus one, what happened? Well, with the probability 0.6, you end up in two. And with the probability 0.4, you end up in three. And from three, if you start in state three, after one transition what happened? Well, with the probability 0.5 you would end up in state one. And then the expected number of transitions from state one until reaching four will be Mu 1. Or with probability 0.5, you end up in two. So again here, if you look at these three here, this is a system of three equations, three linear equations with three unknowns. It has a unique solution. I will let you do the calculation, let me give you the result. What you obtain is Mu 1 will be 110/8. And the reason I'm writing it this way is so that we can compare them. Mu 2 will be 96/8, which is 12. And Mu 3 is 111/8. So here again, a quick sanity check, the number that we get here, 12, is indeed bigger than the five that we have obtained when we restricted ourselves to this set. So we do have Mu 2 greater than 5. Now as the relative value between Mu 1, Mu 2, and Mu 3, it sort of makes sense. Mu 2, the state two, is the one closest to four, it is the one actually linked to four. So in some sense, the expected number of transitions to reach four will always be the smallest one, because starting from the other states, you will have to go to two before going to four. And in general, if you have a general Markov chain with transient states and one absorbing state, and you're asking yourself, what is the expected time to absorption to that unique absorbing state, it will be the unique solution from the system of equations given here. Where the pij are the transition probabilities of your Markov chain. Now we have seen how to solve this problem when we have one unique absorbing state. What happens if you have more than one absorbing state? Like for example, in this case. Well, first of all, a quick note. You realize here that you have one, two, and three, three transition states. And indeed here, you have four as an absorbing state, it's a recurrent state, and once you get to it you stay there forever. And five is also a recurrent state, and once you get to five you stay there forever. So four and five are both absorbing states. And in a previous video, we had seen how to calculate the probability of ending up in four, as opposed to ending up in five. What we know is that the probability of ending up in four plus the probability of ending up in five will be 1. But since the probability of ending up in four is not 1, trying to find the expected number of steps until you reach four specifically does not make much sense. That expectation of that random variable is a random variable, but that expectation will be infinity. Why? Because there is a positive probability that you will end up in five. And if you end up in five, once you get there, the number of steps to go to four will be infinity. So it makes more sense to think about what is the expected time to any absorbing state. So to either four or five. Now If you're interested in that quantity, one trick in order to solve that problem using the technique that we have seen so far, is to combine four and five into one mega state, call it whatever, six. Right? And six is a combination of four and five. It's a big absorbing state. And once you're in six, you stay in six. And now you just have to define exactly what is the probability of transition from one, two, and three, to that mega state. Well here from two, you had, originally, two arcs. You're going to combine these two into one arc, and you're going to sum these probabilities. So you had 0.3 and 0.2. You put in here 0.5. And on this arc you had only one arc, so you maintain that arc. And you have that probability that you had, which I believe was 0.2. Now you go back, if you look at the situation now, it's very close to the one that we have here. All right? See this four that you have here is the six. Now of course, you have another arc here like that, but that's fine. You can stay add the arc here and put it as 0.2. And then you reduce this one to 0.3 to make it square with here. But the idea on how to solve that is identical to this one. You would have to change a little bit of this, but this is the same technique. So in the end, we have seen a technique to find the expected time to absorptions whenever you have absorbing states in a given Markov chain.