Unit 9 Value of Information, Video 4 Expected Value of Perfect Information

[SQUEAKING] [RUSTLING] [CLICKING] RICHARD DE NEUFVILLE: So what do we do in practice? The simple approach is to calculate the upper bound. This is known as the expected value of perfect information. So this is where I'm going to be leading you in this exercise. So the expected value of perfect information is totally hypothetical. It does not exist. But it simplifies the analysis and gives you an upper bound, which is useful because we can definitely exclude some tests if they don't meet-- if they exceed the upper bound of the possible value. So the concept is that there is a perfect test, which exactly which event is going to occur. I will call this-- I'm calling this a Cassandra machine. The Greek mythology, not that anybody should necessarily know it, is that this lady was Cassandra who had the gift, so to speak, of always seeing what the future would be and with the curse that nobody would ever believe her. It was all about the battle for Troy, and she could predict that somebody would die, but nobody would believe her, and so on. So that was the thing, that she could predict absolutely. So this is the Cassandra machine. It predicts absolutely. Now, of course, that does not exist, but what is the benefit of it? So the Cassandra machine is this black box which predicts exactly what test result will occur. It produces the best possible information. Therefore, I mean, it's absolutely true. So if you had the best possible information-- that is, if you ended up with no uncertainty-- you could make the best decisions, and you would have the maximum gain over the decision you'd make without the information. Therefore, it gives you the upper limit on the value of the test. So what we can do if we can get the real value, we can at least set some limits on it. And this is what the Cassandra machine, this hypothetical thing, does. The further beauty of it is that the calculation for this expected value of perfect information is almost trivial. The first time you try to do it, you'll flounder around a bit because you're not used to it. But basically, it's a very simple device and is pretty transparent. So let me give you an example. So remember, this is the raincoat problem, that you're getting up in the morning, and should you decide to take the raincoat or not. There may be a chance of rain, rain or no rain, as in the diagram. There's a prior assessment that the probability is 0.4 or 0.6 of rain or no rain. And there is these values of the situation in the end of 5, minus 2, minus 10, and 4, values that I've invented just to give us numbers to play with. But that's the original problem. Now, just to set ourselves back into this context, the better choice at this point is what? So how do we remind you how it is? If we take the raincoat, which is a top decision, we have a 0.4 chance of having 5, which averages to 2. We have a 0.6 chance of getting a minus 2, which is a minus 1.2. 2 minus 1.2 is 0.8. Conversely, if I don't take the raincoat, I have a probability of 4 of the minus 10. That's minus 4. A probability of 0.6 of 4, 2.4. So it's a negative number. So my best choice is to take the raincoat, right? So I hope that you're all now back into this particular groove. So the setup for the probabilities for the EVPI. It's either going to say it's going to rain or not rain. That is the perfect machine. You press the button, say truth please, and it'll say Cassandra machine says rain. Or it will say the contrary because I've only given it two choices. Now, what are the probabilities it's gonna say that? Well, your best estimate of these probabilities is what you have started with before. You thought there was a 40% chance of getting rain, so 40% of the time, it's going to tell you it's going to rain. And conversely, so that you-- a priori, you have your best estimates for the value of the outcome is what your priors were. Now, in general, once you know what the resolution of the uncertainty is, the best decision is pretty obvious. If you knew it would rain-- going back here, if you knew it was going to rain, you're comparing 5 versus 10. Well, you'd take the raincoat. If you knew for sure it wasn't going to rain, you compare minus 2 and 4, and you don't take the raincoat. So you don't have to do any complicated calculations of the possibilities. You would have a probability of 0.4 that would take the raincoat to get it to 5, and the probability of 0.6 of not taking the raincoat and getting a result of 4. So you can write down your best decisions pretty automatically. Now, the expected value after this case, as I'm running this calculation, doing the 0.4 times the result of 5 and the 0.6 of 4, your perfect value is 4.4. Now, the expected value of perfect information is the value of the increment, not the value at the end of it. It's the value of the increment. What did you get from it? So the expected value after the test was 4.4. The expected value before the test was 0.8. The expected value of perfect information in this case is 3.6. All right, so the nice thing about it is the prior-- the expected value of simple-- perfect information is simple to calculate. The prior probability must equal the probability of the perfect test result. Once you've run the perfect test, the posterior probabilities are either 0 or 1. Either this happens, or it doesn't happen. There's no doubt about that. Once you know that there is a whole lot of 0 probabilities and there's a one probability, you know what you're going to choose, and pretty much you can write the expected value of perfect information formula directly. There's no need to use Bayes' Theorem. So Robert, since you asked the question before-- I'm not trying to put you on the spot. But since you've been doing it, and I'm showing you this. Why did you not use this approach, if you knew about it? Or maybe you didn't know about it. So what is the-- can you have a discussion about this, please, or your response to this, if I'm not embarrassing you or putting you on the spot, which I don't mean. AUDIENCE: No, that's fine. I actually haven't heard this perfect information idea. Yeah, so that's probably the main reason I didn't use this. But the other one was at Chevron, we kind of have a defined procedure in terms of how we justify value of information. So that procedure was utilizing the conditional probabilities. But myself, I was personally a bit uncomfortable because like you were saying, it's hard to figure out what the percentages would be for either a good, medium, or bad outcome. So it was pretty challenging to actually apply, and I think at the end, people weren't super confident in it, like were saying. RICHARD DE NEUFVILLE: Well, I think that that's right. So about 30 or 40 years ago-- I mean, I'm old enough to remember that-- there was a phase where decision analysis was really taught a lot in business schools, which has since gone by. And companies picked up on that because it was certainly better than not looking at uncertainty. And it was nice to have this notion of a Bayes' rule, a Bayes' Theorem. And people thought of themselves as being Bayesian or not. And so there's a whole history about why this was the correct improvement in approach, not thinking about the fact that these other issues, the probabilities-- the associated probabilities and the probability of occurrence and so forth, which are the part of that special factor for the operation of the Bayes' Theorem are just basically hard to get and hard to document and hard to prove. So I think that you are reflecting, which may exist elsewhere-- I'm not pointing a finger at Chevron-- but that they have a decision analysis manual. They have an approach, a procedure, and so forth. But it doesn't really fly. And as you suggest, that a lot of people don't believe in it. And I would be one of those, if I were in Chevron, because it's such a spongy-- the calculation is correct, but the data are spongy. And it's the GIGO, garbage in, garbage out kind of situation, I think. So I understand that you're encouraged to do that, and this EVPI is trivial mathematically. I mean, you don't have to be smart or think through the complexities, but it gives you an upper bound. And that is the useful aspect of it. And I thank you for noting that it is not the usual way of approaching things. But I think that it's a reasonable way to do it, and productive. And I would encourage you to take the message back. Now, I can see that they're not going to change the procedures for you. And so what people do in the case if the procedure is wrong, they sort of try to ignore it, which is what you reflected.