3.2.8 Introduction to Logistical Regression - Video 5 Thresholding

We saw in the previous video that the outcome of a logistic regression model is a probability. Often, we want to make an actual prediction. Should we predict 1 for poor care, or should we predict 0 for good care? We can convert the probabilities to predictions using what's called a threshold value, t. If the probability of poor care is greater than this threshold value, t, we predict poor quality care. But if the probability of poor care is less than the threshold value, t, then we predict good quality care. But what value should we pick for the threshold, t? The threshold value, t, is often selected based on which errors are better. You might be thinking that making no errors is better, which is, of course, true. But it's rare to have a model that predicts perfectly, so you're bound to make some errors. There are two types of errors that a model can make -- ones where you predict 1, or poor care, but the actual outcome is 0, and ones where you predict 0, or good care, but the actual outcome is 1. If we pick a large threshold value t, then we will predict poor care rarely, since the probability of poor care has to be really large to be greater than the threshold. This means that we will make more errors where we say good care, but it's actually poor care. This approach would detect the patients receiving the worst care and prioritize them for intervention. On the other hand, if the threshold value, t, is small, we predict poor care frequently, and we predict good care rarely. This means that we will make more errors where we say poor care, but it's actually good care. This approach would detect all patients who might be receiving poor care. Some decision-makers often have a preference for one type of error over the other, which should influence the threshold value they pick. If there's no preference between the errors, the right threshold to select is t = 0.5, since it just predicts the most likely outcome. To make this discussion a little more quantitative, we use what's called a confusion matrix or classification matrix. This compares the actual outcomes to the predicted outcomes. The rows are labeled with the actual outcome, and the columns are labeled with the predicted outcome. Each entry of the table gives the number of data observations that fall into that category. So the number of true negatives, or TN, is the number of observations that are actually good care and for which we predict good care. The true positives, or TP, is the number of observations that are actually poor care and for which we predict poor care. These are the two types that we get correct. The false positives, or FP, are the number of data points for which we predict poor care, but they're actually good care. And the false negatives, or FN, are the number of data points for which we predict good care, but they're actually poor care. We can compute two outcome measures that help us determine what types of errors we are making. They're called sensitivity and specificity. Sensitivity is equal to the true positives divided by the true positives plus the false negatives, and measures the percentage of actual poor care cases that we classify correctly. This is often called the true positive rate. Specificity is equal to the true negatives divided by the true negatives plus the false positives, and measures the percentage of actual good care cases that we classify correctly. This is often called the true negative rate. A model with a higher threshold will have a lower sensitivity and a higher specificity. A model with a lower threshold will have a higher sensitivity and a lower specificity. Let's compute some confusion matrices in R using different threshold values. In our R console, let's make some classification tables using different threshold values and the table function. First, we'll use a threshold value of 0.5. So type table, and then the first argument, or what we want to label the rows by, should be the true outcome, which is qualityTrain$PoorCare. And then the second argument, or what we want to label the columns by, will be predictTrain, or our predictions from the previous video, greater than 0.5. This will return TRUE if our prediction is greater than 0.5, which means we want to predict poor care, and it will return FALSE if our prediction is less than 0.5, which means we want to predict good care. If you hit Enter, we get a table where the rows are labeled by 0 or 1, the true outcome, and the columns are labeled by FALSE or TRUE, our predicted outcome. So you can see here that for 70 cases, we predict good care and they actually received good care, and for 10 cases, we predict poor care, and they actually received poor care. We make four mistakes where we say poor care and it's actually good care, and we make 15 mistakes where we say good care, but it's actually poor care. Let's compute the sensitivity, or the true positive rate, and the specificity, or the true negative rate. The sensitivity here would be 10, our true positives, divided by 25 the total number of positive cases. So we have a sensitivity of 0.4. Our specificity here would be 70, the true negative cases, divided by 74, the total number of negative cases. So our specificity here is about 0.95. Now, let's try increasing the threshold. Use the up arrow to get back to the table command, and change the threshold value to 0.7. Now, if we compute our sensitivity, we get a sensitivity of 8 divided by 25, which is 0.32. And if we compute our specificity, we get a specificity of 73 divided by 74, which is about 0.99. So by increasing the threshold, our sensitivity went down and our specificity went up. Now, let's try decreasing the threshold. Hit the up arrow again to get to the table function, and change the threshold value to 0.2. Now, if we compute our sensitivity, it's 16 divided by 25, or 0.64. And if we compute our specificity, it's 54 divided by 74, or about 0.73. So with the lower threshold, our sensitivity went up, and our specificity went down. But which threshold should we pick? Maybe 0.4 is better, or 0.6. How do we decide? In the next video, we'll see a nice visualization to help us select a threshold.