Let’s assume the Austrian population is getting tested for the Coronavirus
1% of the population is indeed infected, meaning \(P(C) = 0.01\)
A test is 98% accurate, in the sense that \(P(\text{Test}_C \mid C) = P(\text{Test}_{\neg C} \mid \neg C) = 0.98\)
What is the probability that I have the Coronavirus given that I tested positive? \(P(C \mid \text{Test}_C)\)?
Which value from the confusion matrix did we calculate here (and which ones were given)?
Judging Performance in Classification Problems
All the indicators in the confusion matrix can be relevant
Accuracy depends on base rate
accuracy of 96% in sample with 95% positives → poor performance
accuracy of 80% in sample with 50% positives → good performance
Need for adjustment of values relative to a “naive prediction”
Concepts can be generalized to classification problems with more than two categories
Adjusted Performance Measures
Cohen’s kappa: \[ \kappa = \frac{Acc_{mod} - Acc_{0}}{1 - Acc_{0}} \] where \(Acc_{mod}\) is the accuracy of our model and \(Acc_{0}\) is the expected random accuracy
For any sensible model, it holds that \(0 < \kappa \leq 1\)
If \(\kappa < 0\), our model would be worse than guessing at random
\(\kappa\) is the share of the gap between random and perfect accuracy that the model closes
ROC- and PR-Curve
Trade-off between the measures, which we can exploit by setting the cut-off point for predicted scores
Receiver Operating Characteristic (ROC) curve
Plot of sensitivity against 1-specificity
Helps identify which cut-off point to choose
Precision-Recall (PR) curve
Plot of precision against sensitivity (recall)
Might be preferred over ROC when categories are very imbalanced
Receiver Operating Characteristic
The trade-off through the setting of the cut-off point can be visualized in a ROC curve
Receiver Operating Characteristic
ROC criterion
Choose the cut-off where a gain in sensitivity costs an equal loss in specificity (curve is parallel to the 45° line)
This maximizes Youden’s \(J\) = sensitivity + specificity − 1
Sometimes high sensitivity is preferred over specificity (e.g., medical tests), or vice versa
Area Under the Curve (AUC)
The area underneath the ROC curve can be used as performance measure over all cut-offs
AUC = probability that a random positive gets a higher score than a random negative
The AMAS has been criticized for intransparency (among other things)
One of the few publicly available documents states the usage of a logit model (which was, contrary to public belief, never implemented) for this algorithm
Or rather, two logit models:
One predicting your short-term chance of labor market integration (assignment to group A if chances are high)
One predicting your long-term labor market integration (assignment to group C if chances are low), given a set of demographic variables and your labor market history
The assignment to groups A, B, and C determines if you are eligible for certain types of subsidies
The Infamous “AMS-Algorithmus” II
In the documentation, it is correctly stated that the cut-off point can be chosen in a way to balance sensitivity and specificity (ignoring the strange definitions of Sensitivity and Specificity)
The Infamous “AMS-Algorithmus” III
What is shown here (and what isn’t)?
The Infamous “AMS-Algorithmus” IV
The cut-off points were set (manually) at 25% for group C and 66% for group A
By setting the cut-off points low for group C and high for group A, they achieved high precision in those two classes
The precision in group B is not shown here, as well as other measures that would help us judge the predictive performance
Cross Validation
The real magic behind supervised machine learning
The Problem of Overfitting
Models achieve an extremely good fit through the usage of many variables and high depth
Danger of underestimating the random error \(\sigma^2\) of the data-generating process
Results in a model with low in-sample prediction error but high out-of-sample error
The Solution to Overfitting
Split your data into training, validation and test data
Model is estimated on the training data, and performance measures are calculated on the validation data
Repeat for different hyperparameter values (e.g., the penalty \(\lambda\)) and choose the value which optimizes performance on the validation data
The test data is used at the end to assess the final model
Choosing \(\lambda\) on the validation data makes its performance there optimistic
Only unseen data gives an honest estimate of out-of-sample performance
CART
4830 samples
3 predictor
2 classes: 'successful', 'unsuccessful'
No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times)
Summary of sample sizes: 4346, 4348, 4347, 4347, 4347, 4347, ...
Resampling results across tuning parameters:
cp ROC Sens Spec
5e-04 0.7173981 0.6741891 0.6543656
1e-03 0.7169620 0.7072193 0.6465661
5e-03 0.6996839 0.6867633 0.6737720
5e-02 0.6612506 0.7439858 0.5740977
ROC was used to select the optimal model using the largest value.
The final value used for the model was cp = 5e-04.
Extracting the Model
tree <- tree_caret$finalModelrpart.plot(tree, box.palette ="RdBu", nn =FALSE, type =2)