Evaluation Metrics

The Imbalanced Data Problem

Imagine a dataset of 100 patients, where 99 are healthy (0) and 1 has cancer (1). If a lazy model predicts "0" every single time, it is 99% accurate! But it completely failed its medical purpose.

The Confusion Matrix

To truly evaluate a model, we look at 4 outcomes:

• True Positives (TP): Model predicted 1, actual is 1.
• True Negatives (TN): Model predicted 0, actual is 0.
• False Positives (FP): Model predicted 1, actual is 0. (Type I error)
• False Negatives (FN): Model predicted 0, actual is 1. (Type II error - Dangerous in medical/spam filtering)

Metrics

• Precision: Out of all 1s we predicted, how many were actually 1? (TP / (TP + FP))
• Recall: Out of all actual 1s, how many did we find? (TP / (TP + FN))