Home / Summaries / COMP 3711 / roc-validation-bootstrap

Classification - ROC, AUC, Evaluation Methods

Q: What does a ROC curve plot? What does a point in ROC space mean?

A ROC (Receiver Operating Characteristic) curve plots: X-axis: False-Positive Rate (FPR) Y-axis: True-Positive Rate (Recall / TPR) Each point corresponds to a different decision threshold — how aggressively the model predicts 'positive'. As the threshold decreases, more examples are labelled positive: TPR increases but so does FPR.

6 important questions on Classification - ROC, AUC, Evaluation Methods

What does a ROC curve plot? What does a point in ROC space mean?

A ROC (Receiver Operating Characteristic) curve plots:

X-axis: False-Positive Rate (FPR)
Y-axis: True-Positive Rate (Recall / TPR)

Each point corresponds to a different decision threshold — how aggressively the model predicts "positive". As the threshold decreases, more examples are labelled positive: TPR increases but so does FPR.

What does the AUC measure, and what do AUC = 1.0 and AUC = 0.5 mean?

AUC (Area Under the ROC Curve) is a single-number summary of classifier performance across all possible thresholds:

AUC = 1.0 → perfect classifier (reaches top-left corner: TPR = 1, FPR = 0)
AUC = 0.5 → model no better than random guessing (diagonal line from bottom-left to top-right)
AUC = 0.0 → perfectly wrong (always misclassifies — flip its predictions for a perfect model)

AUC is useful because it evaluates overall discriminatory power independently of error costs or threshold choice.

What is "dominance" in ROC space?

Classifier A dominates classifier B in ROC space if A is higher and to the left of B — meaning A achieves a higher TPR at the same FPR, or the same TPR at a lower FPR (or both). A dominated classifier is objectively worse under all cost assumptions. A classifier that is not dominated by any other is said to be on the Pareto frontier.

What is the holdout method? What are its three data splits?

The holdout method partitions the data into three disjoint sets:

Training set — used to build the model
Validation set — used to tune hyperparameters (e.g., tree depth, λ) during model selection
Test set — used once at the very end for a final unbiased performance estimate

Using the test set for model selection is "cheating" — it leaks test information into the training process and produces an overoptimistic estimate.

Describe k-fold cross validation. When is LOOCV preferred?

k-fold CV:

Partition data into k equal folds
Repeat k rounds: hold out fold i as validation, train on the other k−1 folds
Average the validation error across all k rounds

LOOCV (Leave-One-Out): k = n — each single example is the validation set. Use when:

Data is very scarce and every example matters for training
Downside: high variance per fold estimate; expensive (n training runs)

What is the bootstrap method and when is it used?

Bootstrap = sampling with replacement from the original dataset n times to create a new training set of size n. On average, ~63.2% of original records appear in each bootstrap sample (some appear multiple times, others not at all).
Used for:

Bagging: generate multiple bootstrap training sets, train a model on each, average/vote predictions → reduces variance
Estimating model stability and variance when the dataset is small

The question on the page originate from the summary of the following study material:

COMP 3711

View summary