EVALUATION
METRICS CHEATSHEET.

Components

• TP – True Positives (correctly predicted positive)
• TN – True Negatives (correctly predicted negative)
• FP – False Positives (incorrectly predicted positive)
• FN – False Negatives (incorrectly predicted negative)

Confusion Matrix Layout

                    Predicted
                    +---------+---------+
              +     |    TP   |    FP   |
              A     +---------+---------+
              C     |    FN   |    TN   |
              T     +---------+---------+

Accuracy

• (TP + TN) / (TP + TN + FP + FN)
• Proportion of correct predictions
• Use when: balanced classes
• Avoid when: imbalanced classes

Precision

• TP / (TP + FP)
• Proportion of positive predictions that were correct
• Use when: FP cost is high (e.g., spam detection)
• Also known as: Positive Predictive Value (PPV)

Recall / Sensitivity

• TP / (TP + FN)
• Proportion of actual positives correctly identified
• Use when: FN cost is high (e.g., disease detection)
• Also known as: True Positive Rate (TPR)

Specificity

• TN / (TN + FP)
• Proportion of actual negatives correctly identified
• Also known as: True Negative Rate (TNR)

F1 Score

• 2 * (Precision * Recall) / (Precision + Recall)
• Harmonic mean of precision and recall
• Use when: need balance between precision and recall
• Range: 0 (worst) to 1 (best)

Fβ Score

• (1 + β²) * (P * R) / (β² * P + R)
• β > 1 favours recall, β < 1 favours precision
• F0.5 – favours precision
• F2 – favours recall

ROC‑AUC

• Area Under the ROC Curve
• ROC: TPR (y) vs FPR (x) at all thresholds
• 0.5 – random guessing
• 1.0 – perfect classifier
• Use when: evaluating model performance across thresholds

PR‑AUC

• Area Under the Precision‑Recall Curve
• More informative than ROC for imbalanced data
• Precision (y) vs Recall (x) at all thresholds

Log Loss (Cross‑Entropy)

• L = -1/N Σ yᵢ log(pᵢ) + (1-yᵢ) log(1-pᵢ)
• Measures uncertainty of probability predictions
• Lower is better
• Penalises confident wrong predictions heavily

Matthews Correlation Coefficient (MCC)

• (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))
• Balanced metric for imbalanced data
• Range: -1 (total disagreement) to +1 (perfect)
• 0 = random prediction

Balanced Accuracy

• (Recall + Specificity) / 2
• Average of per‑class recall
• Useful for imbalanced classes

Cohen's Kappa

• Measures inter‑rater agreement
• Accounts for chance agreement
• Range: -1 to 1
• 0 = no agreement, 1 = perfect

Micro‑Averaging

• Aggregate all TP, FP, FN across classes
• Weighted by class frequency
• Use when class imbalance is present

Macro‑Averaging

• Average of per‑class metrics
• Each class has equal weight
• Use when all classes are equally important

Weighted‑Averaging

• Weighted by class support (number of samples)
• Balance between micro and macro

Hamming Loss

• Proportion of incorrect labels
• Lower is better
• Use for multi‑label classification

MSE (Mean Squared Error)

• 1/n Σ (yᵢ - ŷᵢ)²
• Penalises large errors heavily
• Same units as squared target
• Sensitive to outliers

RMSE (Root Mean Squared Error)

• √MSE
• Same units as target
• More interpretable than MSE
• Sensitive to outliers

MAE (Mean Absolute Error)

• 1/n Σ |yᵢ - ŷᵢ|
• Robust to outliers
• Same units as target
• Less sensitive to outliers than MSE

MAPE (Mean Absolute Percentage Error)

• 100/n Σ |(yᵢ - ŷᵢ) / yᵢ|
• Percentage error
• Undefined when yᵢ = 0

R² (Coefficient of Determination)

• 1 - (Σ(yᵢ - ŷᵢ)² / Σ(yᵢ - ȳ)²)
• Proportion of variance explained
• Range: -∞ to 1
• 1 = perfect, 0 = baseline (mean)
• Can be negative (worse than baseline)

Adjusted R²

• 1 - (1 - R²) * (n - 1) / (n - k - 1)
• Penalises added features
• Use for model selection

MSE vs MAE

• MSE: penalises large errors more
• MAE: penalises all errors equally
• MSE is differentiable (better for optimisation)
• MAE is more interpretable

MASE (Mean Absolute Scaled Error)

• MAE / MAE of naive forecast
• Scale‑independent
• Good for comparing across datasets

Silhouette Score

• a = mean intra‑cluster distance
• b = mean nearest‑cluster distance
• score = (b - a) / max(a, b)
• Range: -1 (poor) to +1 (excellent)
• 0 indicates overlapping clusters

Davies‑Bouldin Index

• Average similarity between clusters
• Lower is better
• Ratio of within‑cluster to between‑cluster distance
• 0 = optimal

Calinski‑Harabasz Index

• Variance Ratio (between / within)
• Higher is better
• Also known as Variance Ratio Criterion

Inertia (Within‑Cluster SSE)

• Σ (distance from centroid)²
• Lower is better
• Used by K‑Means
• Decreases with more clusters (elbow method)

Adjusted Rand Index (ARI)

• Measures similarity to ground truth
• Adjusted for chance
• Range: -1 to 1
• 0 = random, 1 = perfect

Mutual Information (MI)

• Measures shared information
• Adjusted MI (AMI) – adjusted for chance
• Normalised MI (NMI) – scaled to 0‑1

Mean Average Precision (MAP)

• Average of precision at each relevant item
• Used for ranking and IR
• Higher is better

NDCG (Normalised Discounted Cumulative Gain)

• Measures ranking quality
• Considers position of relevant items
• DCG = Σ relᵢ / log₂(i+1)
• NDCG = DCG / IDCG (ideal)
• Range: 0 to 1

Precision@k / Recall@k

• Precision/recall at top‑k results
• Used in search and recommender systems
• Evaluates relevance of top results

MRR (Mean Reciprocal Rank)

• Average of 1 / rank of first relevant item
• Used for search and QA
• Range: 0 to 1

Cross‑Validation (k‑fold)

• Split data into k folds
• Train on k‑1, validate on 1
• Repeat k times
• Average performance
• k = 5 or 10 (common)

Stratified k‑fold

• Preserves class distribution
• Better for imbalanced data
• Each fold has same proportion of classes

Train‑Validation‑Test Split

• Train: 60‑70%
• Validation: 15‑20% (tuning)
• Test: 15‑20% (final evaluation)
• Never use test for tuning

Cross‑Validation vs Train/Test

• CV: better use of data, slower
• Train/Test: faster, less compute
• Use CV for small datasets
• Use Train/Test for large datasets

Classification Quick Picks

• Best overall: F1, MCC
• Imbalanced: MCC, F1, PR‑AUC
• Default: Accuracy (if balanced)
• Probabilistic: Log Loss
• Ranking: ROC‑AUC

Regression Quick Picks

• Best overall: RMSE
• Interpretable: MAE
• % error: MAPE
• Variance explained: R²
• Robust: MAE