Precision and Recall: Measuring Different Classification Mistakes

Classifier mistakes do not always have equal cost.

A fraud system can block legitimate payments or miss fraudulent ones. A screening system can create unnecessary follow-up or miss a real condition.

Precision measures how trustworthy positive predictions are; recall measures how completely actual positives are found.

Choose the balance from consequences and downstream capacity.

A concrete example: medical screening

Out of 100 people with a condition, a screen finds 95:

• recall is high.

It also flags many people without the condition:

• precision is lower.

A specific follow-up test can confirm the screen. The first stage favors not missing cases.

Confusion matrix

Binary classification produces:

• true positive,
• false positive,
• true negative,
• false negative.

Every metric summarizes combinations of these counts.

Inspect the matrix, not only one score.

Precision

Precision is:

true positives / predicted positives

It answers:

Of the cases the model flagged, how many were actually positive?

High precision reduces wasted follow-up and false accusation.

Recall

Recall is:

true positives / actual positives

It answers:

Of all positive cases, how many did the model find?

High recall reduces missed positives.

False-positive rate

False-positive rate is:

false positives / actual negatives

It differs from 1 - precision.

ROC analysis uses true-positive rate, which equals recall, against false-positive rate.

Specificity

Specificity is:

true negatives / actual negatives

It measures how well negatives are correctly excluded.

Medical contexts often discuss sensitivity and specificity.

Decision threshold

Models commonly output a score.

A threshold turns the score into positive or negative. Lowering it:

• flags more cases,
• raises recall,
• often lowers precision.

The operating threshold is a product policy.

Precision-recall curve

A PR curve plots precision against recall across thresholds.

It is especially informative for rare positive classes because it focuses on positive predictions.

Compare against the baseline positive prevalence.

ROC curve

ROC plots recall against false-positive rate.

It is useful for ranking discrimination but can look strong when the negative class is huge and the absolute false-positive count is operationally unacceptable.

Inspect PR and business counts.

Prevalence

Precision depends on how common positives are.

A model with fixed sensitivity and specificity has lower precision when deployed in a population with rarer positives.

Evaluate at realistic prevalence.

Accuracy trap

If fraud is 0.1 percent, predicting "not fraud" always gives 99.9 percent accuracy.

It has zero recall for fraud.

Accuracy is useful only when class balance and mistake costs support it.

F1 score

F1 is the harmonic mean of precision and recall.

It summarizes balance but treats the two symmetrically and ignores true negatives.

Use it only when that weighting matches the decision.

F-beta

F-beta weights recall more or less:

• beta greater than one emphasizes recall,
• beta below one emphasizes precision.

One number still cannot encode all costs and capacities.

Top-k and review capacity

Human review teams may handle 1,000 alerts daily.

Evaluate:

• precision at 1,000,
• recall among top 1,000,
• captured value,
• and reviewer time.

Threshold should reflect real capacity.

Cost-sensitive evaluation

Assign estimated costs:

• missed fraud,
• false block,
• review,
• customer support,
• delayed treatment.

Optimize expected value with safety and fairness guardrails.

Costs can change by case severity.

Calibration

A calibrated score supports risk tiers:

• below 1 percent,
• 1–10 percent,
• above 50 percent.

Calibration and ranking are distinct. A model can rank correctly while probabilities are too high.

Multi-class metrics

For several classes, compute per-class precision and recall.

Aggregate through:

• macro average: equal class weight,
• micro average: aggregate decisions,
• weighted average: frequency weight.

Rare classes can disappear inside micro averages.

Subgroup performance

Overall metrics can hide different error rates across:

• regions,
• devices,
• demographic groups,
• and product types.

Evaluate affected groups with enough sample and fairness context.

Delayed labels

True outcomes may arrive later.

Fraud confirmation and medical diagnosis can take weeks. Monitor proxy metrics carefully and backfill final precision and recall once labels mature.

Do not treat unresolved cases as negatives.

Include review capacity in threshold selection

Many positive predictions enter a queue for human review. That queue has a finite capacity.

Imagine a fraud model scoring 100,000 payments each day while investigators can review 500. A threshold producing 4,000 alerts is not operationally meaningful even if its recall looks excellent on a chart. Most alerts will wait too long or never be examined.

Evaluate precision and recall at the actual review budget:

• precision among the top 500 scores,
• recall captured by those 500,
• time-sensitive value of catching a case,
• investigator effort per alert,
• and how the queue is distributed across regions or groups.

Capacity can also change. Weekends, major sales, or staff shortages alter the feasible threshold. A production policy may therefore use dynamic queues or separate thresholds by case type while preserving documented safeguards.

Report uncertainty around the metrics

Precision and recall are estimates from a sample. Small groups can show dramatic swings from only a few cases. Report confidence intervals or repeated-sample variation, and avoid declaring one model superior when the observed difference is within ordinary sampling noise.

Knowledge check

1. What question does precision answer?
2. What question does recall answer?
3. How does lowering a threshold usually change them?
4. Why does prevalence affect precision?
5. When can F1 be a misleading summary?

The one idea to remember

Precision describes correctness among positive predictions; recall describes coverage of actual positives. Select thresholds and metrics from false-positive and false-negative consequences, real prevalence, review capacity, calibration, and subgroup outcomes.