Overfitting and Underfitting: Learning the Pattern, Not the Noise

Perfect training performance can be a warning.

A model may memorize quirks of the examples it has seen rather than learn relationships that hold for new cases.

Overfitting learns training-specific noise; underfitting fails to capture enough useful structure.

The goal is the best performance on representative unseen data.

A concrete example: image lighting

A model recognizes training images almost perfectly.

Most training photos were taken under one lighting setup. New images from another camera and angle fail.

The model learned background and lighting cues that correlated with labels in training but not in deployment.

Training error

Training error measures performance on data used to fit parameters.

It generally falls as the model:

• becomes more flexible,
• trains longer,
• or memorizes examples.

Low training error alone says little about generalization.

Validation error

Validation data estimates new-example behavior during development.

If training error continues improving while validation worsens, overfitting is likely.

Validation must represent deployment and remain free of leakage.

Underfitting

An underfit model performs poorly on both training and validation.

Possible causes:

• model too simple,
• inadequate features,
• insufficient training,
• excessive regularization,
• incorrect target,
• or optimization failure.

Adding data alone may not fix a model unable to express the relationship.

Overfitting

An overfit model performs much better on training than validation.

Causes include:

• excessive capacity,
• small dataset,
• noisy labels,
• many feature choices,
• long training,
• leakage,
• or repeated validation tuning.

The gap is evidence, not one universal threshold.

Bias and variance

Underfitting is associated with high bias: assumptions are too restrictive.

Overfitting is associated with high variance: predictions change too much with the training sample.

Real systems can have both in different regions of the data.

Model capacity

Capacity is the range of patterns a model can represent.

Examples:

• deeper tree,
• more polynomial terms,
• larger neural network,
• more parameters.

More capacity helps only when data and regularization support it.

Learning curves

Plot performance against training-data size.

If validation improves as more data is added, collection may help. If training and validation are both poor and close, improve representation or model.

Learning curves guide the next investment.

Regularization

Regularization discourages overly complex solutions.

Methods include:

• L1 or L2 penalties,
• dropout,
• weight decay,
• tree pruning,
• early stopping.

The strength is a hyperparameter chosen on validation.

Early stopping

Stop training when validation performance stops improving.

This prevents continued fitting of training noise.

Use a patience window because validation metrics fluctuate.

Data augmentation

Create realistic variations:

• crop image,
• adjust brightness,
• add audio noise,
• paraphrase text cautiously.

Augmentation teaches invariance expected in deployment.

Unrealistic transformations can corrupt labels.

More data

More representative labeled data often reduces variance.

Duplicate or biased data does not necessarily help. Focus on:

• missing conditions,
• rare classes,
• difficult examples,
• and deployment sources.

Quality and diversity matter.

Feature selection

Irrelevant features let models fit accidental correlations.

Remove or regularize through:

• domain reasoning,
• ablation,
• sparsity,
• and validation.

Do not select features using the final test set.

Cross-validation

Several validation folds reveal whether performance depends heavily on one split.

Large variation signals unstable learning or grouped data leakage.

Respect time and entity boundaries.

Hyperparameter overfitting

Trying thousands of configurations adapts to validation noise.

The chosen model can overfit validation even when each individual training run is proper.

Use nested evaluation, bounded search, and final untouched test.

Data leakage versus overfitting

Leakage can look like exceptional generalization but is a separate validity error.

The model uses information unavailable at prediction time or shared across splits.

Always investigate suspiciously high metrics.

Distribution shift

A model may generalize within held-out historical data and fail after the world changes.

This is not ordinary overfitting alone. Monitor:

• inputs,
• outcomes,
• calibration,
• and subgroup performance.

Retraining needs representative recent labels.

Double descent

Modern large models can show complex behavior where validation error falls, rises, then falls again as capacity grows.

This does not eliminate overfitting concerns. Empirical scaling, regularization, data quality, and held-out evaluation remain necessary.

Product complexity

A slightly better model can add:

• latency,
• cost,
• explanation difficulty,
• and operational risk.

Choose the simplest model meeting product objectives and guardrails.

Diagnose before choosing a remedy

Different failure patterns call for different interventions:

• high training and validation error suggests underfitting, weak features, noisy labels, or an impossible target;
• low training error with much higher validation error suggests overfitting;
• good random-split results but poor future results suggests time leakage or distribution shift;
• unstable results across splits suggests too little data or highly variable subgroups;
• strong aggregate results with weak segment results suggests representation or objective problems.

Do not automatically add a larger model when metrics disappoint. Plot learning curves, inspect errors, compare simple baselines, audit labels, and repeat evaluation across meaningful slices. A mislabeled dataset cannot be repaired by more capacity, and a shifted production population cannot be repaired merely by stronger regularization.

Knowledge check

1. What performance pattern suggests overfitting?
2. Why can an underfit model remain poor with more data?
3. How do learning curves guide action?
4. What does regularization discourage?
5. How can hyperparameter search overfit validation?

The one idea to remember

Underfitting misses meaningful structure; overfitting memorizes sample-specific noise. Use representative splits, learning curves, appropriate capacity, regularization, augmentation, data diversity, and untouched evaluation to optimize performance on future cases rather than training memory.