Features and Labels: What a Model Knows and What It Learns

A predictive model learns from two design choices:

what information it receives,
which outcome it is trained to predict.

Features must match information available at decision time, and labels must represent the outcome the product truly cares about.

Mistakes in either can produce impressive metrics for an invalid task.

A concrete example: loan default

Possible features:

verified income,
existing debt,
repayment history,
requested amount.

The label may be:

default within twelve months.

A debt-collection status recorded six months after the decision cannot be a feature because it reveals the future label.

Features

Features are model inputs.

They can be:

numerical,
categorical,
text,
image,
audio,
sequence,
graph,
or derived aggregate.

Each should have a definition, source, owner, and availability time.

Labels

Labels are target answers in supervised learning.

Examples:

fraud confirmed,
delivery duration,
product category,
user retained.

Labels can be delayed, noisy, censored, or subjective.

Prediction time

Define the exact prediction moment:

At payment authorization, predict whether the transaction will later be confirmed as fraud.

Every feature must be known by that moment.

The same value available tomorrow is not valid today.

Data leakage

Leakage gives the model information unavailable in real use.

Sources include:

future fields,
post-outcome notes,
target-derived aggregates,
preprocessing fit on all data,
duplicate entities across train and test.

Leakage makes evaluation unrealistically easy.

Target leakage

A feature directly or indirectly encodes the target.

For hospital mortality prediction, "discharge status" reveals the outcome.

Remove or time-bound fields based on how they are generated.

Train-serving skew

Training computes a feature one way, production computes it another.

Causes:

different code,
time window,
missing-value handling,
data source,
timezone.

Share definitions and compare online values with offline reconstruction.

Numerical features

Numerical values may need:

scaling,
logarithm,
clipping,
bucket,
or units.

Tree models and neural networks respond differently to scaling.

Preserve meaningful extremes rather than clipping blindly.

Categorical features

Represent categories through:

one-hot encoding,
target-independent hashing,
learned embedding,
ordinal encoding only when order exists.

Handle unseen categories explicitly.

Missing values

Missing can mean:

not collected,
not applicable,
unavailable,
declined,
or pipeline failure.

One generic zero can erase these distinctions. Use indicators or domain-specific handling.

Text and images

Raw media becomes numerical representation through:

tokenization,
embeddings,
pretrained encoders,
pixel normalization,
augmentation.

Pretrained representations carry assumptions and bias from their source data.

Aggregates

Features such as:

purchases in 30 days,
average support response,
failed attempts in one hour

need event-time definitions and cutoff. Calculating with future events leaks information.

Labels from human review

Human labels need:

clear guidelines,
training,
agreement measurement,
adjudication,
and uncertainty.

Ambiguous tasks may need probabilistic or multi-rater labels rather than fake certainty.

Proxy labels

The desired outcome may be unavailable, so a proxy is used.

Clicks proxy interest, arrests proxy crime, repayment proxy creditworthiness.

Proxies can encode institutional behavior and misalign with the real goal.

Delayed labels

Fraud or churn labels may arrive weeks later.

Training data excludes recent examples without mature outcomes. Monitoring needs proxy signals until final labels arrive.

Record label maturity.

Censored labels

An outcome may remain unknown:

customer still active,
loan not yet matured,
patient lost to follow-up.

Treating unknown as negative biases training.

Survival analysis or careful cutoff can help.

Feature selection

More features can add:

noise,
cost,
privacy risk,
latency,
and overfitting.

Select through domain reasoning, validation, regularization, and ablation.

Feature importance

Importance estimates which inputs influence a model.

They do not prove causation and can be unstable among correlated features.

Use them for diagnosis with domain review.

Feature store

A feature store manages reusable definitions and values for:

offline training,
online serving,
lineage,
freshness,
and access.

It helps parity but adds platform and governance work.

Privacy and fairness

Features can expose sensitive traits directly or through proxies.

Review:

necessity,
consent,
retention,
group outcomes,
and inference risk.

Removing one protected field does not remove all correlated information.

Keep feature and label lineage

For every important feature and label, record where it came from and how it was transformed. Useful lineage includes:

source table or event,
owning team,
event time and processing time,
units and allowed values,
missing-value behavior,
transformation code version,
and known quality limitations.

This documentation turns debugging from guesswork into investigation. If a delivery-time model suddenly worsens, lineage can reveal that distance changed from kilometres to metres or that “delivered” began including cancelled orders.

Lineage also supports point-in-time correctness. Training must reconstruct only information available when each historical prediction would have been made. A current customer profile may contain facts learned after the target event; joining it directly to old examples silently leaks the future.

Use feature contracts at serving time

Training and production code must agree on names, types, units, categories, defaults, and freshness. Enforce those expectations with a feature contract. Reject or quarantine invalid records instead of letting a missing field become an unexplained zero.

Knowledge check

Why must prediction time be defined before feature selection?
What is train-serving skew?
Why can missing values need several meanings?
How can proxy labels misdirect a model?
Why does feature importance not prove causality?

The one idea to remember

Features are what a model can know at prediction time, and labels are the outcome it learns to approximate. Valid ML depends on time-correct inputs, meaningful targets, leakage prevention, consistent transformations, label maturity, and careful privacy and fairness review.