Data Quality: Fitness for a Specific Use

A pipeline can deliver every file on time and still produce a misleading decision.

Rows may be duplicated, refunds omitted, units mixed, or definitions inconsistent.

Data quality is fitness for a particular use, not merely successful storage or pipeline execution.

The same dataset can be adequate for one question and unsafe for another.

A concrete example: revenue without refunds

A dashboard contains every sale row correctly.

The refund source stopped loading three days ago. Gross sales look accurate, but net revenue is overstated.

Each row is syntactically valid while the dataset is incomplete for its intended metric.

Accuracy

Accuracy asks whether values represent reality correctly.

Examples:

correct amount,
correct address,
correct diagnosis,
correct product category.

Ground truth may require authoritative comparison, sampling, or human review.

Completeness

Completeness asks whether required records and fields are present.

Check:

missing partitions,
null rate,
source counts,
expected entities,
and late arrivals.

One missing source can bias an otherwise large dataset.

Consistency

Consistency asks whether related representations agree.

Examples:

order total equals line totals plus tax,
customer region uses one code set,
dashboard and ledger agree,
one ID maps to one entity.

Cross-system consistency often requires reconciliation.

Timeliness

Timeliness asks whether data is fresh enough for use.

Daily executive reporting and fraud decisions need different delay.

Define freshness objective from business consequence.

Validity

Validity checks whether data follows format and domain rules:

date exists,
status belongs to allowed set,
amount is nonnegative,
identifier has expected structure.

Valid data can still be inaccurate.

Uniqueness

Uniqueness identifies unintended duplicates.

Define the key:

event ID,
order ID,
one active subscription per account.

Two legitimate line items in one order are not duplicates merely because values match.

Integrity

Referential integrity asks whether relationships are valid:

order customer exists,
payment references order,
event has known device.

Distributed pipelines may receive related records at different times, so checks need late-arrival policy.

Quality depends on grain

A dataset's row meaning must be explicit.

Is one row:

order,
order line,
payment attempt,
or daily customer summary?

Without grain, uniqueness and totals are ambiguous.

Source quality

Catch problems near the source:

required field,
database constraint,
type,
user confirmation,
sensor calibration.

Downstream cleanup cannot reliably infer missing intent.

Data contracts

A data contract defines:

schema,
semantics,
owner,
freshness,
quality expectations,
and change policy.

It makes source and consumer responsibilities visible.

Pipeline checks

Automated checks include:

row count,
null percentage,
uniqueness,
range,
accepted values,
schema,
reconciliation,
distribution drift.

Thresholds should account for normal seasonality.

Reconciliation

Compare independent records:

order system versus payment provider,
source table versus warehouse load,
invoice total versus ledger.

Reconciliation catches silent loss and duplicate processing that local checks miss.

Distribution drift

Data can remain valid while its pattern changes unexpectedly.

Examples:

average age jumps,
one country disappears,
all events use one device version.

Drift may signal a real business change or pipeline defect.

Outliers

An outlier can be:

error,
fraud,
rare valid event,
or important new behavior.

Do not delete outliers automatically. Investigate and preserve raw evidence.

Quality dimensions conflict

Waiting for late data improves completeness but reduces timeliness.

Strict validation improves validity but can reject legitimate new values.

Quality policy should state which tradeoffs apply to the use.

Ownership

Every important dataset needs:

business owner,
technical owner,
quality objective,
escalation,
and correction process.

A central data team cannot understand every domain meaning alone.

Quality incidents

Treat severe data errors as incidents:

assess affected reports and decisions,
stop propagation,
correct source or pipeline,
backfill,
communicate,
verify,
learn.

Deleting a bad dashboard result does not repair downstream exports already used.

Remediation and backfill

Correct the earliest authoritative point that contains the error, then use lineage to identify derivatives.

A repair plan may quarantine new output, patch source records, rerun affected partitions, refresh models, and notify consumers. Preserve an audit trail; silently editing a published number can make earlier decisions impossible to explain.

Quality objectives

Important datasets can define measurable objectives:

99.9 percent loaded within two hours,
no duplicate payment IDs,
refund reconciliation difference below a threshold,
required-field completeness above 99.99 percent.

Objectives focus response, but thresholds must reflect business use rather than convenient round numbers.

Data observability

Observe:

volume,
freshness,
schema,
lineage,
distribution,
and quality tests.

Connect alerts to consumer impact. One broken table may affect dozens of dashboards and models.

Manual review

Some quality cannot be fully automated:

image labels,
nuanced categories,
free-text meaning,
or ground-truth sampling.

Human review needs guidelines, sampling design, disagreement handling, and audit.

Quality score caution

One quality score can hide which dimension failed.

A dataset with 99 percent score may be unusable if the missing one percent contains every refund.

Show dimension-level evidence and intended use.

Knowledge check

How can every row be valid while a revenue dataset is poor quality?
How do accuracy and validity differ?
Why must dataset grain be explicit?
What does reconciliation detect?
Which tradeoff exists between completeness and timeliness?

The one idea to remember

Data quality means accurate, complete, consistent, timely, valid, unique, and related enough for a named decision. Define the use, enforce contracts near sources, reconcile independently, monitor distributions and lineage, and assign owners who can correct failures.