MLOps: Operating Data, Models, and Code as One System

A machine-learning model file is only one component of a production prediction system.

Its behavior depends on training data, feature code, labels, parameters, libraries, serving infrastructure, thresholds, and current environment.

MLOps applies reproducibility, testing, delivery, monitoring, lineage, and governance to data and models as well as ordinary software code.

The objective is a controlled model lifecycle, not maximum pipeline automation.

A concrete example: one model release

A release record includes:

training dataset snapshot,
feature definitions,
label version,
source commit,
dependencies,
hyperparameters,
evaluation metrics,
model artifact digest,
approval,
deployment configuration,
live performance.

The team can reproduce, compare, and roll back the decision.

Data pipeline

Training starts with governed data ingestion:

source,
schema,
quality,
lineage,
privacy,
retention.

A model trained on silently incomplete data is not fixed by reliable serving.

Feature pipeline

Features need consistent calculation offline and online.

Manage:

definitions,
event-time cutoff,
freshness,
ownership,
backfill,
and training-serving parity.

Feature stores can centralize this lifecycle.

Label pipeline

Labels may come from:

business outcomes,
human review,
delayed events,
or adjudication.

Version definitions, maturity, quality, and corrections.

Changing the label changes the task.

Experiment tracking

Track each training run:

code,
data,
parameters,
metrics,
environment,
artifacts,
notes.

Experiment tracking prevents "best model on someone's laptop" from becoming an unrepeatable release.

Reproducibility

Reproduce:

data snapshot or query,
code,
package versions,
random seeds,
hardware where relevant,
configuration,
and preprocessing.

Some distributed training remains nondeterministic, so define acceptable equivalence.

Training orchestration

Pipelines coordinate:

extract,
validate,
transform,
train,
evaluate,
package,
and register.

Retries must be idempotent, and expensive stages should reuse verified outputs safely.

Model registry

A registry stores model versions and metadata:

artifact,
metrics,
lineage,
stage,
owner,
approval,
known limitations.

Registration does not equal production approval.

Evaluation gates

Before release, compare against:

baseline,
current champion,
minimum accuracy,
calibration,
fairness,
robustness,
latency,
and cost.

One aggregate metric should not decide every risk.

Model packaging

Package model with:

preprocessing,
schema,
runtime dependencies,
and prediction contract.

Separate threshold and policy when they need independent governance, but version their combination.

Batch inference

Batch scoring processes datasets on a schedule.

It needs:

model version,
input snapshot,
idempotent output,
deadline,
and backfill.

Users should know how fresh scores are.

Online inference

Online serving responds per request.

It requires:

low latency,
autoscaling,
schema validation,
feature retrieval,
timeout,
fallback,
and observability.

The model may be only a fraction of total latency.

Deployment strategies

Use:

shadow,
canary,
A/B test,
regional wave,
or blue-green.

Shadow compares output without affecting decisions. Canary limits operational and outcome exposure.

Model and policy

The deployed decision combines:

model score,
threshold,
business rule,
human review,
and fallback.

Monitor the full policy outcome, not model accuracy in isolation.

Monitoring

Track:

service latency and errors,
feature quality,
input drift,
prediction distribution,
calibration,
outcome performance,
subgroup behavior,
cost.

Labels may arrive late, requiring layered signals.

Retraining

Retraining can be:

scheduled,
triggered by mature data,
performance decline,
or product change.

Automatic retraining should not automatically deploy without evaluation and governance appropriate to impact.

Continuous training risk

A broken source can continuously produce worse models.

Use data-quality gates, baseline comparisons, approval, and rollback.

Automation scales both correctness and error.

Lineage

Connect prediction and model version back to:

data,
features,
labels,
code,
parameters,
and approval.

Lineage supports audits, debugging, and affected-prediction analysis after a defect.

Governance

High-impact models need:

intended use,
prohibited use,
owner,
documentation,
fairness review,
privacy,
human oversight,
appeal,
and retirement criteria.

Governance should be part of the pipeline, not a detached document.

Security

Protect:

training data,
model artifacts,
registry,
serving endpoint,
feature store,
and pipeline credentials.

Models can leak data or be manipulated through poisoned inputs.

Rollback

Preserve:

previous model,
feature definitions,
runtime,
threshold,
routing.

A rollback model may expect an older feature schema. Test compatibility before an incident.

Retirement

Remove:

endpoint,
scheduled jobs,
monitoring,
credentials,
stored artifacts according to policy,
and downstream dependencies.

Archive required evidence without keeping an abandoned decision system alive.

Assign ownership for every model release

Automation does not remove accountability. Each production model needs named owners for:

training data and labels,
feature pipelines,
model behavior,
serving infrastructure,
business policy,
security and privacy,
and incident response.

A release record should state who approved the model, which evidence they reviewed, what limitations remain, and when the next review is due. Ownership must survive team reorganizations; an abandoned model is still making live decisions.

Use operational service objectives where they fit. Online models may need latency and availability targets, while batch models need completion deadlines and reconciliation checks. Add model-specific objectives such as maximum stale-feature rate, minimum outcome coverage, or a deadline for receiving mature labels. These make “healthy” observable rather than subjective.

Separate retraining from release approval

A pipeline can retrain frequently while promotion remains controlled. Compare the candidate with the active model, run data and behavior tests, inspect important segments, verify policy compatibility, and use staged rollout. Automatic retraining should produce a candidate and evidence, not an unquestioned production decision.

Knowledge check

Why is a model artifact insufficient to reproduce a release?
What does a model registry contain?
How do batch and online inference differ operationally?
Why should continuous training not imply automatic deployment?
What must model rollback preserve besides weights?

The one idea to remember

MLOps operates the complete lifecycle joining data, features, labels, code, experiments, models, policy, deployment, monitoring, retraining, governance, rollback, and retirement. Reproducibility and controlled evidence matter more than automating every step.