Training, Models, and Inference: Creating and Applying Learned Systems

A machine-learning system has at least two distinct phases.

Training adjusts a model's parameters using data and an objective; inference applies the trained model to new input to produce an output.

Training may take hours or months on specialized hardware. Inference may need to answer one user request in milliseconds.

The two phases share a model but have different engineering constraints.

A concrete example: image classification

During training:

1. Load many labeled images.
2. Transform them into numeric tensors.
3. Run them through the model.
4. Compare predictions with labels.
5. Calculate a loss.
6. Adjust model parameters.
7. Repeat across the dataset.

During inference:

1. Receive one new image.
2. Apply the same required preprocessing.
3. Run the final parameters.
4. Return class scores.

Inference does not normally adjust the model. It uses what training produced.

Data preparation

Before training, teams may:

• Collect examples
• Remove duplicates
• Correct labels
• Normalize formats
• Split training and evaluation sets
• Balance or weight classes
• Remove prohibited data
• Document provenance

Data preparation often determines more of model quality than changing one algorithm setting.

Cleaning must preserve meaningful variation. Removing every unusual example can produce a tidy dataset and a model that fails on real edge cases.

The model is a parameterized function

A model maps inputs to outputs:

output = model(input, parameters)

The architecture defines how calculations connect. Parameters are the adjustable numeric values learned during training.

A linear model has relatively few parameters. A deep neural network may contain billions.

The saved model artifact may include:

• Architecture
• Parameter values
• Tokenizer or feature definitions
• Normalization statistics
• Label mapping
• Configuration

Without matching preprocessing, the parameters may be unusable.

The objective and loss

The training objective expresses what should improve.

A loss function produces a number representing prediction error for the current examples. Optimization seeks parameter values with lower expected loss.

Examples:

• Classification loss
• Squared error for numeric prediction
• Token prediction loss
• Ranking loss

The loss is a mathematical proxy for the real goal.

Reducing average error may still leave unacceptable failures for a rare subgroup or high-impact case. Evaluation needs several measures beyond training loss.

Optimization

Training commonly uses gradient-based optimization.

A simplified loop is:

1. Make predictions.
2. Calculate loss.
3. Compute how parameter changes would affect loss.
4. Update parameters a small amount.

The learning rate controls update size.

Too large can make training unstable. Too small can be slow or become stuck.

Optimizers, schedules, initialization, batch size, and regularization all influence the path through a complex parameter landscape.

Epochs, steps, and batches

A batch is a subset of training examples processed together.

A step is one parameter update.

An epoch is one pass through the training dataset, though sampling methods can make the definition less exact.

Batches allow efficient parallel hardware use and provide a noisy estimate of the full dataset's gradient.

Larger batches use more memory and can change optimization behavior. More epochs do not always improve generalization; they can increase overfitting.

Checkpoints

Training periodically saves checkpoints containing model and optimizer state.

Checkpoints allow:

• Recovery after hardware failure
• Comparison across training stages
• Early stopping
• Fine-tuning from an intermediate model
• Reproducibility

They can be large and sensitive. A checkpoint may encode information learned from private data and can have significant commercial value.

Access, retention, provenance, and secure storage matter.

Pretraining and fine-tuning

Large models may first undergo broad pretraining on a large dataset and general objective.

Then fine-tuning adjusts them for a narrower task or preferred behavior using additional data.

Transfer learning allows useful representations to be reused, reducing the data and compute needed for each application.

Fine-tuning does not erase every behavior from pretraining. Evaluation should include inherited capabilities, biases, security concerns, and domain-specific errors.

Validation guides selection

Teams evaluate candidate checkpoints and hyperparameters on validation data.

They may choose:

• Architecture
• Learning rate
• Number of training steps
• Decision threshold
• Regularization
• Fine-tuning dataset

Repeated decisions based on the same test set leak information into development.

A final test set should remain independent until the major design is fixed, followed by continued production monitoring.

Inference serving

During inference, the system must:

• Receive and validate input
• Apply preprocessing
• Run the model
• Convert outputs into useful results
• Enforce policy
• Return within latency limits

The model may run:

• On a phone
• In a browser
• On a server CPU
• On a GPU or accelerator
• Across several machines

Placement affects privacy, latency, energy, offline use, update control, and cost.

Latency and throughput

Inference systems balance:

• Latency: Time for one result
• Throughput: Results processed per unit time

Batching several requests improves hardware efficiency but may make the first request wait for others.

Interactive assistants need low user-visible latency. Offline document processing can favor throughput.

Service objectives should include preprocessing, queueing, networking, and postprocessing, not only model execution.

Quantization and compression

Models can be made smaller or faster through:

• Quantization
• Pruning
• Distillation
• Architecture optimization

Quantization stores or computes parameters at lower numeric precision. It can reduce memory and increase speed, with possible quality loss.

Distillation trains a smaller student model to imitate a larger teacher.

Optimization must be evaluated on important cases. A small average accuracy change can hide larger damage for certain languages or input types.

Online, batch, and edge inference

Online inference responds to individual requests quickly.

Batch inference processes many records on a schedule, such as nightly demand predictions.

Edge inference runs on a local device close to data collection.

Edge execution can improve privacy, offline function, and latency, but devices have limited power, memory, and update consistency.

The same product may combine all three modes.

Model and data versioning

A prediction should be traceable to:

• Model version
• Code version
• Feature or tokenizer version
• Input data version or timestamp
• Configuration
• Decision threshold

Changing a tokenizer while keeping the same model weights can silently corrupt results.

Versioning enables rollback, audit, comparison, and reproduction.

The model is one artifact inside a larger versioned pipeline.

Production drift and monitoring

After deployment, monitor:

• Input distributions
• Missing features
• Prediction distributions
• Latency and errors
• Resource use
• Outcome quality when labels arrive
• Performance across important groups
• Abuse and adversarial inputs

Ground-truth labels may arrive weeks later or never arrive automatically.

Proxy metrics can help but may not prove model quality. Human review and targeted sampling remain valuable.

Retraining is a controlled release

New data does not automatically justify automatic retraining and deployment.

The data may contain:

• Model-influenced feedback
• Attack inputs
• Temporary anomalies
• Incorrect labels
• Changed policy

A retraining pipeline should validate data, compare candidate performance, run safety checks, preserve rollback, and record approval.

Model updates are software and policy changes with user impact.

Knowledge check

1. What changes during training that normally remains fixed during inference?
2. Why must preprocessing be versioned with a model?
3. What is the role of a checkpoint?
4. How do latency and throughput create a batching tradeoff?
5. Why should retrained models pass controlled release checks?

The one idea to remember

Training creates or tunes a model by optimizing parameters against data and an objective. Inference applies that frozen artifact, but reliable use depends on matching preprocessing, serving constraints, versioning, monitoring, and controlled updates.