Neural Networks: Layered Mathematical Functions With Learned Weights

The name neural network invites comparisons with the brain. The historical inspiration is real, but the implemented system is a mathematical model, not a literal artificial brain.

A neural network is a layered parameterized function that transforms numeric inputs into outputs through many weighted operations.

Training adjusts the weights so the transformation becomes useful for a chosen objective.

A concrete example: recognizing a handwritten digit

An image of a handwritten 7 is represented as pixel values.

A neural network:

1. Receives those numbers.
2. Combines them through layers.
3. Produces ten output scores, one for each digit.
4. Assigns the highest score to its prediction.

During training, wrong predictions create a loss. Optimization changes weights so similar examples are more likely to receive the correct output.

No programmer writes a complete rule for every possible handwritten curve.

An artificial neuron

A simplified neuron calculates:

z = weight1 * input1
  + weight2 * input2
  + ...
  + bias
 
output = activation(z)

Weights control how strongly inputs influence the result. The bias shifts the response.

The activation function introduces nonlinearity.

Without nonlinear activations, stacking many linear layers would still behave like one linear transformation and could not model complex curved relationships.

Layers compose transformations

A network contains:

• Input representation
• One or more hidden layers
• Output layer

Each layer transforms the previous layer's representation.

For an image, early learned features may respond to local edges or texture. Later layers combine patterns useful for shapes and object categories.

This description is a helpful interpretation, not a guarantee that each unit corresponds cleanly to a human concept.

Representations are distributed across many activations.

The forward pass

During a forward pass, input flows through the network to produce output.

For a classifier:

pixels -> hidden representations -> class scores

During inference, the forward pass is the main model computation.

During training, the forward pass is followed by:

1. Loss calculation
2. Gradient computation
3. Parameter update

The same model can produce different behavior if preprocessing or output interpretation changes.

Backpropagation

Backpropagation efficiently computes how each parameter contributed to the loss using the chain rule from calculus.

It works backward from the output through the computation graph.

An optimizer then uses those gradients to adjust weights.

Backpropagation does not decide the objective, data, or architecture. It is a method for calculating parameter sensitivities.

The quality of what is learned still depends on the surrounding choices.

Deep learning

A network with many computational layers is called deep.

Depth can let models build hierarchical and reusable representations. Improvements in:

• Data
• Accelerators
• Optimization
• Architectures
• Distributed training

made deep networks practical for vision, speech, language, biology, and other domains.

More layers and parameters also increase compute, energy, memory, debugging difficulty, and risk of opaque behavior.

Scale is a resource, not a substitute for evaluation.

Convolutional networks

Convolutional neural networks apply learned filters across local regions.

For images, the same filter can detect a pattern wherever it appears, sharing parameters across positions.

This gives useful assumptions:

• Nearby pixels relate.
• Patterns can appear in different locations.

Convolution is also used for audio and other grid-like data.

Modern vision systems may use attention-based architectures too, but convolution remains an important example of architecture encoding domain structure.

Sequence models

Text, speech, and time series have order.

Recurrent neural networks process sequence state step by step. Long short-term memory networks were designed to preserve information over longer spans.

Transformers use attention to relate positions more directly and support parallel training.

Architecture determines which relationships are easy for the model to represent, but no architecture provides unlimited context or guaranteed understanding.

Attention

Attention calculates how strongly one representation should incorporate information from others.

In a sentence, a token can assign different weights to earlier tokens when building its current representation.

Self-attention operates among positions in the same sequence.

Multiple attention heads can learn different relationship patterns.

Attention weights are internal computational values. They are not always a faithful human explanation of why the final prediction occurred.

Embeddings

An embedding represents an item as a vector of numbers.

Items with useful similarities may occupy nearby regions in the learned vector space.

Embeddings can represent:

• Words or tokens
• Images
• Products
• Users
• Graph nodes

Similarity reflects the training objective and data, not an objective universal meaning.

An embedding can also encode unwanted associations and sensitive attributes.

Normalization and residual connections

Deep networks can be difficult to optimize.

Architectures use techniques such as:

• Normalization
• Residual connections
• Careful initialization
• Regularization
• Gradient clipping

A residual connection allows a layer to learn a change relative to its input rather than reconstruct everything.

These techniques improve numerical and optimization behavior. They do not remove the need for representative data or prevent semantic errors.

Overfitting and regularization

A high-capacity network can memorize training examples.

Regularization methods include:

• Weight penalties
• Dropout
• Data augmentation
• Early stopping
• Noise

Data augmentation creates plausible variations, such as crops or rotations, to teach desired invariances.

An invalid augmentation can damage learning. Flipping a handwritten digit or medical image may change its meaning.

Domain knowledge should guide transformations.

Outputs are scores, not automatic truth

A neural classifier usually produces scores transformed into probabilities or probability-like values.

The highest score becomes a prediction, but confidence can be poorly calibrated.

Inputs unlike training data may still receive confident output.

Applications should consider:

• Rejection or abstention
• Uncertainty thresholds
• Human review
• Out-of-distribution detection
• Consequence-specific decisions

The model's numeric confidence is evidence, not certainty.

Interpretability is difficult

Millions of weights interact to produce one result.

Interpretability techniques may:

• Highlight influential input regions
• Approximate local feature importance
• Inspect learned representations
• Compare counterfactual inputs
• Probe internal activations

These methods can reveal clues but may be unstable or incomplete.

An explanation tool should itself be validated. A visually persuasive heat map can create confidence without proving causal reasoning.

For high-stakes use, combine interpretability with performance testing, constraints, audit, and human process.

Neural networks are not always the best model

For small structured datasets, methods such as linear models or decision trees may be:

• Easier to train
• More interpretable
• Less expensive
• Equally or more accurate

Neural networks are especially useful when large data and complex unstructured patterns justify their capacity.

Start with a meaningful baseline. Complexity should earn its operational and explanatory cost.

Knowledge check

1. What does an artificial neuron calculate?
2. Why do neural networks need nonlinear activation functions?
3. What is the difference between a forward pass and backpropagation?
4. What does an embedding represent?
5. Why is a model's high output confidence not proof that it is correct?

The one idea to remember

A neural network is a trainable layered mathematical function. Learned weights create useful representations, but capability comes from data, architecture, objectives, and optimization, not from a literal artificial brain.