Logs, Metrics, and Traces: Complementary Evidence from Running Systems

A production system leaves many kinds of evidence.

One stream of text cannot efficiently answer every question. Operators need to know whether a problem is broad, which request path is slow, and what exact event occurred.

Metrics show how much, traces show where work traveled, and logs explain discrete events and context.

The three signals complement each other when they share identifiers and consistent meaning.

A concrete example: slow checkout

An alert reports high checkout latency.

The investigation moves through:

1. metrics show the slowdown began after a deployment and affects one region,
2. traces show most time is spent in the order database,
3. logs identify a new query and the affected request context,
4. database metrics confirm lock contention.

Each signal narrows a different part of the question.

What is observability?

Observability is the ability to understand internal system behavior from external evidence.

It is not synonymous with installing a monitoring product. Useful observability depends on:

• intentional instrumentation,
• stable semantics,
• context propagation,
• accessible queries,
• and evidence tied to user outcomes.

Logs

Logs record discrete events:

• request failed,
• job started,
• permission denied,
• migration completed,
• dependency timed out.

They can include rich context but become expensive and difficult to query at large volume.

Structured logs

Structured logs use fields rather than only prose:

event=payment_failed
order_id=8472
provider=example_pay
error_code=timeout
trace_id=abc123

Fields support filtering and aggregation. Keep human-readable messages, but do not force machines to parse changing sentences for critical meaning.

Log levels

Common levels include:

• debug,
• info,
• warning,
• error,
• critical.

Use severity consistently. If normal recoverable events appear as errors, alerting and investigation become noisy. Debug volume should be controllable without redeploying where practical.

Sensitive logs

Do not log:

• passwords,
• authentication tokens,
• full payment data,
• secret keys,
• or unnecessary personal content.

Redaction should occur before data enters the logging pipeline. Access and retention should reflect the sensitivity that remains.

Metrics

Metrics represent numerical observations over time.

Examples:

• request count,
• error count,
• latency histogram,
• CPU,
• queue depth,
• active users,
• completed orders.

They are efficient for dashboards, trends, objectives, and alerts.

Metric types

Common types include:

• counter: increases, such as requests,
• gauge: current value, such as queue depth,
• histogram: distribution, such as latency,
• summary: precomputed distribution statistics in some systems.

Choosing the right type determines which questions can be answered later.

Labels and cardinality

Metric labels divide a measurement by dimensions:

• method,
• route,
• region,
• status,
• version.

Do not use unbounded values such as user ID or request ID as metric labels. High cardinality creates enormous time-series counts and cost.

Detailed identifiers belong in logs or traces.

Traces

A distributed trace follows one operation across components.

A trace contains spans representing units of work:

• browser request,
• API handler,
• database query,
• queue publication,
• downstream call.

Parent-child relationships show the critical path and concurrency.

Trace context

Trace identifiers must propagate through:

• HTTP headers,
• message metadata,
• background jobs,
• and internal calls.

Losing context at a queue or thread boundary breaks the end-to-end view.

Trust boundaries should validate external tracing headers to prevent abuse or accidental collision.

Sampling

Recording every trace can be expensive.

Sampling strategies include:

• fixed percentage,
• rate limit,
• always sample errors,
• keep slow traces,
• or tail-based selection after outcome is known.

Sampling must preserve enough representative and rare evidence for diagnosis.

Correlation

Signals become more useful when they share:

• service name,
• environment,
• version,
• region,
• trace ID,
• and business operation identifiers.

A metric exemplar can link a latency spike to a representative trace, which links to contextual logs.

Resource versus business signals

CPU and memory explain infrastructure pressure.

Business signals show whether users succeed:

• checkout completion,
• report delivery,
• authentication success,
• or payment confirmation.

Healthy CPU does not prove healthy users, and high CPU may be expected during useful work.

Instrumenting errors

Classify errors by:

• operation,
• type,
• retryability,
• dependency,
• user impact,
• and version.

Do not rely only on exception text, which changes and can contain sensitive values.

Expected validation rejection should not be grouped with server failure.

Retention

Evidence has different value over time.

Keep:

• high-resolution recent metrics,
• aggregated long-term trends,
• logs according to operational and legal needs,
• and sampled traces for investigation.

Infinite retention increases cost and privacy risk.

Clock and timestamps

Use synchronized clocks and consistent time zones, typically UTC.

Distributed clocks remain imperfect, so trace parent relationships and sequence identifiers can be more reliable than timestamps alone for causality.

Include event occurrence time and ingestion time when delayed delivery matters.

Instrumentation overhead

Telemetry consumes:

• CPU,
• memory,
• network,
• storage,
• and engineering attention.

Instrumentation should fail safely and avoid blocking critical application work when the telemetry backend is unavailable.

Measure the observability system itself.

Dashboards

A useful dashboard answers one operational question:

• Is checkout healthy?
• Is queue backlog recovering?
• Did the release change latency?

Organize by user outcome and dependency, with links to deeper evidence. A wall of unrelated graphs is not situational awareness.

Open standards

Common semantic conventions and protocols improve portability and correlation.

Standardization does not remove the need to define application-specific events and business measures.

Instrumentation names are contracts consumed by dashboards, alerts, and objectives.

Knowledge check

1. Which question is each of logs, metrics, and traces best suited to answer?
2. Why are structured logs easier to operate?
3. What problem does high-cardinality metric labeling create?
4. Why must trace context cross queue boundaries?
5. How should telemetry behave when its backend is unavailable?

The one idea to remember

Use metrics to summarize behavior, traces to follow individual work across components, and logs to preserve discrete context. Connect the signals with stable identifiers, protect sensitive data, control cardinality and cost, and instrument user outcomes as well as machines.