Analytics and Causal Questions: Association Is Not Intervention

Analytics describes patterns:

• users with notifications stay longer,
• one region buys more,
• premium customers contact support less.

Those patterns do not automatically reveal what would happen if the product changed something.

Correlation describes association; causal inference asks how an outcome would change under an intervention.

The missing comparison is the counterfactual: what would have happened to the same unit without the intervention?

A concrete example: notification retention

Users who enable notifications retain at a higher rate.

Possible explanations:

• notifications improve retention,
• already engaged users enable notifications,
• certain device users differ,
• a campaign affects both enablement and retention.

The observed association alone cannot separate them.

Causal question

State the intervention:

For eligible new users, what is the effect of enabling a reminder notification on 30-day retention?

Define:

• population,
• treatment,
• comparison,
• outcome,
• time horizon.

Vague "impact" questions produce vague evidence.

Counterfactual

For one user, only one path is observed:

• notification enabled,
• or not enabled.

The unobserved alternative is the counterfactual.

Causal methods construct credible comparison groups to estimate it.

Confounding

A confounder influences both treatment and outcome.

Engagement can make users:

• more likely to enable notifications,
• more likely to retain.

Comparing enabled versus disabled users attributes some existing engagement difference to the feature.

Selection bias

The analyzed sample may not represent the target population.

Examples:

• only survey responders,
• only surviving customers,
• only users who completed onboarding,
• only records with complete data.

Selection can create or hide associations.

Reverse causality

The assumed direction can be reversed.

Customers may contact support less because they are satisfied, or appear satisfied because they had no difficult problem requiring support.

Time ordering helps but does not eliminate confounding.

Randomized experiments

Random assignment makes treatment independent of preexisting characteristics on average.

Differences in outcome can then be attributed more credibly to the assigned intervention, assuming:

• compliance,
• no interference,
• correct measurement,
• and valid analysis.

Randomization is the strongest common product method.

Observational adjustment

When experiments are impossible, analysts can adjust for measured confounders through:

• matching,
• regression,
• weighting,
• stratification.

Unmeasured confounders can remain.

More variables do not guarantee unbiased estimation.

Causal diagrams

A directed acyclic graph can represent assumptions:

• treatment,
• outcome,
• confounders,
• mediators,
• colliders.

The diagram helps decide which variables to adjust for.

It makes assumptions visible rather than letting a model choose blindly.

Mediators

A mediator lies on the causal path:

notification -> return visits -> retention

Adjusting for return visits can remove part of the effect being estimated.

Decide whether the question asks total or direct effect.

Colliders

A collider is influenced by two variables.

Conditioning on it can create a false association. For example, selecting only users who contacted support may connect otherwise unrelated product problems and user behavior.

Not every available variable should enter a model.

Difference in differences

This method compares change over time between treated and comparison groups.

It can help when one region adopts a policy before another, assuming trends would otherwise have remained parallel.

Check pre-treatment trends and concurrent changes.

Regression discontinuity

If treatment follows a threshold, compare units just above and below it.

Example:

• accounts above a score receive assistance.

Near the threshold, units may be similar except treatment, if the score cannot be manipulated precisely.

Instrumental variables

An instrument affects treatment but influences outcome only through treatment under strong assumptions.

Valid instruments are difficult to find and explain.

The method does not rescue arbitrary observational data.

Natural experiments

Policy changes, outages, or rollout boundaries can create quasi-random variation.

They offer useful evidence when assignment is plausibly unrelated to outcome drivers.

Document why the comparison is credible.

Sensitivity analysis

Observational conclusions depend on assumptions about unmeasured confounding, model form, and sample selection.

Sensitivity analysis asks how strong an unmeasured factor or alternate specification would need to be to change the conclusion. A result that disappears under small reasonable changes deserves less confidence.

Negative controls

A negative-control outcome or exposure should not be causally affected under the proposed explanation.

If the analysis "finds" an effect there too, residual bias or measurement error may remain. Negative controls do not prove validity, but they can falsify overly confident designs.

Interference

One person's treatment can affect another:

• social network feature,
• marketplace pricing,
• collaboration tool.

Standard independent-user experiments can underestimate or distort effects.

Randomize clusters or model network spillover.

Heterogeneous effects

Average effect can hide differences:

• new versus experienced users,
• regions,
• accessibility needs,
• device types.

Subgroup analysis should be planned and supported by enough data to avoid storytelling from noise.

Measurement

The chosen metric must represent the outcome.

Clicks can rise while satisfaction falls. Retention can improve through manipulative lock-in.

Use guardrails and long-term measures.

Prediction versus causation

A prediction model asks:

Who is likely to churn?

A causal model asks:

Who would avoid churn if we intervene?

The most predictable high-risk user is not always the most treatable.

External validity

An effect estimated for one population, time, and implementation may not transfer elsewhere.

Record who entered the study, product version, market conditions, and treatment delivery. Replication and staged rollout test whether the causal result generalizes to the decision now being considered.

Knowledge check

1. What counterfactual does a causal question need?
2. How can engagement confound notification analysis?
3. Why can adjusting for a mediator change the question?
4. What assumption supports difference in differences?
5. How do prediction and causal targeting differ?

The one idea to remember

Analytics can reveal associations, but causal claims require a credible comparison for what would happen under another intervention. Randomization is strongest; observational methods depend on explicit assumptions about confounding, selection, timing, interference, and measurement.