Explainable AI: Evidence for a Particular Audience and Decision

An AI explanation is useful only if it helps someone perform a real task: review a case, challenge an outcome, debug a model, assess compliance, or decide whether to trust a system.

Explainability is audience-specific evidence about system behaviour, not a universal picture of what a model “thought.”

Different people need different scope, language, fidelity, and actionability.

A concrete example: loan decision

A loan reviewer may need:

the case data used,
major factors affecting the score,
missing or stale fields,
uncertainty,
applicable policy,
and similar cases.

An applicant may need the principal reasons, data correction, and realistic recourse. A regulator needs population outcomes, validation, governance, and consistency.

One explanation does not serve all three.

Start with the question

Possible explanation questions include:

Why this output?
Why not another output?
What would change it?
Which data was used?
How does the model generally behave?
Where does it fail?
Who approved its use?

Choose methods after defining the question.

Local explanations

A local explanation describes one prediction or decision.

It may show:

influential features,
a nearby example,
a rule approximation,
a counterfactual,
or evidence retrieved.

Local explanations do not necessarily describe global model behaviour.

Global explanations

Global explanations describe patterns across the model:

overall feature effects,
important interactions,
performance by segment,
decision boundaries,
common failures,
and sensitivity.

They help developers, risk teams, and regulators understand the system as a whole.

Intrinsically interpretable models

Some models have structures people can inspect more directly:

short decision trees,
sparse linear models,
scoring systems,
and constrained rule lists.

Interpretability decreases as features, interactions, and preprocessing become complex. A simple model using opaque features may not be understandable.

Use the simplest system that meets validated needs.

Post-hoc explanations

Post-hoc methods explain a trained model after the fact.

They can perturb inputs, fit a local approximation, assign contribution values, retrieve examples, or visualize internal patterns.

Because they are approximations, test fidelity and stability. A plausible explanation can be wrong about the actual model.

Feature attribution

Feature-attribution methods assign contributions to input features.

Results depend on:

baseline,
feature dependence,
model behaviour,
and method.

Correlated features can split or transfer importance unexpectedly. Attribution shows model influence under assumptions, not causal effect in the world.

Counterfactual explanations

A counterfactual describes a nearby hypothetical input that would produce a different output.

“If verified income were higher by this amount, the application would meet the threshold” can support recourse.

Counterfactuals should respect feasible, lawful, and controllable changes. Telling someone to change age or ethnicity is not useful recourse.

Example-based explanations

Showing similar prior cases can make a prediction concrete.

Similarity must be meaningful, privacy-protective, and not expose another person's sensitive data. Historical decisions may reflect past bias.

Explain why the example is considered similar.

Uncertainty

An explanation should communicate uncertainty in prediction, evidence, and explanation method.

Avoid presenting a ranked list of factors as exact truth. Show confidence, missing data, out-of-distribution warnings, and cases where the method is unreliable.

Uncertainty should influence action.

Data provenance

People need to know:

which data source,
when it was collected,
how it was transformed,
and whether it can be corrected.

An accurate model explanation built on wrong personal data does not resolve the decision problem.

Explanation versus justification

An explanation describes how the system produced an output. A justification argues that the decision is appropriate under policy or values.

A model may accurately use income and still violate a policy that forbids that use. Do not let technical explanation substitute for legitimacy.

Explanation versus causality

Model associations are not necessarily causal.

A feature can predict an outcome because it is a proxy or consequence. Changing it may not change the real-world result.

Use causal evidence when making intervention claims.

Security and gaming

Detailed explanations can help legitimate challenge but may also enable fraud, evasion, model extraction, or exposure of confidential information.

Provide enough evidence for accountability while limiting unnecessary internals. High-risk decisions can use secure reviewer access and monitored appeals.

Security cannot justify a meaningless explanation.

Evaluate explanations

Test whether the target audience can:

understand the output,
identify data errors,
make a better decision,
predict model behaviour,
choose appropriate recourse,
and recognize uncertainty.

Also test fidelity, consistency, privacy, and subgroup differences. Visual appeal is not evidence of usefulness.

Documentation

Explain the broader system through:

intended use,
prohibited use,
training and evaluation data,
performance,
limitations,
monitoring,
owners,
and appeal.

Case-level explanation belongs inside this system context.

Knowledge check

How do local and global explanations differ?
Why can feature attribution be misleading with correlated inputs?
What makes a counterfactual actionable?
How does explanation differ from justification?
Which tests show an explanation is useful?

The one idea to remember

Explainable AI is not one diagram of model thought. Start with the audience and decision, then provide faithful evidence about inputs, behaviour, uncertainty, provenance, limitations, and recourse while distinguishing prediction from causality and explanation from legitimacy.