Choosing Technology with Evidence: Match the Tool to the Real Problem

Popular technology can be excellent and still be wrong for a particular workload, team, or risk.

Choose technology by defining the problem and constraints, testing uncertain assumptions with representative evidence, and including operation and exit in the decision.

Reputation creates a shortlist, not a conclusion.

A concrete example: database selection

Candidates are compared using:

• actual query shapes,
• data size and growth,
• write patterns,
• consistency,
• failure needs,
• backup and restore,
• team skills,
• compliance,
• and total cost.

A generic “transactions per second” benchmark cannot represent the application.

Define the outcome

State what the system must enable:

• user workflow,
• business capability,
• service objective,
• and scale.

Avoid starting with “Which vector database?” when the real question is how to retrieve authorized current documents for a support answer.

Requirements and constraints

Separate:

• must have,
• should have,
• could have,
• and prohibited.

Constraints can include:

• latency,
• availability,
• consistency,
• data location,
• budget,
• team,
• hardware,
• accessibility,
• licences,
• and deadline.

Challenge requirements that are inherited without evidence.

Workload

Describe:

• request volume,
• distribution,
• peak,
• input size,
• read/write mix,
• concurrency,
• growth,
• regions,
• and failure consequences.

Averages hide peaks and rare high-cost operations.

Use production traces or realistic synthetic data.

Shortlist credible options

Use:

• official documentation,
• architecture fit,
• support lifecycle,
• community,
• vendor health,
• security,
• and comparable deployments.

Keep the shortlist small enough for serious evaluation. Include the current system and “do nothing” where appropriate.

Test the risky assumptions

A proof of concept should target what might invalidate the choice:

• scale,
• integration,
• consistency,
• latency,
• model quality,
• migration,
• permission filtering,
• or team operability.

Do not spend the prototype polishing features common to every option.

Representative benchmarks

Benchmark:

• real data shape,
• realistic hardware,
• expected concurrency,
• warm and cold states,
• failure,
• and production configuration.

Include end-to-end latency and cost. Report percentiles and variance.

Publish scripts and assumptions internally so results can be reproduced.

Correctness before speed

A fast result is useless if it violates consistency, permission, or quality.

Validate:

• expected output,
• edge cases,
• data integrity,
• authorization,
• recovery,
• and numerical or model accuracy.

Only compare performance among options that meet required correctness.

Operability

Ask whether the team can:

• deploy,
• monitor,
• debug,
• scale,
• patch,
• back up,
• restore,
• and respond to incidents.

A technology that requires rare expertise has hiring, support, and continuity cost.

Try an incident or restore exercise during evaluation.

Security and privacy

Assess:

• identity,
• access control,
• encryption,
• secrets,
• isolation,
• audit,
• vulnerability response,
• data lifecycle,
• and supply chain.

Features on a checklist may differ in quality and edition. Test the exact deployment and contract.

Reliability

Evaluate:

• failure modes,
• regional architecture,
• quotas,
• consistency during failure,
• backup,
• restore,
• support,
• and service commitments.

Provider uptime does not equal application availability if architecture uses one fragile configuration.

Ecosystem and maturity

Maturity can provide:

• documentation,
• libraries,
• integrations,
• expertise,
• known failure modes,
• and stable upgrades.

New tools can offer important capability but carry unknowns. Match novelty to strategic value and risk tolerance.

Team fit

Consider existing skills, learning time, hiring, and development workflow.

Do not choose only what the current team already knows if it cannot meet the need. Do not choose fashionable novelty while ignoring who will operate it at 3 a.m.

Training and support belong in the plan.

Total cost

Include:

• licences,
• usage,
• infrastructure,
• people,
• integration,
• support,
• security,
• downtime,
• compliance,
• growth,
• and migration.

Model several usage scenarios and price changes.

“Open source” and “managed” shift costs rather than eliminating them.

Lock-in and exit

Inspect:

• proprietary APIs,
• data format,
• egress,
• contracts,
• identity,
• skills,
• and integrated services.

Test export and migration for critical data. Accept lock-in when differentiated value exceeds measured exit risk.

Decision matrix

A weighted matrix can make criteria and tradeoffs visible.

Avoid false precision:

• define scoring evidence,
• review weights with stakeholders,
• note uncertainty,
• and include disqualifying constraints.

The matrix supports judgment; it does not replace it.

Architecture decision record

Record:

• context,
• options,
• evidence,
• decision,
• consequences,
• assumptions,
• and review triggers.

Future teams can revisit when scale, pricing, regulation, or product needs change.

Pilot and rollout

After selection, stage adoption:

• bounded workload,
• shadow traffic,
• one team,
• or one region.

Monitor quality, cost, incidents, and support. Preserve rollback until the new system proves itself.

Avoid benchmark theatre

Warning signs include:

• vendor-authored comparison only,
• tiny toy data,
• one average number,
• missing configuration,
• no correctness test,
• no failure test,
• and changing criteria after results.

Good evidence makes limitations visible.

Build an evidence scorecard

For each criterion, record:

• claim,
• required threshold,
• evidence source,
• test environment,
• observed result,
• uncertainty,
• owner,
• and decision impact.

Distinguish vendor documentation, internal measurement, external reference, and assumption.

An option should not receive a high reliability score merely because it advertises replication. Evidence might require a restore exercise and a region-failure test in the proposed configuration.

The scorecard also reveals where two options are effectively equal and where more research could genuinely change the choice.

Knowledge check

1. Why should selection begin with an outcome?
2. What should a proof of concept target?
3. Why must correctness precede performance comparison?
4. Which factors determine operability?
5. What should an architecture decision record preserve?

The one idea to remember

Choose technology against a real workload, constraints, team, and lifecycle. Test the assumptions most likely to fail, compare only correct solutions, include operation, security, cost, and exit, then record the evidence and revisit when conditions change.