Algorithms and Recommendation Systems: Ranking What Appears Next

Two people opening the same video, shopping, or social application may see different items.

That difference is produced by a recommendation pipeline.

A recommendation system selects and ranks candidate items using predictions, context, constraints, and platform objectives.

It is rarely one mysterious algorithm. It is a sequence of data collection, retrieval, scoring, filtering, experimentation, and policy decisions.

A concrete example: choosing the next video

A video platform may have millions of possible videos.

It cannot run its most expensive model on all of them for every page.

Instead:

1. Generate a few thousand candidates.
2. Remove ineligible or unsafe items.
3. Predict several outcomes for each candidate.
4. Combine scores and business rules.
5. Add diversity and freshness.
6. Return a ranked list.

The final feed reflects both user signals and platform choices.

Candidate generation

Candidate generation reduces a huge catalog to a manageable set.

Sources can include:

• Items similar to recent activity
• Popular items
• Content followed by the user
• Trends in a region
• New releases
• Editorial selections
• Paid promotions
• Results from embedding similarity

Using several sources avoids dependence on one signal.

If an item never enters the candidate set, the ranking model cannot show it no matter how relevant it would have been.

Collaborative filtering

Collaborative filtering uses behavior across users and items.

A simplified idea is:

People with patterns similar to yours engaged with these items.

It can discover relationships without understanding item content.

Challenges include:

• New users with no history
• New items with no interactions
• Popularity dominating niche interests
• Behavior reflecting exposure rather than true preference
• Sensitive inferences from similar users

Interaction does not always mean approval. A person may watch a video because it is upsetting or because autoplay began it.

Content-based recommendation

Content-based methods use item attributes:

• Topic
• Genre
• Text
• Image
• Creator
• Price
• Technical features

If a user reads database articles, the system can recommend other database material based on content similarity.

This helps new items without interaction history.

It can also narrow recommendations too much, repeatedly offering close variants of what the user already knows.

Hybrid systems combine collaborative and content signals.

Embeddings and similarity

Models can represent users and items as vectors in an embedding space.

Nearby vectors indicate similarity useful for the training objective.

An item embedding may reflect:

• Content
• Audience behavior
• Context
• Learned latent properties

Vector search retrieves nearby candidates efficiently.

Similarity is not neutral. If training optimizes clicks, the embedding captures patterns useful for clicks, including biases and exposure effects.

Ranking predicts outcomes

A ranking model may estimate:

• Probability of clicking
• Expected watch time
• Chance of purchase
• Likelihood of hiding the item
• Long-term retention
• User satisfaction survey result

The system combines these signals into a score.

A model optimized only for click probability may favor sensational titles. A watch-time objective may favor long content whether or not it is useful.

The objective strongly shapes the environment users experience.

Multi-objective ranking

Real systems balance several goals:

• Relevance
• Freshness
• Diversity
• Safety
• Creator fairness
• Revenue
• User control
• Platform health

One approach combines weighted scores. Another applies hard eligibility constraints before ranking.

Tradeoffs are value decisions. Giving safety a weight of zero is a choice; so is deciding which content counts as unsafe.

Technical optimization cannot decide those values without governance.

Cold-start problems

A cold-start user has little behavior history.

The system can use:

• Explicit onboarding choices
• Current context
• Regional popularity
• Broad exploration
• Demographic data where appropriate and permitted

A cold-start item has no interactions.

Content features, editorial review, and dedicated exploration help new items receive exposure.

Without exploration, already popular items keep collecting data while new content remains invisible.

Exploration and exploitation

Exploitation shows items the system already predicts will work well.

Exploration tries uncertain items to learn preferences and discover new value.

Too much exploitation creates repetition and limits learning. Too much exploration produces irrelevant results.

Bandit algorithms and controlled experiments manage this balance.

Exploration has ethical constraints. A medical or safety platform should not experiment with harmful content merely to collect engagement data.

Diversity and serendipity

The highest-scoring ten items may all be nearly identical.

Reranking can introduce diversity across:

• Topic
• Creator
• Format
• Viewpoint
• Familiarity
• Time

Serendipity offers useful items that are not obvious repeats.

Diversity should not be cosmetic. Showing ten creators who all represent the same narrow source can still produce a limited information environment.

Define what kind of diversity serves the product and users.

Context matters

Recommendations can use:

• Time of day
• Device
• Location
• Current page
• Session behavior
• Network condition
• Recent intent

A user may want energetic music during exercise and quiet music at night.

Context improves relevance but increases privacy and inference concerns.

Collect only what has a justified purpose, and distinguish temporary session intent from a permanent user profile.

Feedback loops

Recommendations influence what users see, which influences what they click, which becomes future training data.

This loop can amplify:

• Popularity
• Stereotypes
• Extremes
• One creator's advantage
• Past mistakes

Observed behavior is not an unbiased measure of preference because the system controlled exposure.

Counterfactual evaluation, randomized exploration, position-bias correction, and user research help separate recommendation effect from underlying interest.

Position bias

Items near the top receive more attention simply because they are visible first.

If clicks are treated as pure relevance labels, the model learns that previously top-ranked items are better.

Methods to address this include:

• Randomized position experiments
• Propensity weighting
• Explicit ratings
• Comparing behavior under different layouts

Interface design and ranking cannot be evaluated independently.

The displayed position is part of the treatment.

Safety and moderation

Recommendation can magnify harmful content by distributing it to likely audiences.

Safety systems may:

• Remove prohibited items
• Reduce distribution
• Add warnings
• Limit recommendations to minors
• Require authoritative sources for sensitive topics
• Detect coordinated manipulation

Moderation errors include both harmful content shown and legitimate content unfairly restricted.

Appeals, transparency, language coverage, and human expertise are part of system quality.

Manipulation and gaming

Creators and attackers adapt to ranking rules.

They may use:

• Clickbait
• Fake engagement
• Coordinated accounts
• Keyword stuffing
• Misleading thumbnails
• Review fraud

The system becomes an adversarial environment.

Abuse detection, robust signals, rate limits, identity controls, and policy enforcement need to evolve as incentives change.

Publishing every ranking detail can improve transparency but also make manipulation easier, so platforms must choose meaningful disclosure carefully.

Evaluation and experiments

Offline evaluation uses historical data and metrics such as ranking quality.

Online experiments compare real user outcomes.

Both have limits:

• Historical data reflects old recommendations.
• Short experiments may miss long-term effects.
• Engagement may not equal satisfaction.
• Average results may hide harm to groups.

Useful evaluation combines behavioral metrics, surveys, retention, quality review, safety measures, creator impact, and long-term study.

User control

Recommendation systems can offer:

• Hide or dislike controls
• Topic management
• Chronological views
• Explanation of why an item appears
• History deletion
• Profile reset
• Sensitive-topic controls

Controls should have visible effect.

An explanation such as "because you watched X" helps users correct mistaken assumptions and understand data use.

The system should not require endless negative feedback to escape a pattern created from one accidental click.

Knowledge check

1. Why does recommendation usually begin with candidate generation?
2. How do collaborative and content-based recommendations differ?
3. What is the exploration-exploitation tradeoff?
4. How does position bias affect training data?
5. Why is engagement an incomplete measure of recommendation quality?

The one idea to remember

Recommendations are ranked predictions shaped by exposure, objectives, constraints, and feedback loops. What a platform optimizes and permits matters as much as the predictive model itself.