Your Options Data Probably Only Has the Front Week — Audit Before You Trust

Part 4 of Quant Engineering. This is lesson 2 in the Market Data Pipelines track.

Nothing in this series is trading advice. It is engineering: how to build and validate the machinery that measures strategies, and how that machinery fails.

The decision

When someone says "I have four and a half years of NIFTY options data," there are two radically different things they might possess. The first is four and a half years of complete chains: every listed contract, from listing to settlement, across all expiries. The second — vastly more common in retail-accessible datasets — is four and a half years of front-week coverage: for each expiry, only the final handful of sessions before settlement, because that is when the contract was near, liquid, and worth recording.

The two look identical from the outside. Same folder sizes in the terabyte range, same date span, same instrument. They differ only in a dimension nobody thinks to check: coverage per contract as a function of time-to-expiry. And that dimension decides, before you write any engine code at all, which strategy questions your dataset can answer honestly and which it will answer with confident garbage.

This post is the audit that measures that dimension, the classifier that applies it to a strategy corpus, and the story of discovering — several weeks later than I should have — that my own store was the second kind.

The discovery

My options store was organized as one partition per expiry. Reasonable layout. The trouble surfaced obliquely, as it always does: the expiry-indexing investigation from the previous post kept turning up contracts whose bars "started late." When I finally profiled it properly — for every expiry partition, the first and last trading date present — the pattern was immediate and uniform: each expiry's data began roughly five sessions before its settlement date and ended on it.

Trace that back to the source and it makes complete sense. Datasets like this are typically built by recording the live chain — a collector subscribing to the actively traded contracts each day and archiving what it sees. Indian index options liquidity concentrates overwhelmingly in the nearest weekly expiry, so a liquidity-following collector captures each contract only once it becomes the front week. Nobody lied. The dataset genuinely spans January 2020 to July 2024. But its depth at any moment is one expiry, give or take, and its historical shape is a chain of one-week windows laid end to end — not a rectangle of full chains.

The store had over eleven hundred trading days and hundreds of millions of bars. Every gigabyte of it true, and the phrase "four years of options data" still fundamentally misleading.

What front-week-only data can and cannot answer

Draw the consequence sharply, because it fences your entire research program.

Fully answerable: any strategy whose every leg trades the nearest weekly expiry and whose holding period lives inside that contract's final week. That covers the enormous retail intraday universe — morning straddles and strangles, expiry-day scalps, directional weekly buys, iron condors opened and closed within the week. Data exists for the full life the strategy needs.

Partially answerable: strategies entering early in the front week and holding to expiry are fine; strategies entering before a contract becomes front-week — say, a Friday entry into the following week's expiry — hit missing days at the start of their intended holding period. Whether the backtest is salvageable depends on exactly when your collector started covering each contract, which only the audit below can tell you. "Roughly five sessions" has ragged edges.

Unanswerable, full stop: anything touching monthly contracts before their final week, calendar spreads (which need two expiries alive simultaneously with real prices), multi-week positional structures, and any strategy leg specifying expiry index 2 or beyond. For these, the data does not thin out — it does not exist. An engine that proceeds anyway will manufacture one of two fictions: near-zero trades (selection fails, run reports success — the polite catastrophe of the previous post) or, far worse, trades against forward-filled ghost prices.

In my corpus of ten and a half thousand platform configs, about 3,650 were NIFTY strategies, and the feasibility cut ran straight through them: the great majority were front-week intraday and testable; every monthly and far-expiry config had to be excluded — not because the strategies were bad, but because no honest simulation of them was possible with this store. Marking them excluded by reason of data rather than silently skipping them was one of the more important reporting decisions in the project.

The audit

The measurement is embarrassingly simple once you know to make it. For every expiry, collect the set of trading dates with bars, then examine the distribution of coverage depth:

import pandas as pd
from pathlib import Path
 
def expiry_coverage(options_root: Path) -> pd.DataFrame:
    rows = []
    for exp_dir in sorted(options_root.iterdir()):        # one partition per expiry
        expiry = pd.Timestamp(exp_dir.name)
        dates = set()
        for f in exp_dir.rglob("*.parquet"):
            ts = pd.read_parquet(f, columns=["ts"])["ts"]
            dates.update(ts.dt.normalize().unique())
        dates = sorted(dates)
        rows.append({
            "expiry": expiry,
            "first_day": dates[0],
            "last_day": dates[-1],
            "days_covered": len(dates),
            "lead_days": len([d for d in dates if d < expiry]),
        })
    return pd.DataFrame(rows).sort_values("expiry")
 
cov = expiry_coverage(Path("data/nifty/options"))
print(cov["days_covered"].describe())
print(cov["days_covered"].value_counts().sort_index())

Read the histogram, not the mean. A full-chain dataset shows weeklies clustering near their listed lifetime and monthlies near twenty-plus sessions. A front-week store shows a tight spike at five-ish for everything, monthlies included — the monthly contract exists in the data only for its final week, exactly like a weekly. That spike is the signature.

Then go one level deeper, because averages hide the ragged edges that decide the "partially answerable" category: for each expiry, which weekday does coverage begin on? If coverage reliably starts the session after the previous expiry settles, entries any time inside the front week are safe. If some expiries only enter the data two or three sessions before settlement — thin weeks, holidays, collector outages — then even nominally front-week strategies with early-week entries have holes, and you want a per-expiry exclusion list rather than a global assumption.

While you are in there, the same pass surfaces calendar anomalies for free. My store's expiry dates were Thursdays more than two hundred times in a row — and Wednesday exactly once, in April 2024, where a holiday shifted settlement. That lone Wednesday is a canary: any engine logic hardcoding "expiry means Thursday" disagrees with the data on precisely the weeks where the derived calendar matters most. (It also foreshadows a bigger structural break: the weekly expiry weekday itself was moved by exchange policy in later years — Thursday is not a law of nature. Calendars are derived, never assumed; that post is coming in this track.)

The feasibility classifier

The audit's output should not be a chart you look at once. It should be a gate in front of the engine: a function that takes a strategy config plus the coverage table and returns testable, partial, or untestable — with reasons.

def classify(cfg, coverage: pd.DataFrame) -> tuple[str, str]:
    max_idx = max(int(leg["expiry"]) for leg in cfg["legs"])       # 1-based
    if max_idx >= 2:
        return "untestable", f"leg requests expiry index {max_idx}; store is front-week only"
    if cfg["holding"] == "intraday":
        return "testable", "front-week intraday"
    entry_wd, exit_rule = cfg["entryWeekday"], cfg["exitRule"]
    lead = coverage["lead_days"].quantile(0.05)                    # pessimistic lead time
    if needs_days_before_expiry(entry_wd, exit_rule) <= lead:
        return "testable", f"positional inside observed {lead:.0f}-day lead"
    return "partial", "entry may precede contract coverage on thin weeks"

The details will differ per schema; the architecture should not. Classification happens at validation time, before simulation; the result is stamped into the run manifest; and downstream reporting always states the denominator — "testable: 3,412 of 3,652; excluded for data reasons: 240" — so that a survivorship story cannot creep into your aggregates. Excluding strategies is honest. Excluding them silently converts a data limitation into a selection bias.

There is one more subtlety worth stating because it protects you from over-trusting even the covered region: front-week concentration in the data mirrors front-week concentration in the market. The first bars of a contract's coverage window are its thinnest — wide effective spreads, sparse trades, stale-looking quotes. So even inside the "testable" zone, fills simulated at the very start of a coverage window deserve wider slippage assumptions than fills near expiry. The P&L Attribution track picks that thread up properly.

The audit as a shopping tool

The same measurement that diagnoses the store you have is the sharpest instrument for evaluating the store you are about to buy, and it converts a vague vendor conversation into four falsifiable questions.

First: for a weekly expiry, on what date does your coverage of that contract begin — listing date, or some liquidity threshold? Ask for a specific recent expiry and a per-day row count. Second: how many distinct expiries have bars on a given historical Tuesday? A full-chain store answers three or more (two weeklies plus a monthly, often more); a front-week store answers one, maybe two. Third: what is the strike range per day — all listed strikes, or a band around the money? A store that only carries ATM plus-or-minus ten strikes quietly caps how far your wings, hedges, and adjustment logic can reach, which is a subtler version of the same disease. Fourth: are missing minutes distinguishable from non-trading minutes, or forward-filled into false continuity?

Ask for a sample month and run the coverage histogram on it before paying for the archive. Every question above takes the vendor thirty seconds to answer honestly and takes you one plot to verify — and the vendors who cannot answer them are answering them.

It is also worth being clear-eyed about what upgrading to full chains actually buys, because the instinct after discovering the front-week trap is "get complete data and the problem disappears." It half-disappears. Full chains make calendar spreads and monthly strategies representable; they do not make the early-life bars of far expiries liquid. A monthly contract three weeks from expiry trades thinly outside a narrow strike band, and simulating tight stops against its sparse, wide-spread bars replaces "no data" with "data that cannot support the fill model" — a more dangerous condition precisely because the engine no longer refuses. The feasibility gate does not get deleted when the data improves; its criteria shift from existence of bars to density and spread of bars. Coverage is necessary. It was never sufficient.

Edge cases the audit must survive

Overlap weeks. When the monthly is the nearest expiry, the front week is the monthly. Your coverage table will show the monthly with the same five-day spike as weeklies — correct, but it means "monthly strategies are untestable" needs one exception: monthly legs during their own final week resolve fine. The classifier above already handles it via expiry index rather than contract type, which is the right axis.

Zero-lead expiries. A handful of expiries may show coverage starting on settlement day — data holes, collector restarts. Expiry-day-only coverage supports expiry-day scalps and nothing else; per-expiry exclusion lists beat global rules.

Re-ingestion drift. Coverage is a property of the store, not of the vendor relationship. Every re-download or backfill can change it. The audit belongs in the pre-batch gate suite from part 2, its output diffed under version control, so that the day your store quietly gains (or loses) depth, a diff tells you before your results do.

Check your own engine

• Have you plotted days-of-coverage per expiry as a histogram? Do you know whether you own full chains or front-week windows?
• Do you know the pessimistic (5th percentile) lead time — the earliest pre-expiry day you can rely on across all expiries?
• Is there a feasibility gate that classifies every strategy against measured coverage before simulation, with reasons stamped into results?
• Do your aggregate reports state how many strategies were excluded for data reasons?
• Does anything in your pipeline detect a weekday anomaly in expiry dates (the lone Wednesday) rather than assuming a fixed expiry weekday?
• If your store were silently backfilled to full chains tomorrow, would your coverage audit notice and widen the feasibility gate?

Where this goes next

Two posts of data reality are enough to start building. The next post crosses into the Backtest Engine Core track and nails down the first thing every engine must define before any strategy logic exists: the clock. Minute-close evaluation semantics — when the engine is allowed to know things, when indicators update, and when decisions execute — is where lookahead bias is either designed out or baked in. Everything the engine ever reports depends on getting that one abstraction right.