The Minute-Bar Ledger: What a NIFTY Options Backtest Actually Reads

Part 2 of Quant Engineering. This is lesson 1 in the Market Data Pipelines track.

Nothing in this series is trading advice. It is engineering: how to build and validate the machinery that measures strategies, and how that machinery fails.

The decision

The first code anyone writes for a backtester is strategy logic. The first code anyone should write is a data inventory: a script that walks the entire store and reports, for every dataset, exactly what exists — date ranges, instruments, intervals, row counts, and gaps.

This sounds like bookkeeping. It is actually the decision that determines everything downstream, because a backtest is a function of its data before it is a function of its logic. Every result your engine will ever produce is bounded by three questions the inventory answers: what period can I test, what instruments can I test, and where are the holes that will masquerade as trading signals?

I skipped this step on my first engine build. The bill arrived weeks later, itemized: an options dataset that ended ten weeks before the spot dataset did, per-expiry data that existed only for the final five sessions of each contract's life, and numeric fields that were sometimes strings. Every one of those facts was discoverable in an afternoon of inventory. Instead each one surfaced as a mysterious backtest anomaly that took days to trace backward to the data.

Three datasets, not one

The mental model most people carry into index options backtesting is "the NIFTY data." There is no such thing. There are at least three datasets, and they differ in every dimension that matters.

Spot index bars are the simplest: one instrument, one bar per minute, no expiries, no strikes. The NIFTY 50 index is computed by the exchange from constituent stock prices; it has no bid, no ask, no volume of its own. That last point matters: any volume column in a spot index file is either zero, a placeholder, or a vendor artifact, and any logic that filters spot bars by volume is filtering on noise. The spot series is the natural reference for strike selection and for indicator calculations — but note that it is not tradeable, and it can print values during pre-open and closing windows that no order could have transacted at.

Futures bars carry real traded prices with real volume, one series per contract month, three months listed at a time. The front-month future typically trades at a premium or discount to spot — the basis — which decays toward zero at expiry. Some retail platforms use the current-month future, not spot, as the reference price for strike selection and signal indicators. This single convention difference moves strikes: on a day when the basis is 40 points and the strike grid is 50 points wide, a spot-referenced ATM straddle and a futures-referenced ATM straddle are different contracts roughly half the time. Your engine must pick one convention, implement it, and stamp it on every result. We will return to this repeatedly.

Options bars are where all the complexity lives. Every strike-and-expiry combination is its own instrument with its own OHLCV series: NIFTY weekly expiries with strikes at 50-point intervals near the money mean a single trading day easily involves several hundred live contracts. Coverage is ragged by nature — deep out-of-the-money strikes may not trade for minutes at a stretch, freshly listed strikes begin mid-history, and (the trap that gets its own post) many datasets only include each contract's final week of life. Options bars are also the only dataset where a missing bar is ambiguous: it can mean "did not trade this minute" (fine, carry the last price with care) or "data hole" (dangerous), and nothing in the file distinguishes them.

The anatomy of one bar

Before aggregating millions of bars, be precise about one. A minute bar is a tuple — timestamp, open, high, low, close, volume, sometimes open interest — and every field carries a convention someone chose.

The timestamp convention is the one that bites. Does 09:15:00 label the bar covering 9:15:00–9:15:59, or the bar that closed at 9:15:00 (covering 9:14:00–9:14:59)? Vendors differ. Get it wrong and every indicator, every entry time, every square-off in your engine is shifted by one minute — which sounds cosmetic until you realize a 9:20-entry straddle strategy is now entering on the 9:19 candle, one minute into a different volatility regime, and your expiry-day square-off is landing after the close. Establish the convention empirically: find a known event (the 9:15 open itself works — the first bar of the session has a distinctive range) and check which label it carries.

An NSE equity-derivatives session runs 9:15 to 15:30 — 375 minutes. A complete instrument-day therefore has at most 375 one-minute bars, and the count is your cheapest integrity check. Spot series should hit 375 nearly every regular day. Options contracts will not, and the shape of their shortfall is informative: an ATM contract with 370 bars has normal micro-gaps; one with 40 bars either listed mid-day, went deep out of the money, or your data has a hole.

Two artifacts deserve standing suspicion. Bars timestamped before 9:15 come from the pre-open auction and should be excluded from intraday logic — they are a different market mechanism. And zero-volume bars whose open, high, low, and close are all identical are usually vendor forward-fills, not trades; treating them as tradeable prices lets your simulated stop-loss execute at a price nobody transacted.

Coverage windows: the dead tail

Here is the inventory finding that reshaped my entire project, stated as the general lesson: your datasets will not end on the same date, and the difference is dead weight you must explicitly fence off.

In my store, the spot series ran from January 2020 to early October 2024. The options data ran from January 2020 to the end of July 2024. That ten-week tail of spot-only data is worthless for options backtesting — no contracts exist to trade — but nothing stops an engine from happily running a strategy across it. What does the engine do when strike selection finds no contracts? If the answer is "skips the day silently," your last ten weeks of results are a fiction of zero trades diluting every average. If the answer is "carries the last known premium," it is worse: the engine invents a frozen market where short options never move against you.

The corrected engine computes the intersection of coverage across every dataset a strategy touches, and refuses to simulate outside it. The refusal must be loud — a validation error naming the requested range and the available range — because a warning in a log nobody reads is how the fiction returns.

Coverage has a second dimension besides dates: intervals. My store carried options bars at one, three, and five minutes. If your engine resamples one-minute bars into higher intervals itself (the better design — one source of truth), the resampling rules are conventions again: a "9:20 five-minute candle" should aggregate 9:20:00 through 9:24:59, labeled at the open. Off-by-one resampling silently shifts every indicator built on it.

Sizing the problem

An inventory should also tell you the scale you are engineering for, because scale drives the storage decisions later in this track. Rough arithmetic for a NIFTY store like mine: about 1,100 trading days across four and a half years; on each day, one spot series plus a handful of futures series plus a few hundred live option contracts; each contract holding up to 375 one-minute bars. That lands in the high hundreds of millions of option bars — small enough to live on one machine in compressed parquet, large enough that a badly partitioned store turns every backtest into a full scan. The difference between a 90-second run and a 40-minute run over this data is layout, not hardware, and we will spend a whole post on it.

Open interest, special sessions, and the calendar's fine print

Two more fields of the ledger deserve a standing note in the inventory, because both look optional until the day they are not.

Open interest rides along on many options bar feeds, and it is a different kind of number from everything else in the row: OHLCV describes the minute, but open interest describes the standing state of the contract, updated on the exchange's schedule rather than per trade. Vendors variously report it per minute (often just forward-filled), per snapshot, or end-of-day only. If any strategy logic reads open interest — OI-based strike filters are popular in the retail world — the inventory must establish its actual update frequency, because a forward-filled OI column evaluated intraday gives strategy rules a stale number wearing a fresh timestamp. And if your dataset carries no OI at all, that fact belongs in the feasibility notes: an entire family of platform configs becomes untestable, which you want to know at classification time, not at debug time.

Special sessions are the calendar's fine print. The Indian market runs a Muhurat trading session on Diwali — an evening session, outside normal hours, that will either be missing from your data or present with timestamps that violate every session-boundary assumption your loop makes. Budget days and election-result days have seen shortened or extended sessions and circuit-halt interruptions; the odd technical-glitch day has closed a segment early. Each produces a day whose bar count is legitimately far from 375 and whose first or last bar is legitimately outside 9:15–15:30. The inventory should carry an explicit allowlist of known irregular sessions, so that integrity checks can flag unexpected anomalies without crying wolf on the expected ones — and so the engine can decide per strategy whether an irregular day is simulatable or should be excluded with a reason. A backtest that silently trades a straddle through a 90-minute Muhurat session as if it were a full day is manufacturing a data point no live trader could have experienced.

The inventory script

The concrete deliverable of this post: a walk-the-store script whose output you keep under version control, so coverage changes when data is re-ingested show up as diffs. The shape, trimmed to essentials:

import pandas as pd
from pathlib import Path
 
def inventory(store: Path) -> pd.DataFrame:
    rows = []
    for dataset_dir in sorted(store.iterdir()):          # spot/ futures/ options/
        for f in sorted(dataset_dir.rglob("*.parquet")):
            df = pd.read_parquet(f, columns=["ts"])
            rows.append({
                "dataset": dataset_dir.name,
                "file": f.relative_to(store).as_posix(),
                "rows": len(df),
                "first": df["ts"].min(),
                "last": df["ts"].max(),
                "days": df["ts"].dt.normalize().nunique(),
            })
    return pd.DataFrame(rows)
 
inv = inventory(Path("data/nifty"))
print(inv.groupby("dataset")[["rows", "days"]].sum())
print(inv.groupby("dataset")[["first", "last"]].agg(["min", "max"]))

Extend it with the checks this post argued for: per-day bar counts against the 375 ceiling, pre-9:15 timestamps, zero-volume flat bars, dtype assertions on every numeric column (my archive of platform strategy configs stored quantities as strings — "10" not 10 — and the same disease infects market data files), and per-expiry day counts for options, which is the check that exposes the front-week-only trap two posts from now.

Run it before every backtest batch, not once. Data stores mutate: re-ingestion, vendor corrections, a new download that silently changed conventions. The inventory is your regression test against your own data.

Check your own engine

• Can you state, from memory, the exact first and last date of every dataset your backtests touch? If not, the inventory does not exist yet.
• Does your engine compute the coverage intersection across datasets and refuse to run outside it — loudly?
• Do you know whether your bar timestamps label the open or the close of the minute? Have you verified it empirically rather than from vendor docs?
• Are pre-open bars excluded from intraday logic?
• Are zero-volume flat bars distinguished from real trades anywhere in your fill logic?
• Is every numeric column's dtype asserted at load time?
• If your data were re-delivered tomorrow with one convention changed, would anything in your pipeline notice?

Where this goes next

The inventory tells you what exists. The next post in the series takes the single most expensive lesson from my validation project — the expiry-indexing convention mismatch that produced a 96 percent silent failure rate — and turns it into a general method for verifying conventions against observed data instead of trusting documentation. It is the cheapest insurance in this entire library.