Testing &
Validation

What this week covers

Here is what makes testing quantitative systems different from testing almost any other software: a broken data pipeline usually produces a better backtest, not a crash.

Leak tomorrow's price into today's signal and the Sharpe goes up. Drop the companies that went bankrupt and the returns go up. Use a vendor's current value where a historical one belongs and the fit improves. Every one of those bugs looks exactly like success. Nothing throws. The chart looks great.

In ordinary software, bugs make things visibly worse, so users report them. Here, the highest-impact bugs make the numbers more attractive — which means nobody will ever report them to you. They are caught by tests written specifically to look for them, or they are not caught at all.

Six patterns that do the catching:

• Unit tests against known values — the function computes what it claims to compute.
• Look-ahead detection — the signal at time t cannot see past t.
• Point-in-time tests — historical data is what was known then, not what's true now.
• Roll-schedule tests — continuous contracts survive their expiries.
• Golden-output tests — a known-correct result stays correct.
• Survivorship checks — the universe includes the dead.

These are the standard hazards of the field — every quant shop meets them, and the tests below are the standard answers.

1 — Unit tests against known values

The foundation, and the least glamorous. Take a function whose answer you can work out independently — by hand, from a textbook, from a published worked example — and assert it produces that answer.

The point isn't proving the function works today. You already believe it works today; that's why you committed it. The point is regression protection: six months from now someone optimizes this function and changes its behavior at a boundary they didn't think about. The test is what tells them.

from math import isclose


def test_rolling_zscore_known_values():
    # A 5-point window of 1..5: mean 3, sample std 1.5811.
    # Final point z = (5 - 3) / 1.5811 = 1.2649
    s = pd.Series([1, 2, 3, 4, 5], dtype=float)
    z = rolling_zscore(s, window=5)

    assert pd.isna(z.iloc[:4]).all()            # not enough history yet
    assert isclose(z.iloc[4], 1.26491, abs_tol=1e-5)


def test_black_scholes_call_matches_textbook():
    # Hull, worked example: S=42, K=40, r=0.10, sigma=0.20, T=0.5 -> 4.76
    px = black_scholes_call(S=42, K=40, r=0.10, sigma=0.20, T=0.5)
    assert isclose(px, 4.76, abs_tol=0.01)

Notice where the expected values come from: somewhere other than the code under test. A test whose expected value was produced by running the function proves only that the function is deterministic. Hand calculation, a textbook, or a second independent implementation are what make it a real check.

Test the edges, not the middle

The happy path is the case you already coded correctly. Bugs live at the boundaries:

def test_rolling_zscore_zero_variance_does_not_return_inf():
    # A halted symbol prints the same price all week. std = 0 -> x/0.
    # This MUST NOT silently produce inf, which would become a giant position.
    s = pd.Series([7.0] * 10)
    z = rolling_zscore(s, window=5)

    assert not np.isinf(z).any(), "zero-variance window produced inf"
    assert pd.isna(z.iloc[4:]).all(), "flat window should be NaN, not a signal"

That test is worth more than the previous two combined, and it shows the shape of the whole week. A halt produces a flat price window; a flat window has zero variance; zero variance divides by zero; and inf becomes the largest position the sizing logic will permit. The strategy does that on the worst possible day — the one where the exchange halted the symbol.

The market data itself supplies the edge cases: halts, holidays, early closes, expiries, delistings, limit-up days, zero-volume sessions. You don't have to invent adversarial inputs. You have to remember they exist.

2 — Look-ahead detection

Look-ahead is the leak where a signal at time t is computed using information that didn't exist until after t. It is the most common serious bug in backtesting and the hardest to catch by reading code, because it almost always enters through an index alignment that looks completely innocent.

The shift test

Take the feature your signal consumes and shift its timestamps forward one bar — simulating it arriving one bar later than you assumed. Re-run. The result must change.

If a one-bar delay changes nothing, the signal was never reading the feature at the boundary you believed. Either it's reading a stale copy, or it isn't reading the feature at all, or the alignment silently absorbs the shift. All three mean the backtest measures something other than the strategy you described.

def test_signal_is_sensitive_to_feature_timing():
    bars = load_fixture("sample_bars.parquet")

    base    = build_signal(bars)
    delayed = build_signal(bars.assign(feature=bars.feature.shift(1)))

    # If a one-bar delay in the input doesn't move the output,
    # the signal is not consuming the feature where we think it is.
    assert not base.equals(delayed), \
        "signal unchanged when feature delayed one bar — check alignment"

The future-perturbation test

The shift test proves the signal reads the feature at some boundary. This one proves the boundary is the right one — it's the stronger test, and the one that actually catches leakage.

Pick a cutoff t. Replace every data point after t with garbage. Recompute the signal at t. It must be bit-identical to before, because nothing after t is allowed to influence it.

def test_no_future_information_leaks_into_signal():
    bars   = load_fixture("sample_bars.parquet")
    cutoff = pd.Timestamp("2022-06-15")

    clean = build_signal(bars).loc[:cutoff]

    # Corrupt the future beyond all recognition.
    poisoned = bars.copy()
    poisoned.loc[poisoned.index > cutoff, ["close", "feature"]] = -999_999.0

    leaked = build_signal(poisoned).loc[:cutoff]

    pd.testing.assert_series_equal(
        clean, leaked,
        obj="signal at/before cutoff changed when FUTURE data was corrupted",
    )

3 — Point-in-time tests

A subtler leak, and one that arrives from the vendor rather than from your code.

Many data providers serve the current value of a historical field. Ask for a company's Q3 2021 earnings and you may get the number as restated in 2023 — not the number the market actually traded on in October 2021. Ask for an index's constituents "as of 2015" and some vendors hand back today's list. Your backtest then trades on information that did not exist, using a dataset that looks entirely plausible.

The property to test: a value stamped at date D must equal what was knowable at date D, including its revision history.

def test_fundamentals_are_as_reported_not_as_restated():
    # This company restated FY2021 revenue upward in 2023. A point-in-time
    # store must still return the ORIGINAL number when asked what was
    # knowable in Oct 2021.
    known_then = pit_store.get(sym, "revenue_q3_2021", as_of="2021-10-20")
    known_now  = pit_store.get(sym, "revenue_q3_2021", as_of="2024-01-01")

    assert known_then == ORIGINAL_FIGURE,  "as-of-2021 read returned a restated value"
    assert known_now  == RESTATED_FIGURE,  "restatement missing from current view"
    assert known_then != known_now,        "store has no revision history at all"


def test_index_membership_is_as_of_date():
    members = universe.constituents(as_of="2015-06-30")

    assert DELISTED_IN_2016 in members,   "point-in-time universe is back-filled from today"
    assert IPO_IN_2019 not in members,    "future constituent leaked into 2015 universe"

The third assertion in the first test is the important one. known_then != known_now fails if the store has no revision history — if it serves one value for every as-of date. That's the failure that looks like success: every query returns a number, nothing errors, and every number is quietly from the future.

4 — Roll-schedule and contract-spec tests

Futures expire. A "continuous contract" is a fiction — successive contracts stitched into one series — and the stitch is where the bugs live.

Two contracts trade at different prices for entirely structural reasons: carry, storage, seasonality. Join them naively and you manufacture a gap that never happened. A backtest reads that gap as a return, and if the roll is quarterly, you've invented a tradeable signal that fires like clockwork four times a year.

Whether the stitching is done by your code or by the vendor, the test is the same — walk the series through its seams and assert they're clean:

def test_no_synthetic_jumps_in_continuous_series():
    df = load_continuous(sym, start, end)
    rets = df.close.pct_change().abs()

    # This contract does not move 15% in a day. If it "did", we are
    # looking at a roll seam, not a market event.
    spikes = rets[rets > MAX_PLAUSIBLE_DAILY_MOVE]
    assert spikes.empty, f"{len(spikes)} suspected roll seams: {list(spikes.index.date)}"


def test_roll_behavior_matches_the_roll_we_asked_for():
    # Volume-based rolls cluster near expiry as liquidity migrates.
    # If the seams land on the 1st of the month like clockwork, the
    # series is calendar-rolled and the setting doesn't mean what we think.
    seams = detect_seams(load_continuous(sym, start, end))
    days  = {s.day for s in seams}
    assert len(days) > 3, f"seams cluster on days {days} — looks calendar-rolled"


def test_contract_multiplier_matches_the_exchange():
    spec = load_contract_spec(sym)
    assert spec.multiplier == EXPECTED_MULTIPLIER
    assert spec.currency   == "USD"

The second test is worth understanding as a general technique. When behavior lives in a dependency you configure rather than code you wrote, you can't step through it — so you assert on the fingerprint it leaves in the output. That applies far beyond rolls.

The third looks trivial and catches a genuinely nasty class of bug. A wrong multiplier scales every P&L figure by a constant. Nothing crashes. The equity curve has the right shape — it's just wrong by orders of magnitude, and Sharpe, being scale-invariant, looks completely normal. Contract specs are reference data, and reference data earns an assertion precisely because it looks too boring to check.

5 — Golden-output regression tests

Once a backtest result has been checked and believed, freeze it. Commit the fixture and the expected numbers. Every future change must reproduce them exactly, or explain itself.

This is the test that protects you from your own refactors. Change how missing bars are handled, and six strategies quietly shift — nobody notices for a month, because each one still "looks fine."

GOLDEN = {
    "sharpe":       1.242_1,
    "annual_ret":   0.187_3,
    "max_dd":       0.142_8,
    "n_trades":     418,
    "final_equity": 1_487_233.19,
}


def test_reference_backtest_matches_golden_output():
    result = run_backtest(
        strategy="reference_v3",
        fixture="fixtures/sample_2015_2023.parquet",  # frozen data, in-repo
        seed=42,                                      # no wall-clock, no RNG drift
    )

    for metric, expected in GOLDEN.items():
        actual = getattr(result, metric)
        assert isclose(actual, expected, rel_tol=1e-6), (
            f"{metric}: {actual} != golden {expected}\n"
            f"If this change is intentional, update GOLDEN in the same commit "
            f"and say why in the message."
        )

Three things make a golden test trustworthy, and it's worthless without all three:

• The fixture is frozen and committed. A golden test pointed at a live database changes when the data changes, and teaches you nothing.
• The run is deterministic. Fixed seed, no datetime.now(), no dependence on dict ordering, no parallel reduction that sums in a different order each run.
• The failure message says what to do. A golden test should fail when behavior intentionally changes. The message has to distinguish "you broke something" from "update the number."

6 — Survivorship-bias checks

Build a universe from the companies that exist today and you have quietly excluded every company that went bankrupt, got delisted, or was acquired. Those are precisely the worst performers in history. The backtest then reports the returns of a portfolio that only ever held survivors — a portfolio nobody could have constructed at the time, because the information needed to build it didn't exist yet.

The tell is an equity curve that looks too good and too smooth: no sharp idiosyncratic losses, because every name that blew up was removed before the test began.

What makes this one worth real attention is where it enters. Not in a strategy, not in a backtest engine — in how a list of symbols was assembled. It's a property of the universe definition, which is why reading the strategy code will never reveal it.

def test_universe_contains_delisted_names():
    universe = build_universe(start="2010-01-01", end="2020-01-01")

    # Names that were in the index during the window and are gone now.
    for dead in KNOWN_DELISTINGS_IN_WINDOW:
        assert dead in universe.symbols, \
            f"{dead} absent — universe appears to be survivor-only"


def test_universe_does_not_shrink_going_back_in_time():
    # A point-in-time universe stays roughly stable as you walk back.
    # A survivor-biased one shrinks, because fewer of TODAY's names existed.
    n_2010 = len(build_universe(as_of="2010-01-01").symbols)
    n_2020 = len(build_universe(as_of="2020-01-01").symbols)

    assert n_2010 > n_2020 * 0.7, \
        f"universe collapses going back ({n_2010} in 2010 vs {n_2020} in 2020)"


def test_delisted_name_has_a_terminal_return():
    # A bankrupt name must end with a real loss, not merely stop reporting.
    px = load_prices(BANKRUPT_SYM, start="2010-01-01", end="2020-01-01").dropna()

    assert len(px) > 0, "delisted name has no data at all"
    assert px.iloc[-1] < px.iloc[0] * 0.1, \
        "delisting recorded without the loss that preceded it"

That third test catches the half-fixed case, which is the one you'll actually meet. The universe includes delisted names — good — but their price series simply stops on the delisting date instead of recording the collapse. The backtest holds the position, sees no more data, and marks it at the last good price. A bankruptcy becomes a flat line instead of a −95% return.

Common mistakes