What’s the question?
A backtest needs a universe, and the cheapest one to build is the index membership list as it stands today. That list is not the index of 2005. It omits every company that failed, was taken over, or shrank far enough to be dropped, and those companies left for reasons written into their returns.
Survivorship bias is the name for the error that follows. Naming it is easy; sizing it is the useful part, because the correction is not free. A point-in-time membership record, which stores who belonged to the index on any given date rather than only who belongs now, is a heavier thing to carry than one current list, and the index membership endpoint serves one through as_of. The measurement below holds everything constant except membership: two equal-weight portfolios, the same twenty years of monthly total returns, the same index, differing only in which companies are eligible in each month.
The approach
- For each year from 2005 to 2024, take the S&P 500 as it stood on 1 January of that year. That is Book A, reset every January.
- Take the roster as the API serves it today and run it across the same twenty years. That is Book B, the list a backtester reaches for first.
- Pull monthly total returns for every company either book ever holds, January 2005 to December 2024. Hold each book in equal weight, resetting weights monthly; a company enters the month it prints a return and leaves when it stops printing one.
- Require agreement between two independent fields: a company-month whose reported return does not match the step in that company’s own split-adjusted price, by more than 0.5 in logs, leaves the panel. That removes 61 of 169,341 company-months, from both books alike.
- Compound both books and compare, by year and over the full window.
Companies are carried by entity id rather than by ticker. A ticker points at whoever holds it now, so a symbol reassigned after a bankruptcy would swap one company for another in the middle of the panel, which is the failure this exercise is meant to measure rather than commit.
Code
import numpy as np
import pandas as pd
import xfinlink as xfl
xfl.set_api_key("YOUR_API_KEY") # free at https://xfinlink.com/signup
YEARS = list(range(2005, 2025))
BLOCKS = [("2005-01-01", "2009-12-31"), ("2010-01-01", "2014-12-31"),
("2015-01-01", "2019-12-31"), ("2020-01-01", "2024-12-31")]
pit = {y: {int(x) for x in
xfl.index("sp500", as_of=f"{y}-01-01", limit=1000)["entity_id"].dropna()}
for y in YEARS}
current = {int(x) for x in xfl.index("sp500", limit=1000)["entity_id"].dropna()}
ids = sorted(set().union(*pit.values()) | current)
frames = []
for start, end in BLOCKS:
for i in range(0, len(ids), 50):
frames.append(xfl.prices(entity_id=ids[i:i + 50], start=start, end=end,
interval="1mo", fields=["adj_close", "return_daily"],
max_rows=200000))
px = pd.concat(frames, ignore_index=True)
px["m"] = px["date"].dt.to_period("M")
px = px.drop_duplicates(["entity_id", "m"])
R = px.pivot(index="m", columns="entity_id", values="return_daily").sort_index()
A = px.pivot(index="m", columns="entity_id", values="adj_close").sort_index()
R = R.mask((np.log1p(R) - np.log(A).diff()).abs() > 0.5)
rows = []
for m in R.index:
a = R.loc[m, [c for c in R.columns if c in pit[m.year]]].dropna()
b = R.loc[m, [c for c in R.columns if c in current]].dropna()
rows.append({"m": m, "pit": a.mean(), "sur": b.mean()})
P = pd.DataFrame(rows).set_index("m")
for k in ("pit", "sur"):
print(k, (1 + P[k]).prod() ** (12 / len(P)) - 1, (1 + P[k]).prod())Full script with formatting and visualisation: survivorship-bias-point-in-time-sp500-python.py
Output
==============================================================================
WHAT A CURRENT MEMBERSHIP LIST ADDS TO A BACKTEST THAT NEVER HAPPENED
==============================================================================
Universe S&P 500 members, monthly total returns, January 2005 - December 2024
Book A the roster as it stood on 1 January of each year, reset annually
Book B the roster as the API serves it today, held across all 20 years
Weights equal, reset every month; a company enters the month it has a
return and leaves when it stops printing one
Companies across the 20 January rosters 901
Companies on the roster today 504
January 2005 members still on it today 234 of 498
Panel 240 months x 910 companies, 169,280 company-months, 61 set aside by the agreement screen
Range 1625% (GME, 2021-01) to -99.2% (LEH, 2008-09)
annual vol x growth names names
2005 2024
Point-in-time roster 9.97% 17.53% 6.69x 466 496
Today's roster 15.42% 16.94% 17.60x 404 502
Difference 5.45pp 2.63x
Calendar year returns
year point-in-time today gap (pp)
2005 7.47% 14.20% 6.74
2006 16.18% 22.28% 6.10
2007 0.72% 11.48% 10.76
2008 -40.05% -33.37% 6.68
2009 48.44% 46.74% -1.69
2010 21.60% 25.67% 4.07
2011 -0.01% 3.77% 3.78
2012 17.23% 22.83% 5.60
2013 36.27% 41.07% 4.80
2014 13.97% 18.59% 4.62
2015 -2.59% 3.74% 6.33
2016 15.57% 19.41% 3.84
2017 18.59% 27.11% 8.52
2018 -7.73% -2.64% 5.09
2019 30.00% 34.48% 4.48
2020 12.91% 20.66% 7.75
2021 29.69% 31.94% 2.25
2022 -10.96% -10.55% 0.41
2023 14.50% 23.21% 8.71
2024 12.89% 19.26% 6.38
Today's roster wins in 19 of 20 years; median gap 5.35pp
Five-year blocks, annualised
2005-2009 point-in-time 2.28% today 8.77% gap 6.49pp
2010-2014 point-in-time 17.23% today 21.79% gap 4.57pp
2015-2019 point-in-time 9.87% today 15.57% gap 5.70pp
2020-2024 point-in-time 11.00% today 15.91% gap 4.91pp
Robustness: clipping each month's cross-section at the 1st and 99th percentiles
gives 9.86% against 14.94%, a gap of 5.07pp.What this tells us
The point-in-time book compounds at 9.97% a year. The same rule run on today’s membership list compounds at 15.42%, which is 5.45 percentage points a year of return no investor could have collected, and on a dollar the two books end twenty years apart at 6.69x and 17.60x. Book A sits close to what an equal-weight S&P 500 has actually delivered over a comparable span, near 10% a year, which is the check that matters.
Two mechanisms produce the gap, both the same act of conditioning on an outcome. The first is familiar: companies that failed are absent from today’s list, so their losses never enter Book B. Lehman Brothers sat in the index through 2008 and prints -99.2% that September, the worst month in the panel, and it belongs to Book A alone. The second is quieter and does more work. A company that entered the index in 2015 entered because it had already grown, and Book B credits it with every earlier month it has a price for, so the selection runs backwards across the whole panel. An event study of S&P 500 additions looks at the same selection in the weeks before inclusion.
Turnover sets the scale: 901 distinct companies pass through the twenty January rosters to fill about 500 slots, and of the 498 members in January 2005, 234 remain on today’s list. The distortion is stable rather than episodic. Today’s list wins in 19 of 20 calendar years, each five-year block gives between 4.57 and 6.49 points, and clipping the extreme 2% of each monthly cross-section still leaves 5.07 points. Only 2009 favours the point-in-time book, when the rebound rewarded distressed names today’s list no longer contains. Both books here hold their members in equal weight; a test of the same roster question under cap weighting asks how the gap changes when large companies carry more of the portfolio.
So what?
Five percentage points a year is larger than almost any documented factor premium. A momentum screen or a quality tilt tested on a current membership list therefore starts with more free return than the effect it is trying to demonstrate, and the measured edge can be entirely the universe. A momentum strategy backtest run on both kinds of roster measures this separately for the winner and the loser legs. Ask which list built a backtest before asking anything about its signal. The same question applies to the benchmark the result is judged against; how to pick a benchmark for a backtest covers that side.
Two habits fix it. Build the universe at each rebalance date from the roster of that date, so the portfolio holds what was holdable. Then carry each holding by a persistent company identifier rather than by ticker, because a delisted company’s symbol is frequently reassigned and joining on symbol quietly replaces the failure with its successor’s returns. The first habit removes the bias; the second stops it creeping back in through the join.