1. Identify the decision timestamp
For every trade, ask what was genuinely known then. Restated financials, today's index members and cleaned delisting histories can quietly leak the future into the past.
2. Separate discovery from evaluation
If the same data selected the rule and judged it, the reported result is in-sample. A later untouched period, another market or a pre-registered replication provides stronger evidence.
3. Count the hidden experiments
One displayed strategy may be the winner of hundreds of lookbacks, thresholds and universes. The more choices tried, the less surprising an attractive chart becomes.
4. Demand a tradable cost model
Commission is only one cost. Spread, market impact, financing, borrow fees, futures rolls, taxes and rejected orders can matter more. Costs should rise with turnover and fall with liquidity.
5. Inspect the return path
CAGR and Sharpe ratio hide timing. Review drawdown depth, duration, worst month, recovery time and concentration by year. A strategy may earn most of its return in a short regime that no longer exists.
6. Test neighboring rules
A robust economic effect should not disappear when a 12-month lookback becomes 11 or 13 months. Parameter cliffs are evidence of overfitting or a data artifact.
7. Compare gross and net exposure
Two portfolios with the same net market exposure can carry very different gross leverage, sector bets, duration, currency risk and liquidity. Name the exposures before naming the alpha.
8. Look for a mechanism and a failure mechanism
A behavioral or risk-based story is not proof, but it gives the hypothesis something falsifiable. Write down the environment in which the effect should weaken or reverse.
9. Verify the benchmark
Cash, a broad index and a risk-matched benchmark answer different questions. A levered strategy should not be praised merely for beating an unlevered benchmark.
10. End with a decision rule
Decide in advance what evidence would pause, resize or reject the strategy. Without that rule, every loss can be explained away and every gain can be called validation.