One of the risks of systematic or algorithmic trading is that the trading system can be over-fit in the market. Overfitting means that a strategy designed to work on a certain set of market data does not generalize well to other data. Such a system may look good in historical tests, but it will act badly in real time.
On statistical significance test can be used to determine if a trading system or method is overfit and therefore unlikely to be profitable in the future. Specifically for students t test can be applied to the average transaction for the trading system or method concerned. The test determines whether the average transaction is significantly greater than zero at a specified confidence level. For example, the test will determine whether the average transaction is greater than zero with, for example, 95% confidence.
The test requires the number of rules and / or restrictions imposed by the trading system or method. The number of rules and / or restrictions is used to calculate the number of degrees of freedom, which is necessary to calculate the t-value for the t test. There must be a sufficient number of degrees of freedom to ensure that the system is not over-fit or over-optimized for the market. In an over-fit or over-optimized strategy, the parameters of the trading system are selected to operate in specific markets or under limited market conditions. When applied to new markets or other market conditions, the strategy is unlikely to hold out.
The number of degrees of freedom is the number of transactions minus the number of restrictions. In the case of too few transactions, the profitability of the system or method may be due to an accidental settlement of transactions. The more transactions, the greater the number of degrees of freedom and the more likely the calculated average profit is not a statistical fluke, but a real result that is likely to hold out in the future.
To count the number of restrictions, Thomas Hoffman (Babcock, Bruce. The Business One Irwin Guide to Trading Systems. Richard D. Irwin, Inc. 1989, p. 89) suggests examining the rules of a trading system and reviewing all conditions count the resulting transactions. Let’s say you have a trading system that buys when today’s closing price is less than yesterday’s closing price in an upward trend. It defines an upward trend as when a shorter moving average is greater than a longer moving average. For simplicity, suppose the sales side is the reverse and there are no stops. It is a simple stop-and-reverse system.
The moving average crossover state is likely to count as three constraints: one for the condition itself and one for each moving average period. The pricing pattern would be another constraint for a total of four long side constraints. There would be four more for the short side with a total of eight restrictions. For example, if there were only eight trades, there would be no degrees of freedom and you shouldn’t have confidence in the average trade, even if it were very high. On the other hand, if there were 100 trades, there would be 92 degrees of freedom, which should give you much more confidence in the average trade number.
The t test can be expressed as a confidence interval for the average transaction:
BI = t * SD / sqrt (N)
where CI is the confidence interval around the average transaction, t is that of the student t statistics, SD is the standard deviation of the transactions, N is the number of transactions and sqrt stands for “square root”. The t statistics depend on the number of degrees of freedom and the level of confidence.
The confidence interval means that the average transaction is likely to be between T – CI and T + CI. In order for the system to be profitable at the specified confidence level, the average transaction, T, must be greater than zero at the lower limit, T – CI; i.e. T> CI.
If this condition is true at the specified confidence level, it means that the system or method is inherently profitable subject to the assumptions of the test; that is, the strategy is not over-fit. One of these assumptions is that the statistical properties of the transactions remain the same. In particular, if the average trade and standard deviation remain the same in the future, the results remain valid. However, as markets change and evolve over time, the properties of the statistical distribution of transactions may also change, so caution should be exercised when interpreting the results.