In conclusion we can state that optimisation is something variable in terms of data window since systems need to be kept in synchronisation with the market. Before computer power became so cheap and easy to employ for the majority of market players, an "out-of-sample period" was always recommended after optimisation by all the trading systems' developers. The "out-of-sample period" is a data window (usually 10 to 20% of the whole optimisation data window) we keep outside the optimisation process and which we apply to the optimised trading system in order to verify its forecasting power over unseen data. If the system performs in the same manner over the unseen data of the "out-of-sample data" it means that the system is a robust one and it can be traded with confidence.
So far we have only discussed ideas that you can read in any of the books in circulation about trading systems. But this is an obsolete view of optimisation, maybe dating back to those times where computer power was neither cheap nor widely available. Today optimisation has evolved into a more efficient and proper method of testing and making a system fit over a long price series. This method goes under the name of "walk forward analysis" or "walk forward testing".
Walk forward testing is a kind of multiple and successive out-of-sample test over the same data series. Let us give an example: a system is optimised over the first two years of the data history and then applied over the subsequent 6 months of unseen data. Then the optimisation window moves ahead by a 6-month period and a new optimisation takes place in order to find the new inputs that will be applied over the forthcoming 6 months of data. And so on. This kind of optimisation is a "rolling" walk forward analysis since the starting period of the optimisation window is always moving ahead by a 6-month period each time we re-optimise the inputs. If the starting period is always the same and the optimisation period gets longer and longer as the time goes by we have an "anchored" walk forward analysis. For intraday systems the "rolling" walk forward analysis is more appropriate since intraday trading systems are more suited to the changing market conditions.
Figure 2.1 : A graphical description of a "rolling" and "anchored" walk forward analysis
Rolling walk forward: out-of-sample (OOS) = 20%:
Run #1 I---------In-sample 80%--------------I OOS 20% I
Run #2 I----------In-sample 80%------------I OOS 20% I
Run #3 I----------In-sample 80%---------------------1 OOS 20% I
Anchored walk forward: out-of-sample (OOS) = 20%:
Run #1 I--------------In-sample 80%---------------I OOS 20% I
Run #2 I-----------------------------In-sample 80%---------------1 OOS 20% I
Run #3 I--------------------------------------------In-sample 80%---------------I OOS 20% I
The equity line resulting from a walk forward run is where we are closest to reality in trading systems development since it is what real trading will produce in our pockets. And with no surprise this walk forward analysis equity line will be deeply different from the equity line we can produce with testing and optimising a trading system on the whole price series. So often traders fool themselves deciding whether a trading system is to be discarded or not based on a whole price series equity line that in reality reveals nothing about the real trading situation after periodic re-optimisation.
A widely accepted way to gauge the forecasting power of a system and its consistency is to calculate the ratio between the annualised net profit relating to the walk forward tests and the annualised net profit reaped during the optimisation periods. This is the walk forward efficiency ratio. If the ratio is above the 100% threshold then the system is efficient and the probability that it will keep its forecasting power during real trading is high. If a trader decides to trade a system with a walk forward efficiency ratio of just 50% (and many traders accept this level as the lowest possible) they should expect a system that performs at least at half the level of the performances indicated into the optimisation test. Statistical evidence also pinpoints that a poorly optimised system could make good performances on some lucky one or two walk forward tests. To avoid this trap the highest possible number of tests should be performed or at least 10 walk forward analysis tests with a test window (that is the data window where we apply the optimised trading systems) of at least 10 to 20% of the whole optimisation price series.
Every comment on the old type of static "out-of-sample" testing on the last part of the price series or on how to optimise a trading system is nowadays obsolete since most professional trading system development software has a walk forward analysis feature (like for example most of the RINA Systems products and in particular Portfolio Maestro). This does not mean that traders should not become accustomed to the ordinary testing and optimisation process. We recommend before using WFA you should do the ordinary homework about optimisation in order to acquire a full view of the system and its performances. To run a full walk forward analysis takes much time, so it is quicker to check the robustness of the system with a shift test and then another shift optimisation. In any case, for the sake of simplicity we will summarise some good tips about optimisation.
If we have many inputs to be optimised the best methodology is to test one or two inputs per turn while all other inputs are kept static. In this way the risk of over-optimisation is kept at the lowest level since it is impossible to find the batch of inputs that will maximise the constraint we gave to the equation simply because the inputs will not be optimised together in the same run.
Was this article helpful?
Guaranteed To Transform You Into A Professional Trader. This Book Is One Of The Most Valuable Resources In The World When It Comes To Create Fortunes Guide. The Mindset Behind The Most Successful Forex Traders In The World. The Costs of NOT Getting All The Info You Need Are Just Too High.