Outofsample trading performance measures

Statistical performance measures are often inappropriate for financial applications. Typically, modelling techniques are optimised using a mathematical criterion, but ultimately the results are analysed on a financial criterion upon which it is not optimised. In other words, the forecast error may have been minimised during model estimation, but the evaluation of the true merit should be based on the performance of a trading strategy. Without actual trading, the best means of evaluating performance is via a simulated trading strategy. The procedure to create the buy and sell signals is quite simple: a EUR/USD buy signal is produced if the forecast is positive, and a sell otherwise.20

For many traders and analysts market direction is more important than the value of the forecast itself, as in financial markets money can be made simply by knowing the direction the series will move. In essence, "low forecast errors and trading profits are not synonymous since a single large trade forecasted incorrectly ... could have accounted for most of the trading system's profits" (Kaastra and Boyd, 1996: 229).

The trading performance measures used to analyse the forecasting techniques are presented in Tables 1.17 and 1.18. Most measures are self-explanatory and are commonly used in the fund management industry. Some of the more important measures include the Sharpe ratio, maximum drawdown and average gain/loss ratio. The Sharpe ratio is a

18 The MAE and RMSE statistics are scale-dependent measures but allow a comparison between the actual and forecast values, the lower the values the better the forecasting accuracy.

19 When it is more important to evaluate the forecast errors independently of the scale of the variables, the MAPE and Theil-U are used. They are constructed to lie within [0,1], zero indicating a perfect fit.

20 A buy signal is to buy euros at the current price or continue holding euros, while a sell signal is to sell euros at the current price or continue holding US dollars.

Table 1.16 Statistical performance measures

Performance measure


Mean absolute error

0 0

Post a comment