In this week’s posting I would like to explore in more detail the concepts of degrees of freedom and optimisation and the impact these two variables can have on the development and outcomes of a trading system.
Degrees of freedom in a trading system relates to the number of criteria that are used to filter price action and/or volume and determine entry points. The more criteria and variables used to determine entry timing points, the fewer degrees of freedom the system will have and vice versa.
The aim is to maximise the degree of freedom the system has to generate entry signals by minimising the criteria per entry signal. This means that you want to use as few filters and indicators as possible to determine your entry points. While there is no certainty, the less constrained a system is by rules the higher the probability that the system will continue to work in the future.
Obviously, the more freedom the system has to generate entries, the more signals that will be generated. The more signals that are generated for a particular trading timeframe, the lower the percentage win rate will typically be. Therefore, this becomes a balancing act to not constrain the system too much but to improve the percentage win rate.
Once a concept is chosen to research, the process used to attempt to improve the research results is to change the parameter settings for a particular concept. For example, a daily period setting of 34 may be used for an exponential moving average. Then change to 30 days and run the same dataset comparing the results to those with the period set at 34. Then a setting of 26, then 21, then 18 can be tested and so on. This process is called optimisation.
It is impossible to conduct historical back-testing without any optimisation but too much optimisation can lead to curve fitted results, meaning that there is a high probability that the researched concepts only produce profitable results on the sample testing data and not on any other data. This would obviously not bode too well on future price action data, meaning that there is a low probability of the system attaining its researched results in future live trading.
Again this is a balancing act. How well the system designer balances the degrees of freedom and optimisation of the system variables and decision criteria comes with the experience of designing systems and then trading them with live price action. As a rule of thumb, if your research results are “too good to be true” then one or both of these issues may be at work in your system.
Over optimisation is a real trap for those new to system development or systems trading. There are countless examples of some miraculous ‘trading systems’ for various markets available on the Internet – systems that “never have losing trades” or “never have losing months” and various other ridiculous and dubious claims. Whilst it is quite possible that these claims could be correct over a very small data sample or date range, in the long term these systems are unlikely to continue their extraordinary success. This is predominantly because these so called ‘systems’ are over optimised or curve fitted to a specific sample of data or market conditions. When applied in real time, or walking forward in the market as conditions change, these systems will inevitably ‘blow up’ or continue to lose money.
A give-away sign of such systems are those that generate very few signals in a liquid market or relative to the number of instruments that are considered for trading. For eample, a system that looks at multiple hundreds of stocks and consistently only generates a couple of signals a month or one that trades a single market and only generates a handful of trades a year over all market cycles.
I hope you have enjoyed reading and learning about the process of system development that we have been discussing over the past several blogs. If you are either developing your own system or considering it you will now have plenty of useful information and a process to work through to achieve your goal. If you have no interest in developing your own system but are interested in purchasing a commercially available system or systems for engaging the market then you now have ample information on which to base your due diligence and analysis of those systems. You will now be able to approach either process with a list of questions to ensure that the system is robust, profitable and has been thoroughly thought out, researched and tested BEFORE it is applied to live trading.
Response to Comment by Nigel:
These are all good exercises to conduct. Others include portfolio equity curve testing, Monte Carlo testing and even testing on randomly generated data. All these add to the robustness of the system which builds confidence to trade it.
Ultimatley though, if one has a system that has a decent edge, has been tested on many years of data that includes different market types and has a large degrees of freedom, a trader will make money provided they can get their mindset right to accept drawdown and continue trading through thick and thin.
As difficult as some think it is to get a robust method it is a far bigger challenge to get a robust mindset.
Regards
Gary
Wise advice again Gary.
I developed a slow trading model that works under all market conditions, but only after much research, backtesting/stress testing and drawing on the best technical literature from the USA and elsewhere. We learnt very early to avoid curve-fitting and optimisation since that can lead to false results.
For instance an 80 day exponential moving average would have avoided the crash of Nov 2007-Mar 2009 and ridden the rally since then. But back-testing shows that in earlier periods it was subject to frequent and expensive whipsaws. There are horses for courses, but a good model has an edge on all courses.
Keep up your excellent newsletter.
Gary, another interesting article on trading system design. Some things I do to check whether I have inadvertently curve-fitted my test results are:
1. Vary all the parameters I have used up a bit (about 10%) and then again down a bit. If the results are similar then the system may be OK.
2. For a weekly system I can change the end of week day from Friday to any other day I like. If this gives very different results then I probably have not got a decent system.
3. Run tests over one set of data (eg the first 150 stocks, alphabetically, in the ASX300) and then, once the system looks good, run it over another set of data (eg the other half of the ASX300) based on the same time period. Again, if the results are very different, there may be a problem.
Quite often I find something that looked quite promising, but turns out not to work. And it means that the urge to tinker with my system is satisfied without damaging it!