Having researched and designed your trading system by working through the steps we have discussed so far it is now time to back-test your ideas to see if the system ‘works. Research is conducted via back-testing concepts against available historical data. This is where your programming skills are required, unless you wish to conduct your research manually or visually.
A word of caution regarding visual back-testing: your bias of wanting the rules that you are testing to work will block out instances of price action that result in loss trades, missing trends by exiting too early. These are called ‘scotomas’ or ‘blind spots’. This is also referred to as ‘perceptual blindness’ where price action that contravenes what you believe should happen (or wish to happen) according to your chosen rules is simply not seen on the chart. This occurs in live trading as well as visual back-testing. When a consistent mindset is achieved the researcher becomes far more objective – objectivity is a subset of consistency – and perceptual blindness is mostly overcome when doing research or in live trading.
Perceptual blindness when doing research will result in a skew towards omitting losing trades, including trades that don’t actually meet the rules and overstating winning trades, which will make the research results look far better than they really are. There is a high probability of disaster taking such a set of rules into live trading.
Computer based research will always be far more objective than visual back-testing. Merely learning a programming language is not enough. Programming techniques also need to be learned which will be covered in any half decent programming course.
Choose a concept or combination of concepts from the list discussed in last week’s blog, or elsewhere, including the internet, magazines such as Stocks & Commodities and books that contain ‘base’ trading systems. Then code into your chosen research tool (e.g. Wealth-Lab, TradeStation, Trade Navigator) the criteria to filter price action and/or volume and to determine entry and exit points. Remember that the chosen concept needs to match your stated requirements for the system you are researching. For example, using a momentum indicator on daily charts simply will not produce a profitable, let alone robust, system that has average hold periods of one to five days.
Typically, a filter should be used to improve the win rate of your system and control the opportunity to trade that is required in your stated objectives. A filter could be:
• a moving average
• comparative relative strength against an Index
• inter-market analysis
• fundamental analysis
• Elliot Wave
• Fibonacci retracements
• a combination of the above, or others
Once you have programmed your research concept, and removed all the bugs from the code, the next step is to test your rules against a set of ‘in sample’ historical data. Ensure that you use clean data that closely represents price action in historical live trading. I suggest that you spend some time researching this topic, especially if you are working with futures markets where historical data mostly needs to be normalised for research purposes. Futures contracts expire at regular intervals and if the data is not ‘back adjusted’ to reflect changes in the values of contract months the data will be virtually irrelevant for back testing purposes..
‘In sample’ testing data should be historical data that has suitable information to generate sufficient trades to provide a large enough statistical sample for the timeframe you are trading. For example, if your trading timeframe is intra-day using five minute bars, then one to two months’ data will be sufficient. If your trading timeframe is medium to long term using daily and weekly bars, then five years of historical data should be used and > 10 years for monthly charts.
As a rule of thumb, the ‘in sample’ data set should cover at least two cycles each of markets moving up, down and sideways. This usually requires a subjective ‘eye-ball’ call of the overall market. Again, be careful not to succumb to your personal biases when using your subjectivity. Basic psychology books will inform you of the generic biases that we all have. These biases may assist us in other parts of our lives such as surviving in business or society but hinder us when it comes to trading.
‘Out of sample’ testing data should then be used to test the system rules on a different set of price action data. You are looking for similar results between the two sets of sample data or, as a minimum, one set not being drastically different to the other.
●——————————————●————————-●—–
In sample data Out of sample data
If testing a single market, such as an index or a currency, then using ‘in sample’ and ‘out of sample’ is very important. However, if testing an equities system, then testing price action across, say, 200 to 500 stocks over many years, will provide a wide variety of price action as a test bed. It then becomes more important to test the system rules in different types of all market action, i.e. rising, sideways and falling markets. Using an overall market index like the All-Ords to determine whether the overall market was a bull or a bear should then be brought into the sample test data and tested with trading rules. Such rules might fall under the heading of risk management rules.
Another issue that requires mention is liquidity, which is more applicable to equity trading systems than single market trading systems. Ensure that your research code includes criteria for excluding price action where there is insufficient liquidity to trade. For example, researching price action where an average of $10,000 a day is traded for a particular instrument will be too illiquid for a portfolio that will trade average position sizes of $10,000. Remember, price action that was “illiquid” 10 years ago may be very liquid now. In my experience, trapping illiquid trades that you simply could not have traded historically artificially improves an edge.
In next week’s journal posting I will discuss the welding together of the concepts we have been discussing over the past few weeks to arrive at a winning trading system.
Response to Comment by Marko:
“You are certainly building a strong case for never attempting this on ones own.”
Absolutely not! Just being realistic about one’s expectations on what it takes to build a system. All too often the task is oversimplified. In fact, enduring this journey is a fantastic experience and those that complete it don’t regret it and also learn heaps. Researching systems condenses ones learning about price movement and hence trading.
“The message is that it is much cheaper to purchase than to try and reinvent the wheel.”
Not always. Depends on what is purchased and how the individual performs with it. Active investors can lose money with even the most profitable of systems beacuse there are more variables involved in trading outcomes that just the system.
Regards
Gary
Re the thoughts of Gary.I agree with your thought of how you perform with the stock.
Do you have a trailing stop?Will greed become your most dominant emotion?Do you like to have a good nights sleep?We are all made up of different needs and emotions.It has been stated by many experts ,finding that stock[at that best price] is far easier than selling at the right time.[only my thoughts]
Regards,
Garry.
Gary, what is the best software to use to back-test? Is Ninja Trader any good? Is there software that will back-test a portfolio rather than simply one share at a time? Can anything useful be achieved without learning a programming language? What is the easiest language to learn?
Sorry for the barage but I have been struggling with these issue for some time.
Response to Comment by Mike:
“what is the best software to use to back-test? Is Ninja Trader any good? Is there software that will back-test a portfolio rather than simply one share at a time?”
The three research products that I have a lot or some exposure to are Wealth-Lab, TradeStation and AmiBroker. All three of these will back test at the portfolio level.
There are quite a few others around these days in which indicators can be programmed but whether they support back testing at the portfolio level I am not sure. I cannot answer the question for Ninja Trader. I do know that MetaStock requires an add-on to acheive portfolio level research.
“Can anything useful be achieved without learning a programming language?”
In my view, no! You will need to learn to program otherwise you will spend many many more unproductive hours conducting manual / visual research. I say unproductive because manual backtesting is fraught with the danger of making mistakes.
“What is the easiest language to learn?”
The benefit of tools such as the three mentioned above is that they have many technical analysis functions preprogrammed which would need to be programmed by yourself if using a language such as VB.Net, C++ or C#. Nevertheless you still be to learn programming techniques, structure and syntax.
If you are starting from scratch perhaps C# would be the preferred language to learn. It is easier to learn than C++. C# is the base language for Wealth-Lab Verion 5 (Pacal was for WLD V4) and C++ is the base language for AmiBroker. Both are far less expensive than Trade Station. I suggest that you limit your decision to either of Wealth Lab or AmiBroker.
Regards
Gary
Alexander Elder, a very reputable writer of trading literature, recommends visual testing over programming. If you take a chart then take your rules and slowly slide the right hand side into the picture as time progresses you can see how your trades would have done and how you would have reacted to your signals in a simulation of real-time trading. He argues that this gives you a better feel for how your markets move and the emotions you will encounter although it is very time consuming. How do you respond to this approach to back-testing given your comment that programming will always be the better approach?
Response to Comment by James:
There defintely is a place for visual testing but the problem is that we all have biases that originate from our beliefs about the market, about ourselves and about many other things which will all cause us to interpret what we see, feel, hear, smell and taste in accordance with those beliefs. Most of us have some beliefs that are dysfunctional with respect to what we are individually trying to achieve in the market.
When doing visual back-testing we will often experience scotomas, or “perceptual blindness”, caused by our dysfunctional beliefs, meaning that we will see what we want to / are trained to see or not see what we don’t want to see / are trained not to see. This happens at a subconscious level so we are not even aware of what we are missing.
Now this won’t happen for every signal that is being back tested but it will happen enough to make a difference to the outcome of the system rules being tested. That is, we will make mistakes with our visual back testing.
Furthermore, we need to back test multiple hundreds of trades for any set of criteria that define an edge which means that there will a high level of errors in our visual back testing. When testing many trading concepts doing it visually will chew up massive amounts of time and your manual results will contain errors.
Nevertheless, every computer programmed system must be inspected visually for the trader to become visually familiar with the entry and exit criteria but this will happen at the step when the concepts for the system are decided upon and then defined in terms of specific criteria. The programming of those criteria and the computerised testing thereof will provide OBJECTIVE results about whether an edge exits or not and, if so, how good the edge is.
By the way, Alexander Elder also has programmers code and test trading concepts that he comes up with even though he doesn’t do it himself.
I trust this clarifies where visual and computerised backtesting fit into the process.
Regards
Gary
You are certainly building a strong case for never attempting this on ones own. The message is that it is much cheaper to purchase than to try and reinvent the wheel.
Test Elder’s methods and you’ll be shocked how bad they are.