Is testing an Expert Advisor using the MT4 Strategy Tester a total waste of time?

.

.

The Big question on everybody’s mind is:-

“Is testing and optimizing an EA using the MT4 Strategy Tester a total waste of time?”

This discussion of the Strategy Tester is for Forex traders who use Expert Advisors as trading tools. It is important for all EA users and especially continuous trading, set and forget users of EA’s to understand the limitations discussed here.

Testing and optimization philosophy

When using the strategy tester to find likely settings to use for a particular EA one should not lose sight of the fact that we are using historic data and performance as a guide to plan future trading approaches. There is therefore no guarantee or likelihood that history will repeat itself and the same results can be achieved. The process starts with an inherent flaw.

But, in the absence on anything better we are like horse race punters who are using the history and information of the say 12 horses in the race to predict a likely outcome. We are trying to stack the odds in our favor by doing an intelligent analysis of the past.

If we do it well, just like horse racing, we will know under which conditions the EA performs best. We will know which conditions the EA should not be used.  We will know how risky or robust our intended settings are after taking the existing market conditions into account. Just like horse racing some horses like rainy conditions, others like hard or soft tracks and other have favorite jockeys. Forex trading is not much different.

We are trading in an uncertain market. Any bit of information that will push our odds over a 50% chance of success could be critical. That information could well be provided by EA testing and optimization.

Objectives of testing and optimization

The MT4 Strategy tester which has been around for over 12 years is one on the most used and abused Forex trading tools around. Here are some of the uses:-

Test the technical correctness of the EA.

When designing and coding an EA the strategy Tester is the ideal tool to test whether the EA is functionally working the way it was intended to. An EA is after all only an automation of what can be done manually. So an EA developer will go through the EAs trading process step by step to see if it is entering and exiting deals as intended. Astute EA user do this too. They want to see exactly what they can expect from the EA.

Find clumps of settings that produced good results

The heat-maps produced by the MT4 Strategy tester are very useful to determine how robust settings are and finding clumps of successful settings. The heat map below show testing results with very specific successful results (the darkest is the most successful) and show that even with a margin of error results are likely to be positive

The heat-map below shows inconsistent results with optimized settings scattered – these setting give very little margin for error and over optimized

Finding the best setting that worked in the past

The most common reasons for using the Strategy tester is to find the best settings that worked in the past. This is where the abuse and misunderstanding comes in. The best setting in the past can be very fragile. They can vary dramatically based on the starting and stopping point within a test. Some tests can go from making an incredible profit to a big loss by starting 1 day earlier or later. All though they add to the information about the EA the best settings can be highly misleading. The the different heat-maps above.

Find conditions when the EA was most profitable and made the biggest losses

Probably the biggest benefit of EA testing is identifying the conditions when the EA is most profitable and when it experiences draw-downs. This teaches traders when to use the EA or not or when to increase lots or decrease lots. This is were having the trades reflected on charts is very useful when viewing the equity graph at the same time.

It also helps EA designers build filters to make the EA more sensitive to current market conditions. Only a very few Forex traders use this facility.

There could even be settings for certain favorable markets and settings for unfavorable markets. Ignoring this is one of the reasons Forex traders often give back their gains after a successful trading period.

Great Forex Testing Tools (Click on boxes below for more info)

Assumed testing conditions

Quality tick data: It is assumed that high quality tick data is used for all testing and optimization. Anything less than that will result in poor conclusions.

Creative testing techniques: The Generic Algorithm is best used to do initial very high level testing and automation. It should not be relied on to produce the final results. Using very small tests with only 10 to 20 alternatives being tested at a time with 10 to 20 iterations not using the Genetic Algorithm produces better and more reliable results.

Testing based on the nature of the EA: If the EA makes decisions based on the close of a candle testing is much more simplified than it decisions are based on price levels. Price level decision making requires high quality tick data and tick data testing methods that give 99% test quality.

[cbpress product=”751″]

Limitations of testing

Spreads. Testing is normally done using fixed spreads. In reality most brokers vary their spreads considerably. Especially near announcements and over weekends. Unless you are using a fixed spread broker your test results can vary considerably from live trading results.

Slow testing: Detailed tick testing can be very slow. All the more reason why a number of small tests are more efficient than one large test.

Tick testing. Even when using the tick testing option the Strategy tester does not use true ticks but manufactures ticks in most cases.

Genetic algorithm. The Genetic Algorithm is good for high level testing and planning your testing strategy. It can miss true optimization settings and give an incomplete picture. Best to adapt testing methods to make minimal or no use of that facility.

Technical aspects of the strategy tester: Occasionally during large tests the tester will hang. All the more reason to use small tests. In most cases stopping and restarting the test immediately will cause the tester to continue from where it stopped.

Competency of testers. Effective testing requires trading experience and the ability to interpret results in a meaningful way. A sound testing approach and strategy is important. There is a high risk of testing occurring using 1 optimization of all settings in one run over a day or two without much thought being given to how practical the optimized setting are in practical trading. Much of the negative publicity about EA testing and optimization has to do with tester experience and ability.

Conclusion

Testing and optimization of an EA, even when done well, has limitations that any Forex trader must be aware of. The process however can provide incredible insights and direction as to how an EA should be used and how robust the EA is. It is up to the trader to use the MT4 Strategy Tester Testing and optimization process to achieve his / her goals and objective. It would be foolhardy to think that test and optimization will guarantee trading success but at the same time to ignore it unless you have something that provides you with more confidence.

Free Testing and Optimization Course

Click on this link for a FREE testing and Optimization course worth $160          FREE OPTIMIZATION COURSE 

Reader feedback

Please add your EA optimization and testing experiences in the comments section below. It would also be interesting to find out if there are traders who have sound better and more reliable methods of using new EA’s.