Is testing an Expert Advisor using the MT4 Strategy Tester a total waste of time?

.

.

Is testing an Expert Advisor using the MT4 Strategy Tester a total waste of time?

The Big question on everybody’s mind is:-

“Is testing and optimizing an EA using the MT4 Strategy Tester a total waste of time?”

 

 

This discussion of the Strategy Tester is for Forex traders who use Expert Advisors as trading tools. It is important for all EA users and especially continuous trading, set and forget users of EA’s to understand the limitations discussed here.

Testing and optimization philosophy

When using the strategy tester to find likely settings to use for a particular EA one should not lose sight of the fact that we are using historic data and performance as a guide to plan future trading approaches. There is therefore no guarantee or likelihood that history will repeat itself and the same results can be achieved. The process starts with an inherent flaw.

But, in the absence on anything better we are like horse race punters who are using the history and information of the say 12 horses in the race to predict a likely outcome. We are trying to stack the odds in our favor by doing an intelligent analysis of the past.

If we do it well, just like horse racing, we will know under which conditions the EA performs best. We will know which conditions the EA should not be used.  We will know how risky or robust our intended settings are after taking the existing market conditions into account. Just like horse racing some horses like rainy conditions, others like hard or soft tracks and other have favorite jockeys. Forex trading is not much different.

We are trading in an uncertain market. Any bit of information that will push our odds over a 50% chance of success could be critical. That information could well be provided by EA testing and optimization.

Objectives of testing and optimization

The MT4 Strategy tester which has been around for over 12 years is one on the most used and abused Forex trading tools around. Here are some of the uses:-

Test the technical correctness of the EA.

When designing and coding an EA the strategy Tester is the ideal tool to test whether the EA is functionally working the way it was intended to. An EA is after all only an automation of what can be done manually. So an EA developer will go through the EAs trading process step by step to see if it is entering and exiting deals as intended. Astute EA user do this too. They want to see exactly what they can expect from the EA.

Find clumps of settings that produced good results

The heat-maps produced by the MT4 Strategy tester are very useful to determine how robust settings are and finding clumps of successful settings. The heat map below show testing results with very specific successful results (the darkest is the most successful) and show that even with a margin of error results are likely to be positive

goodtestingheatmap

 

The heat-map below shows inconsistent results with optimized settings scattered – these setting give very little margin for error and over optimized

bad heatmap

Finding the best setting that worked in the past

The most common reasons for using the Strategy tester is to find the best settings that worked in the past. This is where the abuse and misunderstanding comes in. The best setting in the past can be very fragile. They can vary dramatically based on the starting and stopping point within a test. Some tests can go from making an incredible profit to a big loss by starting 1 day earlier or later. All though they add to the information about the EA the best settings can be highly misleading. The the different heat-maps above.

Find conditions when the EA was most profitable and made the biggest losses

Probably the biggest benefit of EA testing is identifying the conditions when the EA is most profitable and when it experiences draw-downs. This teaches traders when to use the EA or not or when to increase lots or decrease lots. This is were having the trades reflected on charts is very useful when viewing the equity graph at the same time.

It also helps EA designers build filters to make the EA more sensitive to current market conditions. Only a very few Forex traders use this facility.

There could even be settings for certain favorable markets and settings for unfavorable markets. Ignoring this is one of the reasons Forex traders often give back their gains after a successful trading period.

Great Forex Testing Tools (Click on boxes below for more info)

EA Testing TOOLs

Assumed testing conditions

Quality tick data: It is assumed that high quality tick data is used for all testing and optimization. Anything less than that will result in poor conclusions.

Creative testing techniques: The Generic Algorithm is best used to do initial very high level testing and automation. It should not be relied on to produce the final results. Using very small tests with only 10 to 20 alternatives being tested at a time with 10 to 20 iterations not using the Genetic Algorithm produces better and more reliable results.

Testing based on the nature of the EA: If the EA makes decisions based on the close of a candle testing is much more simplified than it decisions are based on price levels. Price level decision making requires high quality tick data and tick data testing methods that give 99% test quality.

Limitations of testing

Spreads. Testing is normally done using fixed spreads. In reality most brokers vary their spreads considerably. Especially near announcements and over weekends. Unless you are using a fixed spread broker your test results can vary considerably from live trading results.

Slow testing: Detailed tick testing can be very slow. All the more reason why a number of small tests are more efficient than one large test.

Tick testing. Even when using the tick testing option the Strategy tester does not use true ticks but manufactures ticks in most cases.

Genetic algorithm. The Genetic Algorithm is good for high level testing and planning your testing strategy. It can miss true optimization settings and give an incomplete picture. Best to adapt testing methods to make minimal or no use of that facility.

Technical aspects of the strategy tester: Occasionally during large tests the tester will hang. All the more reason to use small tests. In most cases stopping and restarting the test immediately will cause the tester to continue from where it stopped.

Competency of testers. Effective testing requires trading experience and the ability to interpret results in a meaningful way. A sound testing approach and strategy is important. There is a high risk of testing occurring using 1 optimization of all settings in one run over a day or two without much thought being given to how practical the optimized setting are in practical trading. Much of the negative publicity about EA testing and optimization has to do with tester experience and ability.

 

Conclusion

Testing and optimization of an EA, even when done well, has limitations that any Forex trader must be aware of. The process however can provide incredible insights and direction as to how an EA should be used and how robust the EA is. It is up to the trader to use the MT4 Strategy Tester Testing and optimization process to achieve his / her goals and objective. It would be foolhardy to think that test and optimization will guarantee trading success but at the same time to ignore it unless you have something that provides you with more confidence.

Free Testing and Optimization Course

Click on this link for a FREE testing and Optimization course worth $160          FREE OPTIMIZATION COURSE 

Reader feedback

Please add your EA optimization and testing experiences in the comments section below. It would also be interesting to find out if there are traders who have sound better and more reliable methods of using new EA’s.

 

3 Comments

  • James King

    February 20, 2015

    Good article. You have made a lot of important points in a short space.

    I have been using Strategy Tester for more than 10 years, mostly for my own EAs.

    I always un-check use of Genetic Algorithm and any of the other choices as I learn more from all the test results, not filtered ones. I used tick data a lot for a couple years but I now think it does not give much of an advantage unless you are trading a scalping strategy and the tick data comes from the broker you are trading your real money at.

    In fact Open prices testing model works very well and extremely fast for me for getting a strategy “in the ballpark” and then I fine tune from there with Every tick model and proper walk forward testing. Most important with walk forward testing is to thoroughly test the most challenging historical periods of data, to actually curve fit looking for the worst case results, NOT the best.

    The Optimization feature is a great one. It really shows you well all the vulnerabilities of your strategy and also which direction to go in with your settings values for better results. I have developed a program I call FORO . It enables saving Optimization reports in a way that they can be much better analyzed and more useful and also easily shared with others publicly or privately as well as being able to view or download both the reports and preset files for any of the passes plus the optimization preset and report. I generally optimize 4 to 6 settings to produce reports with 300 to 600 Passes which are completed overnight using Every tick method.

    FGM only needs M1 data for back-testing, no matter what the strategy is, but it’s important that it be quality M1 data. For that we use the free MT4 M1 data from Histadta … http://www.histdata.com/download-free-forex-data/?/metatrader/1-minute-bar-quotes … which is excellent, including a detailed description of the data itself so you know exactly what you have.

    What’s lacking in Strategy Tester is the ability to back-test Portfolio trading but this can be done by using a program like EA Analyzer v.3.

    I always make sure a strategy can survive a 5 year back-test up to present day, but it is also mostly fine tuned for the last year or two, what will most likely match the future price action best.

    There are definitely shortcomings with Strategy Tester, but with enough experience paying attention to all the details and doing enough back-testing, I have found Strategy Tester to be a great asset.

    To take your back-testing and strategy optimization to a whole new level, then learn how to use Matlab, an amazing tool … http://www.mathworks.com/products/matlab/.

    May the pips be with you,
    James

  • grandong

    February 21, 2015

    I have MAGIC MA EA, Happy Envelope EA & DIAD EA.
    With Respectfully, I have a question?

    What happen with The Happy Envelope, Magic MA & DIAD EA?
    The main topic of All these EA are DIAD. But after optimized, the result is 600$ turned into 2000$ in one year back test.

    So it is not DIAD anymore, but DIAY or DI6M, not even DIAM.

    I am waiting for your better version.

    • info@expert4x.com

      February 22, 2015

      Thanks very much for your comment – Grandong

      The most recent versions of the Double in a Day EA, The Magic MA and the Happy EA’s are available from thses links. If you own them you can download the mosr recent version from the EAFactory website.

      http://www.doubleinadayforex.com/
      http://www.alwayswiththetrendea.com/
      http://www.theforexexpertadvisor.com/

      I would suggest that you use the skills suggested in this article along with the FREE COURSE course supplied to optimize and test these EA if you own them.

      A +300% return seems very good to me ($600 becomes $2000 in 1 year)

      The Double in a Day EA has not changed its name and remains the DIAD

      I hope this helps to clarify your questions

Leave A Response