MetaTrader EAs: The "Set & Forget" Myth [EAS-II]

This is the second post in the MetaTrader Expert Advisor [EAS] series we’ve begun recently.
In case you missed it, here’s a link to the first post:
Commercial Expert Advisors: everything that glitters is not gold.

If you’re wondering what “set & forget” means, it’s a common catch phrase used widely by many commercial MetaTrader EA vendors.
It implies that the prospective user of the EA advertised as “set & forget” need do nothing more than purchase, install and run said EA (usually accompanied by a notice to follow all rules, instructions etc as supplied by the vendor).
In this post, we’ll discuss some popular misconceptions centered around this “set & forget” EA-marketing phenomenon.
We’ll present a set of realities that attempt to dispel the notion, realities that are usually not presented by most EA vendors to retail forex traders.
By the end of this blog post, you’ll have developed a keen, rather chiseled understanding of what to look for in an MT4 Expert Advisor, how EA vendors are much better off listing DARWINs instead, and how “set & forget” is more “marketing sizzle” than it is ever a reality.

.. so let’s begin 🙂

“Set & Forget”: Myth vs. Reality

Almost all commercial MetaTrader EA websites will promote:

1. The EA’s performance over historical and live data,
2. Target asset mix, risk management and execution,
3. Testimonials of other EA users validating the information above.

For the remainder of this post today, we’ll focus on the first point.

Backtests & Live Forward Tests

Profitable backtests showing a MetaTrader EA’s performance over many years, do add some credibility to a vendor’s effort.

However.. as easy to use a platform as MetaTrader is,

any backtest conducted in the MetaTrader 4 Strategy Tester cannot have taken the following into consideration:

• Variable slippage & spread
• Valid, temporally accurate swap charges (especially for trading strategies employing high leverage and/or those trading a large number of currency pairs over several days)
• Variable market impact due to out-of-sample political and/or economic developments
• Event-driven changes in a broker’s margin requirements
• Execution latency, e.g. due to VPS co-location or infrastructural constraints.
• Technical issues & human error, e.g. operating system or other malfunction (e.g. failed VPS or computer hardware), failed credit card payments to VPS services, etc.
• Unforeseen, sudden market movements, e.g. the GBP Flash Crash.
• COMPLETELY different market conditions to those experienced during the backtest (e.g. what would it have looked like had conditions been entirely different? what is the probability of a particular backtest being “lucky”?)

Let’s discuss each of these in turn.

Variable Slippage & Spread

At the present time, MetaTrader’s Strategy Tester enables EA developers to test a fixed spread applied across an entire backtest.
Using additional third-party tools, it is however possible to simulate randomized spread and slippage across an entire backtest.
At bare minimum, unless a vendor has made the effort to at least simulate the effects of these two variables on historical returns, the likelihood is low of the EA performing similar to the backtest had it actually been live during that time.
Indeed, vendors could boost their product’s credibility considerably by presenting its DARWIN listing, showcasing investment attribute scores & distributions.

When confronted with the above, retail forex traders must tread carefully.

Of particular interest in this case, would be scores attained for Capacity & Divergence. i.e. the effects of different levels of pip divergence on strategy returns, and trader vs estimated investor returns respectively.
Furthermore, if the EA vendor is promoting the expert advisor as “only suited to fixed spread environments”, you can rest assured that any live forward tests have been conducted with a non-DMA broker (see here for more on this and associated risks).
On that note, you may also wish to read the section on “when a live track record just isn’t enough” in this recent post.
—

Valid, temporally accurate swap charges

For trading strategies holding trades longer than intraday timeframes (particularly those employing high leverage), accurately accommodating swap charges in MetaTrader’s Strategy Tester is impossible without external tools and access to a time series of broker swap history.
For more information on the exact treatment of swap charges in MetaTrader’s Strategy Tester, please visit this link.
For loss averse trading systems in particular (that hold trades for long periods of time, add to losing positions in an attempt to recover prior losses, etc), “setting and forgetting” such an EA could prove to be a futile exercise if for example, the EA’s target asset mix contains high swap pairs (e.g. EA’s that scalp the NZD/JPY and AUD/JPY around rollover times).

Variable market impact due to out-of-sample developments

Under the definition of “set & forget”, some EA vendors may imply that a MetaTrader expert advisor that survived say, the global financial crisis of 2008 in a backtest, could also survive crises in the future.
Emphasis may be observed in their marketing material on the EA’s robust performance “in all market conditions”, weathering the storms of past crises successfully.
Claims may also be bolstered by references to the length of the backtest as “evidence” to this effect.

When confronted with the above, retail forex traders must tread carefully.

Just as past performance is not indicative of future performance, weathering past storms (let alone in a backtest) is certainly not indicative of weathering completely different storms in future – be they the result of political and/or economic shifts in the market, to name a few.

Event-driven changes in a broker’s margin requirements

Sometimes brokers may need to adjust their margin requirements due to regulatory and/or market risk related issues.
This can significantly impact strategies that rely on excess leverage for example, and is impossible to model chronologically inside a MetaTrader backtest.

Execution latency

Backtests in MetaTrader model returns assuming execution remains statically available.
In live trading however, order execution latency (the difference in time and/or price, between submitted and filled orders) can be frequent depending on factors such as:

1. Trade frequency,
2. Available liquidity,
3. Time of submission (e.g. before, during or after news?)
4. Size of submission (e.g. 0.1 lots vs 100.0 lots)
5. ..to name a few.

Modelling this near-stochastic process adequately is both difficult and infeasible in MetaTrader backtests.
—

Technical issues & human error

Simulating the effects of software/hardware malfunction, bounced payments to VPS services, dropping coffee on your laptop (or server casing?), on the returns of a strategy is certainly not “impossible”.

Possibilities include:

• randomly excluding a variable number of trades..
• ..during variable periods of time,
• ….for variable lengths of time,
• ……over variable samples of underlying asset prices,
• ………at different sub-samples of trading hours,
• ………..running 10,000+ runs in Monte Carlo simulations over the same… and,

……. well, hopefully you can see why a commercial EA vendor may need tremendous motivation (not to mention computational power!) to go to such trouble.. 🙂
Yet the risks of technical issues and human error drastically changing live trading returns, are very real, rendering “set & forget” a very risky proposition.

Unforeseen, SUDDEN market movements

Sudden, potentially catastrophic market movements such as the GBP Flash Crash of October, 2016, bring with them a tidal wave of abrupt changes, not only in price but also in execution and liquidity dynamics.
Historical data will permit commercial EA vendors to showcase their product’s performance using just price during such times, ignoring other important factors.

For example, if an EA survived the GBP Flash Crash in a backtest, using fixed spread whilst assuming perfect execution conditions, odds of a repeat performance are negligible (if not zero) in case the GBP Flash Crash were to happen again on the first day the EA is deployed live.
More reason to treat “Set & Forget” claims with more than just a pinch of salt.. 😉
—

COMPLETELY different market conditions

Lastly, there linger the following questions:

What if a backtest was just lucky? (walk forward optimized or not).

What if the market conditions observed in the backtest never surface again?
What would this backtest look like if ALL the bad trades happened at the beginning (or the end), or some random positional variation thereof?
“The backtest looks great, but..

What is the true probability of me seeing similar performance in future?

Given that this EA has this backtest, what is a realistic range of future returns I am likely to see?”

Next time you see an EA marketed with “Set & Forget”, ask yourself these questions.
Then ask the EA vendor these questions, demanding statistically sound evidence.
Or just ask them to list their product as a DARWIN, and let the Darwinex Analytical Toolkit score its performance for you.

Conclusion

We hope this post has helped readers appreciate that:

1. “Set & Forget” is more marketing sizzle, less anything else.
2. You always need more than just a backtest.
3. Past performance really and truly isn’t an indicator of future performance.
4. It is futile to ignore live trading dynamics in backtesting.
5. If an EA vendor doesn’t have a DARWIN, ask them to list one.
6. If an EA vendor won’t list a DARWIN, ask them to upload a backtest or track record and confirm investment attribute scores attained.
7. If an EA is only meant for “fixed spread environments”, the backtest and live track record are both of little value.

Lastly, if you find yourself considering an EA purchase somewhere, send the seller an email saying:
“I understand you will make money selling this EA – but why not list a DARWIN instead?”

Do you currently rent or sell your MetaTrader EA?

Consider listing a DARWIN instead and tap into investor capital on the Darwin Exchange! (.. more than $1,000,000 in performance fees paid to date).