Posts

darwin api

How To Download Tick Data in MetaTrader 4 & 5

This post describes how to download and save tick data offline, from both MT4 and MT5.

Functional code samples (MetaTrader indicators) have been provided via GitHub, along with instructions on how to download and use them to begin extracting tick data from MetaTrader 4 or 5.

Each tick data point extracted using this post’s implementation contains:

  1. Timestamp (in seconds or milliseconds)
  2. Bid price
  3. Ask price
  4. Bid/Ask Spread

The remainder of this post is organized as follows:

  1. The case for collecting tick data.
  2. Anatomy of a simple Tick Data indicator.
  3. Differences between MT4 and MT5 versions.
  4. Indicator Installation Instructions.

The case for collecting tick data

Traders can benefit in a number of ways by collecting tick data directly from their broker.

This ensures exact consistency (barring any unrelated issues, technical or otherwise) between tick data stored offline and that available via the MetaTrader platform.

Possible use-cases include:

1) Backtesting trading strategies that require tick-level precision.

2) Sampling price data by synchronizing it with e.g. speed of tick arrival instead of block intervals (e.g. M1, M5, etc).

This practice addresses the negative impact of time based intervals effectively ignoring the frequency of price change at different times of day.


For those interested, Mandelbrot and Taylor [1967] were among the pioneers in establishing that sampling by transaction frequency improves statistical soundness:

“Price changes over a fixed number of transactions may have a Gaussian distribution. Price changes over a fixed time period may follow a stable Paretian distribution, whose variance is infinite. Since the number of transactions in any time period is random, the above statements are not necessarily in disagreement.”

Journal link:
https://www.jstor.org/stable/168611


3) Developing strategies, tools, indicators etc. outside the MetaTrader environment (e.g. in Python, R, Julia, Java, C/C++ etc.)

4) Monitoring the evolution of Bid/Ask spreads both inside and outside MetaTrader.

5) Drawing inferences from time-weighted average bid/ask spreads (TWAS) vs. your strategy’s evolution (both in real-time and otherwise).

6) Strategies targeting small gains can benefit from optimizing execution based on the evolution of underlying asset TWAS data.

7) Connecting a real-time FX tick data feed via ZeroMQ to non-MQL trading strategies or R&D software.

.. and the list goes on.


Anatomy of a simple Tick Data Indicator

The indicators implemented in this post have the following structure:

1) Initialization -> OnInit()

This function is called when the indicator is loaded on a chart in the MetaTrader terminal.

For our purposes, the only activity required here is:

  • Open a new CSV file for writing tick data to, e.g. EURUSD_TickData.csv
  • If the file already exists, append data to it instead of creating a separate file for the same asset.

 

Code listing:

int OnInit()
 {
//--- indicator buffers mapping
 
 // Create CSV file handle in WRITE mode.
 csv_io_hnd = FileOpen(Symbol() + "_TickData.csv", FILE_CSV|FILE_READ|FILE_WRITE|FILE_REWRITE, ',');
 // If creation successful, write CSV header, else throw error
 if(csv_io_hnd > 0)
 {
 if(FileSize(csv_io_hnd) <= 5)
 FileWrite(csv_io_hnd, "time_milliseconds", "bid", "ask", "spread");
 
 // Move to end of file (if it's being written to again)
 FileSeek(csv_io_hnd, 0, SEEK_END);
 }
 else
 Alert("ERROR: Opening/Creating CSV file!");
//---
 return(INIT_SUCCEEDED);
 }

2) Tick Processing -> OnCalculate()

This function is called each time a new tick is received.

Here we’ll need to:

  • Check if a file is open for writing the tick’s bid, ask and spread values to.
  • If yes, write tick to file.
  • If no, throw an error highlighting the failure.

 

Code listing:

int OnCalculate(const int rates_total,
 const int prev_calculated,
 const datetime &time[],
 const double &open[],
 const double &high[],
 const double &low[],
 const double &close[],
 const long &tick_volume[],
 const long &volume[],
 const int &spread[])
 {
//---
 
 // If CSV file handle is open, write time, bid, ask and spread to it.
 if(csv_io_hnd > 0)
 {
 if(SymbolInfoTick(Symbol(), tick_struct))
 {
 Comment("\n[Darwinex Labs] Tick Data | Bid: " + DoubleToString(tick_struct.bid, 5) 
 + " | Ask: " + DoubleToString(tick_struct.ask, 5) 
 + " | Spread: " + StringFormat("%.05f", NormalizeDouble(MathAbs(tick_struct.bid - tick_struct.ask), 5))
 + "\n\n* Writing tick data to \\MQL5\\Files\\" + Symbol() + "_TickData.csv ..."
 + "\n(please remove the indicator from this chart to access CSV under \\MQL4\\Files.)"
 );
 FileWrite(csv_io_hnd, tick_struct.time_msc, tick_struct.bid,
 tick_struct.ask, StringFormat("%.05f", NormalizeDouble(MathAbs(tick_struct.bid - tick_struct.ask), 5)));
 } 
 else
 Print("ERROR: SymbolInfoTick() failed to validate tick."); 
 }
 
//--- return value of prev_calculated for next call
 return(rates_total);
 }

3) Termination -> OnDeinit()

This function is called when the indicator is removed from the chart, the chart is closed with the indicator still deployed on it, or the terminal is shut down correctly for any reason.

All we need to do here is check if the file being written to is still open, and close it.

* Important Note: You must remove the indicator from the chart before attempting to open the CSV file being written to. MetaTrader will have exclusive read/write access to the file while the indicator is running.

 

Code listing:

void OnDeinit(const int reason)
{ 
 // Close CSV file handle if currently open, before exiting.
 if(csv_io_hnd > 0)
 FileClose(csv_io_hnd);
 
 // Clear chart comments
 Comment("");
}

Differences between MT4 and MT5 versions

This implementation uses MetaTrader’s MqlTick structure in both the MetaTrader 4 and 5 versions.

For your reference, the definition for MqlTick is:

struct MqlTick 
  { 
   datetime     time;          // Time of the last prices update 
   double       bid;           // Current Bid price 
   double       ask;           // Current Ask price 
   double       last;          // Price of the last deal (Last) 
   ulong        volume;        // Volume for the current Last price 
   long         time_msc;      // Time of a price last update in milliseconds 
   uint         flags          // Tick flags 
  };

There is one subtle but important difference between its use in MetaTrader 4 and 5:

MqlTick.time_msc will return a timestamp in seconds (in MetaTrader 4) and in milliseconds (in MetaTrader 5)

Therefore, for MetaTrader 4 users in particular, it makes sense to use MetaTrader 5 for collecting tick data as subsecond sampling isn’t available in MetaTrader 4.

Other minor differences include using the comma (‘,’) delimiter in FileOpen() operations in MetaTrader 4, as opposed to tab (‘\t’) in MetaTrader 5.


MetaTrader Indicator Installation Instructions

We’ve uploaded functional MetaTrader 4 and 5 versions of this Indicator to our GitHub page under tools -> MQL4 and MQL5 respectively.

First, simply Right-Click and Save-As on the following link to save the file to your computer, and follow the instructions below them:

For MetaTrader 4 users:
Click here to download DLabs_TickData_ToCSV_MT4.mql4

For MetaTrader 5 users:
Click here to download DLabs_TickData_ToCSV_MT5.mql5

Instructions:

  1. Launch MetaTrader, and open your data folder (File -> Open Data Folder)

    MetaTrader 4 Open Data Folder

    MetaTrader Open Data Folder

  2. Copy the “DLabs_TickData_ToCSV_MTx.mqx into your MQLx\Indicators directory, where “x” is either 4 or 5 depending on your platform (MT4 or MT5).
  3. Restart MetaTrader and open a new chart,
  4. Double-click on DLabs_TickData_ToCSV_MTx (as above) under Indicators to load the indicator on any chart.
  5. No input parameters need configuration, therefore click OK and continue.
  6. Open a chart, the symbol for which you’d like tick data saved to disk, e.g. EURUSD.
  7. Tick data including timestamp, bid, ask and spread should now begin saving to a CSV file in the background.

If you want to open the CSV file, you must first remove the indicator from the chart.

To do this, Right-Click anywhere on your chart and select Indicator List:

MetaTrader Indicator List

MetaTrader Indicator List

 

Then select the indicator from the displayed list, and hit Delete to remove it from your chart:

Delete Indicator from MetaTrader chart

Delete Indicator from MetaTrader chart

 

To access the CSV file, click File -> Open Data Folder, and once inside, head over to the MQLx\Files (x described as above) directory and open the CSV file that will have been saved here.

Saved MetaTrader Tick Data

Saved MetaTrader Tick Data

 

The next post in this series discusses how to automate tick data collection using R, MetaTrader and a VPS instance.


As always, we hope you’ve enjoyed this tutorial, and look forward to any feedback you may have for us 🙂

Code samples have been kept as simple as possible to enable the programmer in you to thrive in extensibility, and exercise your creative freedom as best as you can!

Should you have ideas/feedback on how we can extend this implementation further, please do feel free to leave a comment below – we’ll try our best to either release an update, or resolve your query directly.

Also, kindly share this post using the buttons provided, with any colleagues and/or networks you feel would benefit from the content. Or just share it anyway to help us spread the word! 🙂


You may also wish to read:

  1. ZeroMQ – How To Interface Python/R with MetaTrader 4
  2. DO’s and DONT’s of MT4 Backtesting
  3. How To Identify Overfit Trading Strategies
  4. Currency Index Indicator for MetaTrader 4
  5. Constructing a Currency Portfolio in MetaTrader
Currency Index - MetaTrader 4 Indicator

Currency Index Indicator for MetaTrader 4

In this post, we describe the purpose, composition, parity, calculation logic and construction of a Currency Index for each of the 8 major currencies.

These include:

  1. EUR – Euro
  2. USD – US Dollar
  3. GBP – British Pound
  4. JPY – Japanese Yen
  5. AUD – Australian Dollar
  6. NZD – New Zealand Dollar
  7. CHF – Swiss Franc
  8. CAD – Canadian Dollar

MQL4 source code is also provided to enable traders to load and use the indicator on the MetaTrader 4 platform.


What is a Currency Index?

A currency index measures the evolution or strength in one major currency relative to a basket of the other major currencies.

With a single currency pair, e.g. GBP/JPY, an uptrend would symbolize a strengthening of the British Pound vs the Japanese Yen, information based on only those two currencies, and vice versa.

Therefore, by monitoring just the GBP/JPY, a trader knows how the GBP is behaving vs the JPY, but not how the GBP and JPY currencies are behaving independently.

Currency Indexes (MetaTrader 4 Indicator)

Currency Indexes (MetaTrader 4 Indicator)

With a currency index however, an uptrend in the British Pound (GBP) would indicate a strengthening of the British Pound relative to the entire basket of seven other major currencies.

i.e. EUR, USD, JPY, CAD, AUD, NZD, and CHF.

Similarly, in the case of a downtrend, it would indicate a weakening of the GBP relative to the entire basket of seven other major currencies.


Basket Composition & Parity

Each currency index is calculated relative to a basket of currency pairs, chosen such that each pair:

  1. Contains the concerned currency in either the Base or Quote component,
  2. The pairs combined include all 8 major currencies,
  3. Where the concerned currency is the Quote component of a pair, the parity of the pair needs to be reversed. e.g. when calculating the GBP currency index, EUR/GBP needs a parity of -1, making it -EUR/GBP to adjust for relative GBP strength.

 

For example, the basket of currency pairs to calculate the EUR currency index is:

  1. EUR/USD,
  2. EUR/GBP,
  3. EUR/AUD,
  4. EUR/NZD,
  5. EUR/JPY,
  6. EUR/CHF,
  7. EUR/CAD

It contains the concerned currency (EUR), and the other major currencies, i.e. USD, GBP, JPY, AUD, NZD, CHF, CAD. As each currency pair has EUR as the Base component, the parity of each pair in this basket is +1.

 

The remaining currency indexes therefore, have the following composition and parities:

USD: -EUR/USD, -GBP/USD, -AUD/USD, -NZD/USD, USD/JPY, USD/CHF, USD/CAD,

GBP: -EUR/GBP, GBP/USD, GBP/AUD, GBP/NZD, GBP/JPY, GBP/CHF, GBP/CAD,

JPY: -EUR/JPY, -USD/JPY, -AUD/JPY, -NZD/JPY, -GBP/JPY, -CHF/JPY, -CAD/JPY,

AUD: -EUR/AUD, AUD/USD, AUD/JPY, AUD/NZD, -GBP/AUD, AUD/CHF, AUD/CAD,

NZD: -EUR/NZD, NZD/USD, -AUD/NZD, NZD/JPY, -GBP/NZD, NZD/CHF, NZD/CAD,

CHF: -EUR/CHF, -NZD/CHF, -AUD/CHF, CHF/JPY, -GBP/CHF, -USD/CHF, -CAD/CHF,

CAD: -EUR/CAD, -USD/CAD, -AUD/CAD, -NZD/CAD, -GBP/CAD, CAD/CHF, CAD/JPY

Unlike the common standard of using a weighted portfolio of currencies in baskets to reflect each currency’s relative importance by economic factors, this implementation uses an equally weighted portfolio to remove that bias.


Calculation Logic & Construction

Each currency index \(I_x\), at time t, where rates_total time periods are available for processing, can be mathematically represented as follows:

Currency Index Calculation / Formula

Currency Index Calculation / Formula

 

The contribution of each currency pair in an index’s basket is calculated as above, indexed to base 100.00 from the chosen start date.

The MQL4 indicator source code provided allows the user to set a start date for calculating the first point in the currency index therefore starting at 100.00. The default start date is January 01, 2015.


MQL4 Indicator (Source Code)

We have uploaded a functional MetaTrader 4 Indicator to our GitHub page under tools -> MQL4.

First, simply Right-Click and Save-As on the following links to save the files to your computer, and follow the instructions below them:

Click here to download DLabs_CurrencyIndex.mqh

Click here to download DLabs_CurrencyIndex.mq4

Instructions:

  1. Launch MetaTrader 4, and open your data folder (File -> Open Data Folder)

    MetaTrader 4 Open Data Folder

    MetaTrader 4 Open Data Folder

  2. Copy the “DLabs_CurrencyIndex.mqh” into your MQL4\Include directory,
  3. Copy the “DLabs_CurrencyIndex.mq4” into your MQL4\Indicators directory,
  4. Restart MetaTrader 4 and open a new chart,
  5. Double-click on DLabs_CurrencyIndex under Indicators to load the indicator on any chart.
Currency Index - MetaTrader 4 Indicator (Inputs)

Currency Index – MetaTrader 4 Indicator (Inputs)

The indicator takes two inputs:

  1. Currency Index to draw (e.g. EUR, GBP, etc)
  2. Start date to base the index calculation.

Click OK, and the indicator will draw the selected currency index in its own dedicated window, as shown below:

Currency Index MetaTrader 4 Indicator

Currency Index MetaTrader 4 Indicator

Important Note: In order for the indicator to draw correctly and accurately, users must ensure that their History Center contains data for all currency pairs in an index’s basket.


[Additional Resources] Constructing Currency Portfolio Indexes in MetaTrader 4

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

MetaTrader 4 Backtesting

MetaTrader Backtesting – Best Practices for Algorithmic Traders

Backtesting: Historical vs Live Performance

Backtesting: Historical vs Live Performance

MetaTrader backtesting can be tricky business for algorithmic traders. Follow these best practices to engineer robust, reliable trading strategies.

Watch the full video tutorial here (36 minutes).

Simulating a strategy’s historical performance correctly, increases the probability of it generalizing well to unseen market data in future.

As such, it’s important that all backtesting be conducted robustly, careful attention being paid to any factors that could contribute towards historical and live performance being markedly different.

This post organizes proposed best practices into 3 main categories:

  1. Data Handling
  2. Parameter Selection
  3. Variable Factors

1) Data Handling

In this first section, we discuss how to treat data used in MetaTrader 4 backtesting.

Segmenting Historical Data

A common misconception in MetaTrader 4 backtesting is to associate a strategy’s robustness with how well it performs on the largest possible amount of historical data available to the trader.

A strategy that performs remarkably well on the entire dataset is at risk of being overfit (high variance).

Conversely, a strategy that has been backtested on too small a portion of historical data is likely to encounter high bias.

In both cases, the likelihood of the trading strategy generalizing well to unseen market data is poor at best.

Therefore, it is important at the very least, that any backtesting be conducted using independent training, validation and test sets:

  1. Training: The data segment used for training the strategy parameters on (or in other words, checking for decent performance in backtesting).
  2. Validation: The first unseen data segment used to test how the same parameters chosen in training, perform on unseen data. Passing this phase adds confidence that the strategy is robust. Discrepancies in this phase also permit revision of parameters before finally testing on the last batch of test data.
  3. Testing: If a strategy passes both training and validation phases satisfactorily, running a final simulation on this last “held out” segment of data adds another layer of confidence that the trading strategy either exhibits robustness or is prone to excess stagnation or failure in live trading.
Training, Validation, Testing Samples

Training, Validation, Testing Samples

Care should be taken however to not bias the outcome by modifying any parameters during the test sample phase.

If the trading strategy passed in-sample (training), validation, but not testing, it is advisable to go back to the drawing board for any modifications required, and then re-run both in-sample and validation tests again.


Selecting Length of Historical Data

The amount of historical data required for a backtesting exercise is directly proportional to the complexity of the trading strategy being tested.

In other words, the more complex (large number of training parameters) the strategy is, the more data required to ascertain the validity of its underlying hypothesis.

Occam's Razor - Simple vs. Complex Trading Strategy

Occam’s Razor – Simple vs. Complex Trading Strategy

In line with the Occam’s Razor principle, when optimizing parameters to estimate the amount of data required for a backtest, it is generally best to favour simpler over more complex strategies if their output performance is not dramatically different (for better or worse).

 

If the amount of data available is not as large as is optimal for backtesting, traders may consider a cross-validation approach and/or walk forward optimization to make better use of smaller data samples.

Bootstrapping is another possibility, but care should be taken with sampling time series data that exhibit variable momentum over short periods of time, as it becomes considerably difficult to preserve dependency structure when bootstrapping such data.


Backtesting Timeframes Lower Than H1 (hourly)

Trading strategies that operate below hourly (H1) timeframes, are prone to experiencing excess divergence between their simulated and live trading performance.

Monte Carlo Simulation / Optimization of Trading Strategies

Monte Carlo Simulation / Optimization of Trading Strategies

This behaviour is primarily due to a considerable amount of noisy data in timeframes below hourly.

If a strategy models on data over small timeframes, traders must be mindful of overfitting risk (particularly in complex strategies with a large number of parameters).

Robustness testing such as Monte Carlo simulations over robust variations in strategy parameters, can quickly estimate how closely (or not) a strategy could perform in live trading vs. backtesting.


MetaTrader 4 Historical Data – Effects of Interpolation

Data on timeframes lower than H1 (hourly) in MetaTrader 4 is progressively affected by one-minute interpolation as the timeframe decreases in size, e.g. H1 -> M30 -> M15 -> M5 -> M1.

This has implications for trading strategies that are backtested on small precisions of data, with the probability of live trading results differing considerably from backtests increasing with every decrease in timeframe.

Traders should therefore be extra cautious when testing strategies on low timeframes, choosing robust parameter ranges and always comparing a strategy’s performance on lower timeframes with higher timeframes and multiple datasets.

If differences between timeframes and/or multiple datasets are excessively large, e.g. a remarkably profitable strategy on M15 turning in completely the opposite direction on higher timeframes or time series data from difference sources, the probability of the strategy being overfit to one dataset and/or finely interpolated data (in the case of lower timeframes) is usually fairly high.


Re-validating MetaTrader 4 History Center Data

Data in MT4 can at times experience corruption during platform usage. This can happen as a result of discrepancies in tick data transfer, opening charts for assets where history wasn’t already available, to name a few reasons.

Therefore, when backtesting in MetaTrader 4’s Strategy Tester whilst connected to your brokerage account, it makes sense to refresh asset data in the History Center again prior to executing new backtests.

This ensures that it is up to date, and any erroneous ticks or gaps as a result of market opens, interpolation discrepancies or high impact news events are validated prior to backtesting.


2) Parameter Selection

This section discusses some best practices in terms of risk management and strategy optimization.

Using Robust Parameter Ranges

Regardless of the number of parameters in a trading strategy, it’s important that traders choose parameter values that do not fit the underlying data too closely.

For example, it is a more robust practice to use an Exponential Moving Average (EMA) period of 50 instead of 49 or 47, if the resulting backtest performances are slightly different but both positive in both cases.

Similarly, during any genetic or walk-forward optimization, it is good practice to choose robust ranges of parameter values that typically increment in steps of 5 or 10, as opposed to 1 or 2.

Robust range selection in this manner does carry the risk of introducing high bias into the algorithm. Therefore, it is also important for traders to test how using robust parameter values impact the overall performance or ability of the strategy to generalize well to unseen data.


Estimating Impact of Variable Spread & Slippage

Measuring Variable Spread / Slippage Is Important

Measuring Variable Spread / Slippage Is Important

At the time of writing (October 16, 2017) MetaTrader 4’s Strategy Tester does not yet ship with the ability to backtest with variable slippage simulation.

Traders can however run slippage simulations on their backtest results in a a few ways, three of which are:

  1. By importing backtest results into a spreadsheet application (e.g. MS Excel), modifying Open/Close prices of trades with randomly generated values between a sensible range (e.g. between 0.1 and 2.0 or 5.0) and recalculating P/L.
  2. By importing backtest results into a statistical computing environment such as R, GNU Octave or MATLAB and doing the same as in point (1) above.
  3. Using 3rd-party commercial tools.

The same practice can be employed with variable spread simulation, as MetaTrader 4 currently ships with fixed spread testing available in the Strategy Tester.


Maintaining Stable Underlying Strategy VaR

Making use of dynamic stop loss and take profit targets, proportionate to available account equity, allows for realistic backtest outcomes, and aids risk stability (especially if the trader intends on creating a DARWIN).

Dynamic Position Sizing / Liquidity Considerations

Dynamic Position Sizing / Liquidity Considerations

In trading strategies that increase or decrease position sizes by available equity, it is particularly important that minimum/maximum lot sizes be restricted to sensible ranges for the following reasons:

  1. A lot size dropping below the minimum permissible lot size would result in trade rejection.
  2. Lot sizes scaling too large in single trades (e.g. to 100 lots or above) would introduce capacity constraints during live trading that could possibly require fractional position sizing to address, depending on factors such as time of day, strategy capacity and available liquidity.
  3. Such constraints would not be visible in a MetaTrader backtest, and without being factored in during strategy development, possibly create the illusion of an extremely profitable strategy with no liquidity considerations holding it back.

3) Variable Factors

In this final section, we discuss some considerations that trading strategies can benefit from during the design and post-backtest phases.

Correlation of Strategy Returns to Market Volatility

For a detailed discussion on this, please watch the webinar recording on Effects of Market Volatility on Trader Performance here.


Testing for Market Correlation

Comparing the correlation of a trading strategy’s returns to those of its underlying assets, reveals the dependency of the strategy on the same.

A strategy with a low dependency on the assets it trades is less likely to demonstrate unexpected behaviours during periods of decline or sudden turbulence in underlying assets, as opposed to one that relies heavily on underlying asset returns.

To analyze a strategy for Market Correlation as well as 11 other investment attributes, traders are encouraged to upload backtests to the Darwinex platform, review scores and visualize the evolution of their strategies for each attribute.


Optimizing Position Sizes for Capacity

One of the ways in which a strategy’s overall investment capacity can be improved, is by implementing fractional order sizing as opposed to fixed, singular lot sizing.

For example, sending 10 orders of 1 lot each is a more scalable approach than sending 1 order of 10 lots.

For a detailed discussion on scalability, please watch our webinar recording on Scalability  (now titled Capacity on the DARWIN Exchange)


Trading High Impact News

Strategies that trade during high impact news events are impacted negatively by sudden swings in variable slippage and spreads at the time of the event.

One way to reduce (if not eliminate) such negative impact is by having the strategy disable trading at pre-defined times or intervals corresponding to such events.


[Webinar Recording] DO’s and DONT’s of MT4 Backtesting

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

ZeroMQ - Distributed Trading Infrastructure

ZeroMQ – How To Interface Python/R with MetaTrader 4

Zero MQ - Distributed Messaging

ZeroMQ – Distributed Messaging

In this post, we present a technique employing ZeroMQ (an Open Source, Asynchronous Messaging Library and Concurrency Framework) for building a basic – but easily extensible – high performance bridge between external (non-MQL) programming languages and MetaTrader 4.

 

Reasons for writing this post:

  1. Lack of comprehensive, publicly available literature about this topic on the web.
  2. Traders have traditionally relied on Winsock/WinAPI based solutions that often require revision with both Microsoft™ and MetaQuotes™ updates.
  3. Alternatives to ZeroMQ include named pipes, and approaches where filesystem-dependent functionality forms the bridge between MetaTrader and external languages.

 

Click below to watch the video tutorials:

1) How to Interface Python Trading Strategies with MetaTrader

2) Algorithmic Trading via ZeroMQ: Trade Execution, Reporting & Management

3) Algorithmic Trading via ZeroMQ: Subscribing to Market Data

4) Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 1

5) Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 2


In this blog post, we lay the foundation for a distributed trading system that will:

  1. Consist of one or more trading strategies developed outside MetaTrader 4 (non-MQL),
  2. Use MetaTrader 4 for acquiring market data, trade execution and management,
  3. Support multiple non-MQL strategies interfacing with MetaTrader 4 simultaneously,
  4. Consider each trading strategy as an independent “Client”,
  5. Consider MetaTrader 4 as the “Server”, and medium to market,
  6. Permit both Server and Clients to communicate with each other on-demand.

 

Infographic: ZeroMQ-Enabled Distributed Trading Infrastructure (with MetaTrader 4)

Infographic: ZeroMQ-Enabled Distributed Trading Infrastructure (with MetaTrader 4)

Why ZeroMQ?

  1. Enables programmers to connect any code to any other code, in a number of ways.
  2. Eliminates a MetaTrader user’s dependency on just MetaTrader-supported technology (features, indicators, language constructs, libraries, etc.)
  3. Traders can develop indicators and strategies in C/C#/C++, Python, R and Java (to name a few), and deploy to market via MetaTrader 4.
  4. Leverage machine learning toolkits in Python and R for complex data analysis and strategy development, while interfacing with MetaTrader 4 for trade execution and management.
  5. ZeroMQ can be used as a high-performance transport layer in sophisticated, distributed trading systems otherwise difficult to implement in MQL.
  6. Different strategy components can be built in different languages if required, and seamlessly talk to each other over TCP, in-process, inter-process or multicast protocols.
  7. Multiple communication patterns and disconnected operation.

ZeroMQ: Supported Programming Languages

Though we focus on MQL interfaced with Python & R in this post, the basic process described here can be implemented easily in other ZeroMQ-supported languages.

A comprehensive list of ZeroMQ language bindings is available here:

Zero MQ Language Bindings


Who else is using ZeroMQ?

AT&T, Cisco, EA, Los Alamos Labs, NASA, Weta Digital, Zynga, Spotify, Samsung Electronics, Microsoft, CERN and Darwinex Labs.

ZeroMQ also powers at least 5 DARWINS on The DARWIN Exchange, where the underlying trading strategies were written in C++, Python and R.


Planning Flow Control

This post is not intended to be a detailed tutorial on ZeroMQ.

However, it is still important to understand a few things about ZeroMQ that make it particularly suited to the task of connecting external programming languages such as Python and R to MetaTrader 4.

  • It supports TCP, inter-process, in-process, PGM and EPGM enabled multicast networking. We will use the TCP transport type for the implementation in this post.
  • ZeroMQ enables servers and clients to connect “to each other” on demand, particularly useful for designing distributed trading infrastructure.
  • In addition to support for asynchronous communication and disconnected operation, ZeroMQ supports several communication patterns that permit higher-level data transfer, freeing programmers to focus more on the transfer logic rather than low-level mechanisms.
  • These patterns include: Request (REQ) / Reply (REP), Publish (PUB) / Subscribe (SUB) and Push (PUSH) / Pull (PULL).

 

For the implementation in this blog post, we will employ ZeroMQ’s REQ/REP and PUSH/PULL communication patterns. MetaTrader 4 will be our “Server”, and trading strategies will be “Clients”.

Please note that this (MT4=Server, Strategy=Client) is not a MUST – you will need to decide on whatever flow control suits your particular needs best.

For example, you might designate a machine independent of both the trading strategy as well as MetaTrader 4, as your Server, and have Strategies and MT4 both be Clients. There are a number of ways you could achieve the end goal; carefully planning flow control will lead to efficient functionality.

 

Request (REQ) / Reply (REP) Pattern

The Server (MetaTrader 4 EA) will employ a TCP socket of type REP, to receive requests and send responses. A REP socket MUST always initiate a pair of calls: first, a receive, followed by a send.

The Client (Trading Strategy, e.g. in Python) will employ a TCP socket of type REQ, to send requests and receive responses. A REQ socket MUST always initiate a pair of calls too: first, a send, followed by a receive.

For this implementation, the REQ/REP pattern will enable our Clients to send commands to the MetaTrader 4 Server and receive acknowledgements of the same (e.g. OPEN/MODIFY/CLOSE trades, GET BID/ASK RATES, GET HISTORICAL PRICES, etc.)

 

Push (PUSH) / Pull (PULL) Pattern

The Server (MetaTrader 4 EA) will also employ a second, PUSH socket, to send additional information to Clients (Trading Strategies). This is a one-way socket, and the server will only be able to send data to this socket, without being able to receive anything back through the same socket.

The Client (Trading Strategy) will also employ a second, PULL socket, to receive additional information from the Server. This too is a one-way socket, and the client will only be able to receive data from this socket, without being able to send anything through the same socket.

The PUSH/PULL pattern enables servers and clients to exchange data with each other on-demand, but in one direction without expecting a response. This could of course be swapped out for another REQ/REP pattern, depending on your application’s flow control requirements.

 

In summary, for this post’s basic implementation:

  1. The Server will employ two sockets, one REP and one PUSH.
  2. Each Client will employ two sockets, one REQ and one PULL.

 

Infographic: What this flow control plan looks like in practice.

Infographic: ZeroMQ Process Flow Control

 


MetaTrader 4 Expert Advisor – Components

As displayed in the infographic above, the MT4 EA will serve as our ZeroMQ-enabled Server, with three main modules:

  1. MESSAGE ROUTER – This allows the EA to receive commands and send acknowledgements back to connecting Clients (trading strategies) through the REP socket. The Router passes all messages on to the Parser. Note: For this example, the Router doesn’t serve much purpose, but it is good practice to have this intermediary where several strategies connect to the Server (MT4) and some manner of pre-parse actions may need to be performed.
  2. MESSAGE PARSER – Messages received by this module are decomposed into actions for the next module (Interpreter & Executor).
  3. INTERPRETER & EXECUTOR – This module literally “interprets” decomposed messages and performs requested actions accordingly. For example, if the Client is requesting market data, the module gathers it from the MetaTrader 4 History DB and sends it on to the Client via the PUSH socket. Alternatively, if the Client is requesting a BUY or SELL trade be opened on e.g. the EUR/USD, it sends the trade to market and a notification of success/failure/ticket-info to the Client via the PUSH socket.

Implementation Requirements

  1. ZeroMQ – MQL4 Bindings -> Download and install the required files as instructed here: https://github.com/dingmaotu/mql-zmq
  2. For Python -> “pyzmq” library
  3. For R -> “rzmq” library

Sample Code

To give you a head start, we’ve published a functional MetaTrader 4 Expert Advisor with the full implementation discussed in this blog post.

The MQL sample code provided is quite extensible, and can be used as a template in your efforts.

GitHub Links:

  1. DWX ZeroMQ Connector – Python & MQL

Notes:

  1. The Python source code demonstrates how communication patterns are implemented.
  2. It’s fairly simple to integrate this code in your existing Python/R trading strategies.

 


[WEBINAR REPLAY] How to Interface Python/R Trading Strategies with MetaTrader 4


[VIDEO TUTORIAL] Algorithmic Trading via ZeroMQ: Trade Execution, Reporting & Management (Python to MetaTrader)


[VIDEO TUTORIAL] Algorithmic Trading via ZeroMQ: Subscribing to Market Data (Python to MetaTrader)


[VIDEO TUTORIAL] Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 1


[VIDEO TUTORIAL] Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 2


Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange