Posts

darwin api

How To Download Tick Data in MetaTrader 4 & 5

This post describes how to download and save tick data offline, from both MT4 and MT5.

Functional code samples (MetaTrader indicators) have been provided via GitHub, along with instructions on how to download and use them to begin extracting tick data from MetaTrader 4 or 5.

Each tick data point extracted using this post’s implementation contains:

  1. Timestamp (in seconds or milliseconds)
  2. Bid price
  3. Ask price
  4. Bid/Ask Spread

The remainder of this post is organized as follows:

  1. The case for collecting tick data.
  2. Anatomy of a simple Tick Data indicator.
  3. Differences between MT4 and MT5 versions.
  4. Indicator Installation Instructions.

The case for collecting tick data

Traders can benefit in a number of ways by collecting tick data directly from their broker.

This ensures exact consistency (barring any unrelated issues, technical or otherwise) between tick data stored offline and that available via the MetaTrader platform.

Possible use-cases include:

1) Backtesting trading strategies that require tick-level precision.

2) Sampling price data by synchronizing it with e.g. speed of tick arrival instead of block intervals (e.g. M1, M5, etc).

This practice addresses the negative impact of time based intervals effectively ignoring the frequency of price change at different times of day.


For those interested, Mandelbrot and Taylor [1967] were among the pioneers in establishing that sampling by transaction frequency improves statistical soundness:

“Price changes over a fixed number of transactions may have a Gaussian distribution. Price changes over a fixed time period may follow a stable Paretian distribution, whose variance is infinite. Since the number of transactions in any time period is random, the above statements are not necessarily in disagreement.”

Journal link:
https://www.jstor.org/stable/168611


3) Developing strategies, tools, indicators etc. outside the MetaTrader environment (e.g. in Python, R, Julia, Java, C/C++ etc.)

4) Monitoring the evolution of Bid/Ask spreads both inside and outside MetaTrader.

5) Drawing inferences from time-weighted average bid/ask spreads (TWAS) vs. your strategy’s evolution (both in real-time and otherwise).

6) Strategies targeting small gains can benefit from optimizing execution based on the evolution of underlying asset TWAS data.

7) Connecting a real-time FX tick data feed via ZeroMQ to non-MQL trading strategies or R&D software.

.. and the list goes on.


Anatomy of a simple Tick Data Indicator

The indicators implemented in this post have the following structure:

1) Initialization -> OnInit()

This function is called when the indicator is loaded on a chart in the MetaTrader terminal.

For our purposes, the only activity required here is:

  • Open a new CSV file for writing tick data to, e.g. EURUSD_TickData.csv
  • If the file already exists, append data to it instead of creating a separate file for the same asset.

 

Code listing:

int OnInit()
 {
//--- indicator buffers mapping
 
 // Create CSV file handle in WRITE mode.
 csv_io_hnd = FileOpen(Symbol() + "_TickData.csv", FILE_CSV|FILE_READ|FILE_WRITE|FILE_REWRITE, ',');
 // If creation successful, write CSV header, else throw error
 if(csv_io_hnd > 0)
 {
 if(FileSize(csv_io_hnd) <= 5)
 FileWrite(csv_io_hnd, "time_milliseconds", "bid", "ask", "spread");
 
 // Move to end of file (if it's being written to again)
 FileSeek(csv_io_hnd, 0, SEEK_END);
 }
 else
 Alert("ERROR: Opening/Creating CSV file!");
//---
 return(INIT_SUCCEEDED);
 }

2) Tick Processing -> OnCalculate()

This function is called each time a new tick is received.

Here we’ll need to:

  • Check if a file is open for writing the tick’s bid, ask and spread values to.
  • If yes, write tick to file.
  • If no, throw an error highlighting the failure.

 

Code listing:

int OnCalculate(const int rates_total,
 const int prev_calculated,
 const datetime &time[],
 const double &open[],
 const double &high[],
 const double &low[],
 const double &close[],
 const long &tick_volume[],
 const long &volume[],
 const int &spread[])
 {
//---
 
 // If CSV file handle is open, write time, bid, ask and spread to it.
 if(csv_io_hnd > 0)
 {
 if(SymbolInfoTick(Symbol(), tick_struct))
 {
 Comment("\n[Darwinex Labs] Tick Data | Bid: " + DoubleToString(tick_struct.bid, 5) 
 + " | Ask: " + DoubleToString(tick_struct.ask, 5) 
 + " | Spread: " + StringFormat("%.05f", NormalizeDouble(MathAbs(tick_struct.bid - tick_struct.ask), 5))
 + "\n\n* Writing tick data to \\MQL5\\Files\\" + Symbol() + "_TickData.csv ..."
 + "\n(please remove the indicator from this chart to access CSV under \\MQL4\\Files.)"
 );
 FileWrite(csv_io_hnd, tick_struct.time_msc, tick_struct.bid,
 tick_struct.ask, StringFormat("%.05f", NormalizeDouble(MathAbs(tick_struct.bid - tick_struct.ask), 5)));
 } 
 else
 Print("ERROR: SymbolInfoTick() failed to validate tick."); 
 }
 
//--- return value of prev_calculated for next call
 return(rates_total);
 }

3) Termination -> OnDeinit()

This function is called when the indicator is removed from the chart, the chart is closed with the indicator still deployed on it, or the terminal is shut down correctly for any reason.

All we need to do here is check if the file being written to is still open, and close it.

* Important Note: You must remove the indicator from the chart before attempting to open the CSV file being written to. MetaTrader will have exclusive read/write access to the file while the indicator is running.

 

Code listing:

void OnDeinit(const int reason)
{ 
 // Close CSV file handle if currently open, before exiting.
 if(csv_io_hnd > 0)
 FileClose(csv_io_hnd);
 
 // Clear chart comments
 Comment("");
}

Differences between MT4 and MT5 versions

This implementation uses MetaTrader’s MqlTick structure in both the MetaTrader 4 and 5 versions.

For your reference, the definition for MqlTick is:

struct MqlTick 
  { 
   datetime     time;          // Time of the last prices update 
   double       bid;           // Current Bid price 
   double       ask;           // Current Ask price 
   double       last;          // Price of the last deal (Last) 
   ulong        volume;        // Volume for the current Last price 
   long         time_msc;      // Time of a price last update in milliseconds 
   uint         flags          // Tick flags 
  };

There is one subtle but important difference between its use in MetaTrader 4 and 5:

MqlTick.time_msc will return a timestamp in seconds (in MetaTrader 4) and in milliseconds (in MetaTrader 5)

Therefore, for MetaTrader 4 users in particular, it makes sense to use MetaTrader 5 for collecting tick data as subsecond sampling isn’t available in MetaTrader 4.

Other minor differences include using the comma (‘,’) delimiter in FileOpen() operations in MetaTrader 4, as opposed to tab (‘\t’) in MetaTrader 5.


MetaTrader Indicator Installation Instructions

We’ve uploaded functional MetaTrader 4 and 5 versions of this Indicator to our GitHub page under tools -> MQL4 and MQL5 respectively.

First, simply Right-Click and Save-As on the following link to save the file to your computer, and follow the instructions below them:

For MetaTrader 4 users:
Click here to download DLabs_TickData_ToCSV_MT4.mql4

For MetaTrader 5 users:
Click here to download DLabs_TickData_ToCSV_MT5.mql5

Instructions:

  1. Launch MetaTrader, and open your data folder (File -> Open Data Folder)

    MetaTrader 4 Open Data Folder

    MetaTrader Open Data Folder

  2. Copy the “DLabs_TickData_ToCSV_MTx.mqx into your MQLx\Indicators directory, where “x” is either 4 or 5 depending on your platform (MT4 or MT5).
  3. Restart MetaTrader and open a new chart,
  4. Double-click on DLabs_TickData_ToCSV_MTx (as above) under Indicators to load the indicator on any chart.
  5. No input parameters need configuration, therefore click OK and continue.
  6. Open a chart, the symbol for which you’d like tick data saved to disk, e.g. EURUSD.
  7. Tick data including timestamp, bid, ask and spread should now begin saving to a CSV file in the background.

If you want to open the CSV file, you must first remove the indicator from the chart.

To do this, Right-Click anywhere on your chart and select Indicator List:

MetaTrader Indicator List

MetaTrader Indicator List

 

Then select the indicator from the displayed list, and hit Delete to remove it from your chart:

Delete Indicator from MetaTrader chart

Delete Indicator from MetaTrader chart

 

To access the CSV file, click File -> Open Data Folder, and once inside, head over to the MQLx\Files (x described as above) directory and open the CSV file that will have been saved here.

Saved MetaTrader Tick Data

Saved MetaTrader Tick Data

 

The next post in this series discusses how to automate tick data collection using R, MetaTrader and a VPS instance.


As always, we hope you’ve enjoyed this tutorial, and look forward to any feedback you may have for us 🙂

Code samples have been kept as simple as possible to enable the programmer in you to thrive in extensibility, and exercise your creative freedom as best as you can!

Should you have ideas/feedback on how we can extend this implementation further, please do feel free to leave a comment below – we’ll try our best to either release an update, or resolve your query directly.

Also, kindly share this post using the buttons provided, with any colleagues and/or networks you feel would benefit from the content. Or just share it anyway to help us spread the word! 🙂


You may also wish to read:

  1. ZeroMQ – How To Interface Python/R with MetaTrader 4
  2. DO’s and DONT’s of MT4 Backtesting
  3. How To Identify Overfit Trading Strategies
  4. Currency Index Indicator for MetaTrader 4
  5. Constructing a Currency Portfolio in MetaTrader
Currency Index - MetaTrader 4 Indicator

Currency Index Indicator for MetaTrader 4

In this post, we describe the purpose, composition, parity, calculation logic and construction of a Currency Index for each of the 8 major currencies.

These include:

  1. EUR – Euro
  2. USD – US Dollar
  3. GBP – British Pound
  4. JPY – Japanese Yen
  5. AUD – Australian Dollar
  6. NZD – New Zealand Dollar
  7. CHF – Swiss Franc
  8. CAD – Canadian Dollar

MQL4 source code is also provided to enable traders to load and use the indicator on the MetaTrader 4 platform.


What is a Currency Index?

A currency index measures the evolution or strength in one major currency relative to a basket of the other major currencies.

With a single currency pair, e.g. GBP/JPY, an uptrend would symbolize a strengthening of the British Pound vs the Japanese Yen, information based on only those two currencies, and vice versa.

Therefore, by monitoring just the GBP/JPY, a trader knows how the GBP is behaving vs the JPY, but not how the GBP and JPY currencies are behaving independently.

Currency Indexes (MetaTrader 4 Indicator)

Currency Indexes (MetaTrader 4 Indicator)

With a currency index however, an uptrend in the British Pound (GBP) would indicate a strengthening of the British Pound relative to the entire basket of seven other major currencies.

i.e. EUR, USD, JPY, CAD, AUD, NZD, and CHF.

Similarly, in the case of a downtrend, it would indicate a weakening of the GBP relative to the entire basket of seven other major currencies.


Basket Composition & Parity

Each currency index is calculated relative to a basket of currency pairs, chosen such that each pair:

  1. Contains the concerned currency in either the Base or Quote component,
  2. The pairs combined include all 8 major currencies,
  3. Where the concerned currency is the Quote component of a pair, the parity of the pair needs to be reversed. e.g. when calculating the GBP currency index, EUR/GBP needs a parity of -1, making it -EUR/GBP to adjust for relative GBP strength.

 

For example, the basket of currency pairs to calculate the EUR currency index is:

  1. EUR/USD,
  2. EUR/GBP,
  3. EUR/AUD,
  4. EUR/NZD,
  5. EUR/JPY,
  6. EUR/CHF,
  7. EUR/CAD

It contains the concerned currency (EUR), and the other major currencies, i.e. USD, GBP, JPY, AUD, NZD, CHF, CAD. As each currency pair has EUR as the Base component, the parity of each pair in this basket is +1.

 

The remaining currency indexes therefore, have the following composition and parities:

USD: -EUR/USD, -GBP/USD, -AUD/USD, -NZD/USD, USD/JPY, USD/CHF, USD/CAD,

GBP: -EUR/GBP, GBP/USD, GBP/AUD, GBP/NZD, GBP/JPY, GBP/CHF, GBP/CAD,

JPY: -EUR/JPY, -USD/JPY, -AUD/JPY, -NZD/JPY, -GBP/JPY, -CHF/JPY, -CAD/JPY,

AUD: -EUR/AUD, AUD/USD, AUD/JPY, AUD/NZD, -GBP/AUD, AUD/CHF, AUD/CAD,

NZD: -EUR/NZD, NZD/USD, -AUD/NZD, NZD/JPY, -GBP/NZD, NZD/CHF, NZD/CAD,

CHF: -EUR/CHF, -NZD/CHF, -AUD/CHF, CHF/JPY, -GBP/CHF, -USD/CHF, -CAD/CHF,

CAD: -EUR/CAD, -USD/CAD, -AUD/CAD, -NZD/CAD, -GBP/CAD, CAD/CHF, CAD/JPY

Unlike the common standard of using a weighted portfolio of currencies in baskets to reflect each currency’s relative importance by economic factors, this implementation uses an equally weighted portfolio to remove that bias.


Calculation Logic & Construction

Each currency index \(I_x\), at time t, where rates_total time periods are available for processing, can be mathematically represented as follows:

Currency Index Calculation / Formula

Currency Index Calculation / Formula

 

The contribution of each currency pair in an index’s basket is calculated as above, indexed to base 100.00 from the chosen start date.

The MQL4 indicator source code provided allows the user to set a start date for calculating the first point in the currency index therefore starting at 100.00. The default start date is January 01, 2015.


MQL4 Indicator (Source Code)

We have uploaded a functional MetaTrader 4 Indicator to our GitHub page under tools -> MQL4.

First, simply Right-Click and Save-As on the following links to save the files to your computer, and follow the instructions below them:

Click here to download DLabs_CurrencyIndex.mqh

Click here to download DLabs_CurrencyIndex.mq4

Instructions:

  1. Launch MetaTrader 4, and open your data folder (File -> Open Data Folder)

    MetaTrader 4 Open Data Folder

    MetaTrader 4 Open Data Folder

  2. Copy the “DLabs_CurrencyIndex.mqh” into your MQL4\Include directory,
  3. Copy the “DLabs_CurrencyIndex.mq4” into your MQL4\Indicators directory,
  4. Restart MetaTrader 4 and open a new chart,
  5. Double-click on DLabs_CurrencyIndex under Indicators to load the indicator on any chart.
Currency Index - MetaTrader 4 Indicator (Inputs)

Currency Index – MetaTrader 4 Indicator (Inputs)

The indicator takes two inputs:

  1. Currency Index to draw (e.g. EUR, GBP, etc)
  2. Start date to base the index calculation.

Click OK, and the indicator will draw the selected currency index in its own dedicated window, as shown below:

Currency Index MetaTrader 4 Indicator

Currency Index MetaTrader 4 Indicator

Important Note: In order for the indicator to draw correctly and accurately, users must ensure that their History Center contains data for all currency pairs in an index’s basket.


[Additional Resources] Constructing Currency Portfolio Indexes in MetaTrader 4

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange