ZeroMQ – How To Interface Python/R with MetaTrader 4

27 August 2017
The Market Bull
Zero MQ - Distributed Messaging

ZeroMQ – Distributed Messaging

In this post, we present a technique employing ZeroMQ (an Open Source, Asynchronous Messaging Library and Concurrency Framework) for building a basic – but easily extensible – high performance bridge between external (non-MQL) programming languages and MetaTrader 4.

 

Reasons for writing this post:

  1. Lack of comprehensive, publicly available literature about this topic on the web.
  2. Traders have traditionally relied on Winsock/WinAPI based solutions that often require revision with both Microsoft™ and MetaQuotes™ updates.
  3. Alternatives to ZeroMQ include named pipes, and approaches where filesystem-dependent functionality forms the bridge between MetaTrader and external languages.

Click here to watch the Webinar Recording.


In this blog post, we lay the foundation for a distributed trading system that will:

  1. Consist of one or more trading strategies developed outside MetaTrader 4 (non-MQL),
  2. Use MetaTrader 4 for acquiring market data, trade execution and management,
  3. Support multiple non-MQL strategies interfacing with MetaTrader 4 simultaneously,
  4. Consider each trading strategy as an independent “Client”,
  5. Consider MetaTrader 4 as the “Server”, and medium to market,
  6. Permit both Server and Clients to communicate with each other on-demand.

 

Infographic: ZeroMQ-Enabled Distributed Trading Infrastructure (with MetaTrader 4)

Infographic: ZeroMQ-Enabled Distributed Trading Infrastructure (with MetaTrader 4)

Why ZeroMQ?

  1. Enables programmers to connect any code to any other code, in a number of ways.
  2. Eliminates a MetaTrader user’s dependency on just MetaTrader-supported technology (features, indicators, language constructs, libraries, etc.)
  3. Traders can develop indicators and strategies in C/C#/C++, Python, R and Java (to name a few), and deploy to market via MetaTrader 4.
  4. Leverage machine learning toolkits in Python and R for complex data analysis and strategy development, while interfacing with MetaTrader 4 for trade execution and management.
  5. ZeroMQ can be used as a high-performance transport layer in sophisticated, distributed trading systems otherwise difficult to implement in MQL.
  6. Different strategy components can be built in different languages if required, and seamlessly talk to each other over TCP, in-process, inter-process or multicast protocols.
  7. Multiple communication patterns and disconnected operation.

ZeroMQ: Supported Programming Languages

Though we focus on MQL interfaced with Python & R in this post, the basic process described here can be implemented easily in other ZeroMQ-supported languages.

A comprehensive list of ZeroMQ language bindings is available here:

Zero MQ Language Bindings


Who else is using ZeroMQ?

AT&T, Cisco, EA, Los Alamos Labs, NASA, Weta Digital, Zynga, Spotify, Samsung Electronics, Microsoft, CERN and Darwinex Labs.

ZeroMQ also powers at least 5 DARWINS on The DARWIN Exchange, where the underlying trading strategies were written in C++, Python and R.


Planning Flow Control

This post is not intended to be a detailed tutorial on ZeroMQ.

However, it is still important to understand a few things about ZeroMQ that make it particularly suited to the task of connecting external programming languages such as Python and R to MetaTrader 4.

  • It supports TCP, inter-process, in-process, PGM and EPGM enabled multicast networking. We will use the TCP transport type for the implementation in this post.
  • ZeroMQ enables servers and clients to connect “to each other” on demand, particularly useful for designing distributed trading infrastructure.
  • In addition to support for asynchronous communication and disconnected operation, ZeroMQ supports several communication patterns that permit higher-level data transfer, freeing programmers to focus more on the transfer logic rather than low-level mechanisms.
  • These patterns include: Request (REQ) / Reply (REP), Publish (PUB) / Subscribe (SUB) and Push (PUSH) / Pull (PULL).

 

For the implementation in this blog post, we will employ ZeroMQ’s REQ/REP and PUSH/PULL communication patterns. MetaTrader 4 will be our “Server”, and trading strategies will be “Clients”.

Please note that this (MT4=Server, Strategy=Client) is not a MUST – you will need to decide on whatever flow control suits your particular needs best.

For example, you might designate a machine independent of both the trading strategy as well as MetaTrader 4, as your Server, and have Strategies and MT4 both be Clients. There are a number of ways you could achieve the end goal; carefully planning flow control will lead to efficient functionality.

 

Request (REQ) / Reply (REP) Pattern

The Server (MetaTrader 4 EA) will employ a TCP socket of type REP, to receive requests and send responses. A REP socket MUST always initiate a pair of calls: first, a receive, followed by a send.

The Client (Trading Strategy, e.g. in Python) will employ a TCP socket of type REQ, to send requests and receive responses. A REQ socket MUST always initiate a pair of calls too: first, a send, followed by a receive.

For this implementation, the REQ/REP pattern will enable our Clients to send commands to the MetaTrader 4 Server and receive acknowledgements of the same (e.g. OPEN/MODIFY/CLOSE trades, GET BID/ASK RATES, GET HISTORICAL PRICES, etc.)

 

Push (PUSH) / Pull (PULL) Pattern

The Server (MetaTrader 4 EA) will also employ a second, PUSH socket, to send additional information to Clients (Trading Strategies). This is a one-way socket, and the server will only be able to send data to this socket, without being able to receive anything back through the same socket.

The Client (Trading Strategy) will also employ a second, PULL socket, to receive additional information from the Server. This too is a one-way socket, and the client will only be able to receive data from this socket, without being able to send anything through the same socket.

The PUSH/PULL pattern enables servers and clients to exchange data with each other on-demand, but in one direction without expecting a response. This could of course be swapped out for another REQ/REP pattern, depending on your application’s flow control requirements.

 

In summary, for this post’s basic implementation:

  1. The Server will employ two sockets, one REP and one PUSH.
  2. Each Client will employ two sockets, one REQ and one PULL.

 

Infographic: What this flow control plan looks like in practice.

Infographic: ZeroMQ Process Flow Control

 


MetaTrader 4 Expert Advisor – Components

As displayed in the infographic above, the MT4 EA will serve as our ZeroMQ-enabled Server, with three main modules:

  1. MESSAGE ROUTER – This allows the EA to receive commands and send acknowledgements back to connecting Clients (trading strategies) through the REP socket. The Router passes all messages on to the Parser. Note: For this example, the Router doesn’t serve much purpose, but it is good practice to have this intermediary where several strategies connect to the Server (MT4) and some manner of pre-parse actions may need to be performed.
  2. MESSAGE PARSER – Messages received by this module are decomposed into actions for the next module (Interpreter & Executor).
  3. INTERPRETER & EXECUTOR – This module literally “interprets” decomposed messages and performs requested actions accordingly. For example, if the Client is requesting market data, the module gathers it from the MetaTrader 4 History DB and sends it on to the Client via the PUSH socket. Alternatively, if the Client is requesting a BUY or SELL trade be opened on e.g. the EUR/USD, it sends the trade to market and a notification of success/failure/ticket-info to the Client via the PUSH socket.

Implementation Requirements

  1. ZeroMQ – MQL4 Bindings -> Download and install the required files as instructed here: https://github.com/dingmaotu/mql-zmq
  2. For Python -> “pyzmq” library
  3. For R -> “rzmq” library

Sample Code

To give you a head start, we’ve published a functional MetaTrader 4 Expert Advisor with the full implementation discussed in this blog post.

The MQL sample code provided is quite extensible, and can be used as a template in your efforts.

GitHub Links:

  1. ZeroMQ-enabled MT4 Expert Advisor – GitHub
  2. Example ZeroMQ Client in R – GitHub
  3. Example ZeroMQ Client in Python – GitHub

Notes:

  1. The Python and R samples demonstrate how communication patterns are implemented.
  2. It’s fairly simple to integrate this code in your existing Python/R trading strategies.

 


Webinar Recording: How to Interface Python/R Trading Strategies with MetaTrader 4

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange