ZeroMQ – How To Interface Python/R with MetaTrader 4

Zero MQ - Distributed Messaging

ZeroMQ – Distributed Messaging

In this post, we present a technique employing ZeroMQ (an Open Source, Asynchronous Messaging Library and Concurrency Framework) for building a basic – but easily extensible – high performance bridge between external (non-MQL) programming languages and MetaTrader 4.

 

Reasons for writing this post:

  1. Lack of comprehensive, publicly available literature about this topic on the web.
  2. Traders have traditionally relied on Winsock/WinAPI based solutions that often require revision with both Microsoft™ and MetaQuotes™ updates.
  3. Alternatives to ZeroMQ include named pipes, and approaches where filesystem-dependent functionality forms the bridge between MetaTrader and external languages.

 

Click below to watch the video tutorials:

1) How to Interface Python Trading Strategies with MetaTrader

2) Algorithmic Trading via ZeroMQ: Trade Execution, Reporting & Management

3) Algorithmic Trading via ZeroMQ: Subscribing to Market Data

4) Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 1

5) Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 2


In this blog post, we lay the foundation for a distributed trading system that will:

  1. Consist of one or more trading strategies developed outside MetaTrader 4 (non-MQL),
  2. Use MetaTrader 4 for acquiring market data, trade execution and management,
  3. Support multiple non-MQL strategies interfacing with MetaTrader 4 simultaneously,
  4. Consider each trading strategy as an independent “Client”,
  5. Consider MetaTrader 4 as the “Server”, and medium to market,
  6. Permit both Server and Clients to communicate with each other on-demand.

 

Infographic: ZeroMQ-Enabled Distributed Trading Infrastructure (with MetaTrader 4)

Infographic: ZeroMQ-Enabled Distributed Trading Infrastructure (with MetaTrader 4)

Why ZeroMQ?

  1. Enables programmers to connect any code to any other code, in a number of ways.
  2. Eliminates a MetaTrader user’s dependency on just MetaTrader-supported technology (features, indicators, language constructs, libraries, etc.)
  3. Traders can develop indicators and strategies in C/C#/C++, Python, R and Java (to name a few), and deploy to market via MetaTrader 4.
  4. Leverage machine learning toolkits in Python and R for complex data analysis and strategy development, while interfacing with MetaTrader 4 for trade execution and management.
  5. ZeroMQ can be used as a high-performance transport layer in sophisticated, distributed trading systems otherwise difficult to implement in MQL.
  6. Different strategy components can be built in different languages if required, and seamlessly talk to each other over TCP, in-process, inter-process or multicast protocols.
  7. Multiple communication patterns and disconnected operation.

ZeroMQ: Supported Programming Languages

Though we focus on MQL interfaced with Python & R in this post, the basic process described here can be implemented easily in other ZeroMQ-supported languages.

A comprehensive list of ZeroMQ language bindings is available here:

Zero MQ Language Bindings


Who else is using ZeroMQ?

AT&T, Cisco, EA, Los Alamos Labs, NASA, Weta Digital, Zynga, Spotify, Samsung Electronics, Microsoft, CERN and Darwinex Labs.

ZeroMQ also powers at least 5 DARWINS on The DARWIN Exchange, where the underlying trading strategies were written in C++, Python and R.


Planning Flow Control

This post is not intended to be a detailed tutorial on ZeroMQ.

However, it is still important to understand a few things about ZeroMQ that make it particularly suited to the task of connecting external programming languages such as Python and R to MetaTrader 4.

  • It supports TCP, inter-process, in-process, PGM and EPGM enabled multicast networking. We will use the TCP transport type for the implementation in this post.
  • ZeroMQ enables servers and clients to connect “to each other” on demand, particularly useful for designing distributed trading infrastructure.
  • In addition to support for asynchronous communication and disconnected operation, ZeroMQ supports several communication patterns that permit higher-level data transfer, freeing programmers to focus more on the transfer logic rather than low-level mechanisms.
  • These patterns include: Request (REQ) / Reply (REP), Publish (PUB) / Subscribe (SUB) and Push (PUSH) / Pull (PULL).

 

For the implementation in this blog post, we will employ ZeroMQ’s REQ/REP and PUSH/PULL communication patterns. MetaTrader 4 will be our “Server”, and trading strategies will be “Clients”.

Please note that this (MT4=Server, Strategy=Client) is not a MUST – you will need to decide on whatever flow control suits your particular needs best.

For example, you might designate a machine independent of both the trading strategy as well as MetaTrader 4, as your Server, and have Strategies and MT4 both be Clients. There are a number of ways you could achieve the end goal; carefully planning flow control will lead to efficient functionality.

 

Request (REQ) / Reply (REP) Pattern

The Server (MetaTrader 4 EA) will employ a TCP socket of type REP, to receive requests and send responses. A REP socket MUST always initiate a pair of calls: first, a receive, followed by a send.

The Client (Trading Strategy, e.g. in Python) will employ a TCP socket of type REQ, to send requests and receive responses. A REQ socket MUST always initiate a pair of calls too: first, a send, followed by a receive.

For this implementation, the REQ/REP pattern will enable our Clients to send commands to the MetaTrader 4 Server and receive acknowledgements of the same (e.g. OPEN/MODIFY/CLOSE trades, GET BID/ASK RATES, GET HISTORICAL PRICES, etc.)

 

Push (PUSH) / Pull (PULL) Pattern

The Server (MetaTrader 4 EA) will also employ a second, PUSH socket, to send additional information to Clients (Trading Strategies). This is a one-way socket, and the server will only be able to send data to this socket, without being able to receive anything back through the same socket.

The Client (Trading Strategy) will also employ a second, PULL socket, to receive additional information from the Server. This too is a one-way socket, and the client will only be able to receive data from this socket, without being able to send anything through the same socket.

The PUSH/PULL pattern enables servers and clients to exchange data with each other on-demand, but in one direction without expecting a response. This could of course be swapped out for another REQ/REP pattern, depending on your application’s flow control requirements.

 

In summary, for this post’s basic implementation:

  1. The Server will employ two sockets, one REP and one PUSH.
  2. Each Client will employ two sockets, one REQ and one PULL.

 

Infographic: What this flow control plan looks like in practice.

Infographic: ZeroMQ Process Flow Control

 


MetaTrader 4 Expert Advisor – Components

As displayed in the infographic above, the MT4 EA will serve as our ZeroMQ-enabled Server, with three main modules:

  1. MESSAGE ROUTER – This allows the EA to receive commands and send acknowledgements back to connecting Clients (trading strategies) through the REP socket. The Router passes all messages on to the Parser. Note: For this example, the Router doesn’t serve much purpose, but it is good practice to have this intermediary where several strategies connect to the Server (MT4) and some manner of pre-parse actions may need to be performed.
  2. MESSAGE PARSER – Messages received by this module are decomposed into actions for the next module (Interpreter & Executor).
  3. INTERPRETER & EXECUTOR – This module literally “interprets” decomposed messages and performs requested actions accordingly. For example, if the Client is requesting market data, the module gathers it from the MetaTrader 4 History DB and sends it on to the Client via the PUSH socket. Alternatively, if the Client is requesting a BUY or SELL trade be opened on e.g. the EUR/USD, it sends the trade to market and a notification of success/failure/ticket-info to the Client via the PUSH socket.

Implementation Requirements

  1. ZeroMQ – MQL4 Bindings -> Download and install the required files as instructed here: https://github.com/dingmaotu/mql-zmq
  2. For Python -> “pyzmq” library
  3. For R -> “rzmq” library

Sample Code

To give you a head start, we’ve published a functional MetaTrader 4 Expert Advisor with the full implementation discussed in this blog post.

The MQL sample code provided is quite extensible, and can be used as a template in your efforts.

GitHub Links:

  1. DWX ZeroMQ Connector – Python & MQL

Notes:

  1. The Python source code demonstrates how communication patterns are implemented.
  2. It’s fairly simple to integrate this code in your existing Python/R trading strategies.

 


[WEBINAR REPLAY] How to Interface Python/R Trading Strategies with MetaTrader 4


[VIDEO TUTORIAL] Algorithmic Trading via ZeroMQ: Trade Execution, Reporting & Management (Python to MetaTrader)


[VIDEO TUTORIAL] Algorithmic Trading via ZeroMQ: Subscribing to Market Data (Python to MetaTrader)


[VIDEO TUTORIAL] Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 1


[VIDEO TUTORIAL] Build Algorithmic Trading Strategies with Python & ZeroMQ: Part 2


Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

Hedging DARWIN Portfolio Risk with $DWC

In this blog post, we’ll discuss how DARWIN Investors can diversify away some of the excess risk posed to their portfolios by Loss Aversion, a common and well-researched phenomenon in behavioural finance.

In particular, we’ll discuss why it makes sense to include DARWIN $DWC in a portfolio that’s partially or entirely composed of loss averse DARWINS.


But first,

  1. A quick recap on what Loss Aversion is,
  2. Why even the most perfectly diversified portfolio of DARWINs can be susceptible to unforeseen shocks due to loss averse behaviour,
  3. What DARWIN Investors can do to hedge this risk.

Loss Aversion (Illustration)

Loss Aversion (Illustration)

Loss Aversion:

Simply put, traders are said to be loss averse when they hold on to losing trades for extended periods of time, but take quick profits on winning trades.

It yields two main outcomes:

  1. Returns growth looks fairly steady during periods of profitability, small profits smoothing the curve.
  2. Major drawdowns however, are disproportionately larger – sometimes leading to prior profits being wiped out by the closure of large losing trades that were being held on to for a long period of time.

 

Granted, a diversified portfolio of reasonably uncorrelated DARWINs has its advantages in terms of minimizing overall portfolio risk.

However, if it contains DARWINs with poor scores for the Loss Aversion attribute (La), it may still be susceptible to shocks during:

  1. Periods of market turbulence,
  2. Deep unforeseen movements,
  3. Unusually volatile news releases,
  4. Black swan events, etc.

.. where diversification benefit temporarily breaks down, owing in part to losing trade closures that distort the portfolio’s original risk profile.


What can DARWIN Investors do to protect themselves?

In one of our recent posts – $DWC – A Real Time Sentiment Index & Security – we highlighted the fact that DARWIN $DWC replicates the opposite of the Darwinex trader collective’s behaviour.

GBP Flash Crash (October, 2016)

GBP Flash Crash (October, 2016)

It typically rises during times when loss averse traders experience undiversifiable risk.

For example, $DWC profited from the GBP Flash Crash.

Undiversifiable risk also frequently presents itself when loss aversion eventually leads traders towards margin calls, causing sudden, unexpected volatility in the overlying DARWINs.

 

In such situations, portfolios that contain DARWIN $DWC can benefit from DWC hedging away a significant proportion (depending on position management of course) of undiversifiable risk experienced by investors.


When does it make sense to include $DWC in a portfolio?

  1. Investors can include $DWC in their portfolios to hedge against DARWINs with a Loss Aversion (La) score < 4.0 and Capacity (Cp) score > 5.0.
  2. Capacity (Cp) > 5.0 describes DARWINs that primarily trade long term movements. If such DARWINs also have a Loss Aversion (La) score < 4.0, the investor’s portfolio is likely exposed to undiversifiable risk at some point in the future.

    Low Loss Aversion Score

    Low Loss Aversion Score

  3. Check the Correlation of low La DARWINS against the $DWC – If they are negatively correlated, it becomes more likely that $DWC will offset excess risk should the loss averse DARWIN encounter  undiversifiable risk.

    Check Correlation of Loss Averse DARWIN Against DWC

    Check Correlation of Loss Averse DARWIN Against DWC


What composition of assets could well diversified DARWIN portfolios (hedged against loss aversion) contain?

Example Portfolio #1:

  1. 50% Short Term DARWINs (Capacity < 5.0)
  2. 25% Long Term DARWINs (Capacity >= 5.0)
  3. 25% allocation to $DWC as a hedge against loss aversion / undiversifiable risk.

Example Portfolio #2:

  1. 40% Short Term DARWINs (Capacity < 5.0)
  2. 30% Long Term DARWINs (Capacity >= 5.0)
  3. 30% allocation to $DWC as a hedge against loss aversion / undiversifiable risk.

Note: Investors should of course exercise their own discretion in selecting portfolio allocations. The examples above illustrate what such allocations could look like, accounting for multiple timing horizons whilst hedged to a reasonable degree against loss aversion.


Webinar Recording: Effects of Market Volatility on Trader Performance

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

Darwinex – The Open Trader Exchange