ZeroMQ - Transaction Reporting via MetaTrader (ZMQ-III)

ZeroMQ – Transaction Reporting via MetaTrader (ZMQ-III)

In this third installment of our ZeroMQ series, we describe how to use ZeroMQ in non-MQL trading strategies to get the following information:

  1. Account Information (e.g. equity, margin, balance, etc)
  2. Trades at market (live or pending)
  3. Historical Trades

If you haven’t already, please consider reading the following posts before proceeding further in this article:

  1. ZMQ-I: How to Interface Python/R with MetaTrader 4
  2. ZMQ-II: Trade Execution in MetaTrader

 

Previous posts in this series relied heavily on the Request (REQ) / Reply (REP) communication pattern in ZeroMQ for trading related message exchanges.

Unless transaction information factors into the trading strategy or other critical function, we can safely employ the PUSH/PULL communication pattern for retrieving account and trade-related data from MetaTrader 4.

The ZMQ-enabled MetaTrader EA server published on our GitHub profile implements an example InformPullClient() function that sends information via a PUSH socket to client applications listening at the other end:

void InformPullClient(Socket& pushSocket, string message) {

ZmqMsg pushReply(StringFormat("%s", message));
 
 pushSocket.send(pushReply,true); // NON-BLOCKING
 // pushSocket.send(pushReply,false); // BLOCKING
 
}

Any listening applications (e.g. Python/R trading strategies) – in this implementation – will still send requests via the REQ/REP protocol to the ZeroMQ server (MetaTrader EA).

Once received, a request will be parsed into its constituent components and sent through to the Interpreter as usual.


For sake of completeness, here is the diagram describing flow of data between our ZeroMQ clients and servers:

Infographic: ZeroMQ Process Flow Control


RECAP: Message Topology

In our last post on trade execution via ZeroMQ, we defined a few message formats to enable information exchange between MetaTrader and external trading strategies.

Here we’ll dig further into implementing the “ACCOUNT” (account information) and “HISTORY” (trade reporting) formats discussed therein.


Account Information

When it comes to account specifics, traders typically look for the following information:

  1. Balance
  2. Current P/L
  3. Equity
  4. Margin Used
  5. Free Margin
  6. Margin Level
  7. Margin Call Level
  8. Margin Stop Out Level

 

For our purposes in this post, we could associate a unique identifier (e.g. numeric values like 1, 2, 3 etc) for each item in the list above.

The corresponding commands sent by an external trading strategy to the ZeroMQ EA in MetaTrader, would then be as follows:

ACCOUNT|1 # balance
ACCOUNT|2 # current P/L
ACCOUNT|3 # equity
ACCOUNT|4 # margin used
ACCOUNT|5 # free margin
ACCOUNT|6 # margin level
ACCOUNT|7 # margin call level
ACCOUNT|8 # margin stop out level
ACCOUNT|0 # all of the above.

 

Once a message (request for account information) is received, the ZeroMQ EA in MetaTrader will need to parse, interpret and respond accordingly.

Assuming you’re working with the example expert advisor we published in recent posts, you’ll need to implement this message functionality inside the InterpretZmqMessage() function.


As there are several ways to do this, we’ll leave the implementation to the user, but a simple (deliberately inefficient for clarity) example of how to do this, is:

  1. Add a condition to check for the “ACCOUNT” leading tag.

    if(compArray[0] == "ACCOUNT" && compArray[1] = "0") switch_action = 5;
  2. Implement the condition inside the switch() block:

    case 5:
    
     InformPullClient(pSocket, "ACCOUNT Instruction (ALL) Received");
     // IMPLEMENT LOGIC HERE
    
    string account_info = DoubleToString(AccountInfoDouble(ACCOUNT_BALANCE),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_PROFIT),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_EQUITY),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN_FREE),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN_LEVEL),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN_SO_CALL),2) + "|" +
     DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN_SO_SO),2);
    
    InformPullClient(pSocket, StringFormat("%s", account_info));

 

The trading strategy – ZeroMQ client –  will retrieve and parse this message via a PULL socket.

Example message received:
1114.73|-2.97|1111.76|10.00|1101.76|11117.60|100.00|50.00

From left to right, this response is interpreted as:
BALANCE | CURRENT P/L | EQUITY | MARGIN IN USE | FREE MARGIN | MARGIN LEVEL | MARGIN CALL LEVEL | MARGIN STOP OUT LEVEL


Trades at Market (live or pending)

For market and pending orders, traders using MetaTrader will typically require the following information:

  1. Ticket Number
  2. Open Time
  3. Type (BUY or SELL)
  4. Lot Size (e.g. 0.01, 0.1 and so on)
  5. Symbol (e.g. EUR/USD)
  6. Open Price (e.g. 1.12000)
  7. Stop Loss Price
  8. Take Profit Price
  9. Commission Paid
  10. Swap
  11. Current P/L
  12. Trade Comment

The fields above are the same (and in the same order) as those displayed in the MetaTrader terminal.

The OrdersTotal() function in MetaTrader is required to retrieve information on trades either currently in execution, or still pending.

In exactly the same manner as used above to retrieve ACCOUNT information, you’ll need to implement additional functionality inside InterpretZmqMessage() for this.

We’ll demonstrate this below using a single message for now:

HISTORY|0 # Retrieve all open pending and market orders.

Once again, as there are several ways to do this, we’ll keep the following example as simple as possible for you to build upon:

  1. Add a condition to check for the “HISTORY” leading tag.

    if(compArray[0] == "HISTORY" && compArray[1] = "0") switch_action = 6;
  2. Implement the condition inside the switch() block:

    case 6:
    
     InformPullClient(pSocket, "Live/Pending Orders Instruction (ALL) Received");
     // IMPLEMENT LOGIC HERE
    int live_orders = OrdersTotal();
    string live_info = "";
    
    for(int i=0; i < live_orders; i++)
    {
     if(OrderSelect(i,SELECT_BY_POS)==false) continue;
     live_info = live_info + IntegerToString(OrderTicket()) + "," +
      TimeToString(OrderOpenTime()) + "," +
      IntegerToString(OrderType()) + "," +
      DoubleToString(OrderLots(),2) + "," +
      OrderSymbol() + "," +
      DoubleToString(OrderOpenPrice(),5) + "," +
      DoubleToString(OrderStopLoss(),5) + "," +
      DoubleToString(OrderTakeProfit(),5) + "," +
      DoubleToString(OrderCommission(),2) + "," + 
      DoubleToString(OrderSwap(),2) + "," +
      DoubleToString(OrderProfit(),2) + "," +
      "<" + OrderComment() + ">|";
     
    }
    InformPullClient(pSocket, StringFormat("%s", live_info));

 

The trading strategy – ZeroMQ client –  will retrieve and parse this message via a PULL socket.

Example message received:
12345678,2018.02.16 12:00,1,0.01,USDJPY,100.12300,0.00000,0.00000,-0.05,0.00,-1.23,<My Trading Strategy>|

From left to right, this response is interpreted as:
TICKET, OPEN TIME, TYPE (BUY OR SELL), LOT SIZE, SYMBOL, OPEN PRICE, STOP LOSS PRICE, TAKE PROFIT PRICE, COMMISSION, SWAP, COMMENT

 

Note #1: There may be more than one trade currently in execution (or pending).

Using the message format above, each such trade will be separated by a “|” character in the output to the receiving client.

Note #2: Order types are returned via MetaTrader’s OrderType() function, as integer values, corresponding to:

OP_BUY – buy order,
OP_SELL – sell order,
OP_BUYLIMIT – buy limit pending order,
OP_BUYSTOP – buy stop pending order,
OP_SELLLIMIT – sell limit pending order,
OP_SELLSTOP – sell stop pending order.


Historical Trades

Retrieving historical trades from MetaTrader is a matter of changing the relevant function to OrdersHistoryTotal() and implementing a near-similar modification to InterpretZmqMessage() as shown above for live/pending orders.


Conclusion

ZeroMQ enables traders using any of its supported programming languages, to:

  1. Create trading strategies outside MetaTrader
  2. Use MetaTrader as the intermediary to market
  3. Implement advanced trading techniques otherwise difficult (if not impossible) to implement in MetaTrader
  4. Evaluate performance in much greater detail, e.g. by leveraging statistical techniques not available in MetaTrader
  5. Leverage the Darwinex Analytical Toolkit to fine-tune investment-worthiness
  6. .. and much more.

As always, we hope you’ve enjoyed this tutorial, and look forward to any feedback you may have for us.

Kindly share this post using the buttons provided, with your colleagues and networks you feel would benefit from the content.. or just share it anyway to help us spread the word! 🙂


Webinar Recording: How to Interface Python/R Trading Strategies with MetaTrader 4

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

8 replies
  1. Joaquin
    Joaquin says:

    It would be nice if this code was added to Darwinex Github (unless I missed it, in github we only have the code related to the first article in these series of zeromq).
    Also, I’d like to know how I can diagnose errors with zeromq. Let me introduce you to the issues I’m running into:
    I’m using the code in github as a base for my R strategies to use them in MT4. The R codes runs in one computer, and the MT4 runs in another. So I only change the IP en the R code to connect to the MT4 sockets, and the connection is made successfully.
    If I run, for instance, this instruction (this is executed in the R program):
    TRADE|CLOSE|l
    [1] “[SERVER] Processing: TRADE|CLOSE|l” -> this is the answer from MT4
    Everything works all right.
    But if I run this other instruction:
    TRADE|CLOSE|lasdkjflsajdhflasdhfalsdfhalsdjsadhf

    I get no answer from MT4, and MT4 closes unexpectedly, so I guess there must be some buffer overflow somewhere. In the logs folder under MQL4, I can see this error:
    Access violation read to 0x0C236C98 in ‘C:\Users\(…)70E3C72C958D85D2AEDB0F6CA0993748\MQL4\Libraries\libzmq.dll’

    So, now the question is… how can I get around this problem?

    Thanks!

    Reply
    • Joaquin
      Joaquin says:

      OK…

      I have just download the ‘official’ build of libzmq.dll and now at least this bug has stopped. I was using the libzmq suggested in the first article of zmq here, on blog.darwinex.com so I suggest you comment on that.

      Anyway, I still don’t know what could be the best tool to diagnose these errors… I have used wireshark , dbgview and others but in this problem it was of no help at all… Suggestions?

      Reply
      • Joaquin
        Joaquin says:

        ok, I’ve got it… It was the REP socket. From MT4, to R, with the template EA in github, if “reply” is big enough in repSocket.send(reply) , MT4 crashes.

        Big enough is not very big… I haven´t checked exactly, but around 30 characters.

        I will keep on looking for a solution. Now, at least, I know how to solve my problem: Don’t send through REP socket long strings! 🙂

        Reply
        • The Market Bull
          The Market Bull says:

          Hi Joaquin,

          Thank you for documenting your debug journey like this.. this is fantastic for everyone who could have experienced a similar issue! 🙂

          We will update the post in due course, highlight this scenario and reference you as the fixer!

          Thanks again for your interest & have a great weekend!

          Reply
  2. Oguz
    Oguz says:

    I want to send TickData of multiple Symbols to ZMQ, But all of your samples are using MT4 side as server, How can I implement this requirement.

    Single .Net ZMQ Server (binded to 5555)
    Multiple Mql ZMQ Client (connected to 5555)

    Communication between MT4 to .NET is one way (no reply)

    .Net side is not problem I can write, but mql side is problematic.

    Reply
    • The Market Bull
      The Market Bull says:

      Hi Oguz,

      Great question! This series of posts used MetaTrader as the server.

      To achieve what you need, you’ll need to modify the template such that MetaTrader acts as the client instead. You can then have multiple EA instances connecting to your .Net server.

      Am I correct in understanding that you’d like to use tick data from an external source via your .Net enabled server and not MetaTrader?

      This should be entirely possible, yes!

      Reply
    • The Market Bull
      The Market Bull says:

      Hi Frank,

      Thank you for your interest.

      It is indeed technically possible to achieve this. You’d need to implement a switch-case option inside the InterpretZMQMessage() function in the following MQL EA:

      https://github.com/darwinex/dwx-zeromq-connector/blob/master/v2.0.1/mql4/DWX_ZeroMQ_Server_v2.0.1_RC8.mq4

      The EA code is structured in a way that makes it easy for algorithmic traders to extend functionlity as they wish. In your case, you’d need to write a function that sends data back to your Python environment via ZeroMQ, when the additional switch option you implement in InterpretZMQMessage(), is called.

      Take a look at the other switch options for examples on how this works. You’ll find that each switch option has its own dedicated function further down in the code.

      Hope this helps!

      Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *