Posts

Hidden Markov Models & Regime Change: S&P500

In this post, we will employ a statistical time series approach using Hidden Markov Models (HMM), to firstly obtain visual evidence of regime change in the S&P500.

Detecting significant, unforeseen changes in underlying market conditions (termed “market regimes“) is one of the greatest challenges faced by algorithmic traders today. It is therefore critical that traders account for shifts in these market regimes during trading strategy development.

Why use Hidden Markov Models?

Hidden Markov Models for Detecting Market Regime Change (Source: Wikipedia)

Hidden Markov Models for Detecting Market Regime Change (Source: Wikipedia)

 

Hidden Markov Models infer “hidden states” in data by using observations (in our case, returns) correlated to these states (in our case, bullish, bearish, or unknown).

They are hence a suitable technique for detecting regime change, enabling algorithmic traders to optimize entries/exits and risk management accordingly.

We will make use of the depmixS4 package in R to analyse regime change in the S&P500 Index.

 

Hidden Markov Model - State Space Model (Source: StackExchange)

Hidden Markov Model – State Space Model (Source: StackExchange)

With any state-space modelling effort in quantitative finance, there are usually three main types of problems to address:

  1. Prediction – forecasting future states of the market
  2. Filtering – estimating the present state of the market
  3. Smoothing – estimating the past states of the market

We will be using the Filtering approach.

Additionally, we will assume that since S&P500 returns are continuous, the probability of seeing a particular return R in time t, with market regime M being in state m, where the model used has parameter-set P, is described by a multivariate normal distribution with mean μ and standard deviation σ [1].

Mathematically, this can be expressed as:

\(p(R_t | M_t = m, P) = N(R_t | μ_m, σ_m)\)

As noted earlier, we will utilize the Dependent Mixture Models package in R (depmixS4) for the purposes of:

  1. Fitting a Hidden Markov Model to S&P500 returns data.
  2. Determining posterior probabilities of being in one of three market states (bullish, bearish or unknown), at any given time.

We will then use the plotly R graphing library to plot both the S&P500 returns, and the market states the index was estimated to have been in over time.

You may replicate the following R source code to conduct this analysis on the S&P500.

Step 1: Load required R libraries

library(quantmod)
library(plotly)
library(depmixS4)

Step 2: Get S&P500 data from June 2014 to March 2017

getSymbols("^GSPC", from="2014-06-01", to="2017-03-31")

Step 3: Calculate differenced logarithmic returns using S&P500 EOD Close prices.

sp500_temp = diff(log(Cl(GSPC)))
sp500_returns = as.numeric(sp500_temp)

Step 4: Plot returns from (3) above on plot_ly scatter plot.

plot_ly(x = index(GSPC), y = sp500_returns, type="scatter", mode="lines") %>%

layout(xaxis = list(title="Date/Time (June 2014 to March 2017)"), yaxis = list(title="S&P500 Differenced Logarithmic Returns"))

S&P500 Differenced Logarithmic Returns
(June 2014 to March 2017)

S&P500 Differenced Logarithmic Returns (June 2014 to March 2017)

S&P500 Differenced Logarithmic Returns (June 2014 to March 2017)

Step 5: Fit Hidden Markov Model to S&P500 returns, with three “states”

hidden_markov_model <- depmix(sp500_returns ~ 1, family = gaussian(), nstates = 3, data = data.frame(sp500_returns=sp500_returns))

model_fit <- fit(hidden_markov_model)

Step 6: Calculate posterior probabilities for each of the market states

posterior_probabilities <- posterior(model_fit)

Step 7: Overlay calculated probabilities on S&P500 cumulative returns

sp500_gret = 1 + sp500_returns
sp500_gret <- sp500_gret[-1]
sp500_cret = cumprod(sp500_gret)

plot_ly(name="Unknown", x = index(GSPC), y = posterior_probabilities$S1, type="scatter", mode="lines", line=list(color="grey")) %>%

add_trace(name="Bullish", y = posterior_probabilities$S2, line=list(color="blue")) %>%

add_trace(name="Bearish", y = posterior_probabilities$S3, line=list(color="red")) %>%

add_trace(name="S&P500", y = c(rep(NA,1), sp500_cret-1), line=list(color="black"))

S&P500 Market Regime Probabilities
(June 2014 to March 2017)

S&P500 Hidden Markov Model States (June 2014 to March 2017)

S&P500 Hidden Markov Model States (June 2014 to March 2017)

Interpretation: In any one “market regime”, the corresponding line/curve will “cluster” towards the top of the y-axis (i.e. near a probability of 100%).

For example, during a brief bullish run starting on 01 June 2014, the blue line/curve clustered near y-axis value 1.0. This correlates as you can see, with movement in the S&P500 (black line/curve). The same applies to bearish and “unknown” market states.

An interesting insight one can draw from this graphic, is how the Hidden Markov Model successfully reveals high volatility in the market between June 2014 and March 2015 (constantly changing states between bullish, bearish and unknown).

 

References:

[1] Murphy, K.P. (2012) Machine Learning – A Probabilistic Perspective, MIT Press.
https://www.cs.ubc.ca/~murphyk/MLbook/

Influences:

The honourable Mr. Michael Halls-Moore. QuantStart.com
http://www.quantstart.com/

Additional Resource: Learn more about DARWIN Portfolio Risk (VIDEO)
* please activate CC mode to view subtitles.

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

Darwinex – The Open Trader Exchange

Retail vs. Institutional Traders

真正的算法交易讲解和说明系列集 – 第一篇

个人独立交易员真的能否可以与大型机构抗衡?

pastedGraphic.png

 

本文为《真正的算法交易讲解和说明系列集》(Serious Algorithmic Trading)系列的第一篇。

该系列共有四篇博文,其开篇序文部分可参见此处。

交易员们总会在某些时候质疑其与同一市场内的大型机构竞争的实际可能性。

毕竟,最终推动这些市场的就是“大资金”。

现实情况是,机构面临着众多的监管、技术、结构和资本限制,而个人交易员则不受它们束缚。

也正是这些限制导致机构资金往往会展现出个人交易员可看到且可利用的一定可预测性。该现象将在第三篇博文《算法研发的定量方法》(Quantitative Approach to Algorithmic R&D)中进行更为详细的讨论。

因此,对问题的回答是

让我们从规模与市场影响、风险管理以及交易技术的角度来考察原因。

规模与市场影响

个人交易员可设计和执行针对小型市场无效性的交易策略。

这种市场无效性通常在容纳相当有限的资本后就不再具有盈利性。

该有限的资本支持通常在数十万到数百万美元之间,因此对于在交易显著更大规模资本基础的机构资金而言,该机会只能引发它们的极少兴趣或者根本不会引起关注。

注意:为了我们社群的利益,会在每一个在我们“虚拟在线交易所中”所上市的DARWIN资产上显示规模/容量投资属性。这是一个从110的分数,显示相关策略的可投资性。分数越高,策略可管理的资产(AuM)规模也越多,可推广性也也大。

而且,个人交易员的资本总额远低于机构。因此在高度流动市场中,散户的交易活动很难产生任何显著的市场影响,这是一个优势。

风险管理

pastedGraphic.png

风险管理

因无中间部门或合规办公室执行行业标准和监管监察,个人交易员有选择余地,可视情况对自身的风险管理技术建模,提高能够确确实实促进更多收益生成的灵活性。

但是,

虽然这肯定是部分有经验交易员的优势,它是一把双刃剑。

因无“被驳倒”的风险,个人交易员有更大的风险执行“非最优”风险管理决策,这通常会导致不利的结果,如因过度利用杠杆或在机构环境中本不会被许可的激进风险管理而导致账户爆仓。

请注意:我们的“虚拟在线交易所”DARWIN的投资者们解决了这一问题。在我们的“虚拟在线交易所”中“上市的”策略的交易员从不会自行管理投资者资本。

我们管理所有的Darwin 资产,并执行自身的风险管理以向投资者提供固定的风险值,由此将投资者与相关策略的特定风险状况相隔离。交易者仅将其知识产权授权给我们,以获得表现费(Performance Fees )作为交换—— 表现费为为投资者所生成利润的20%。

此外,会在所有列出的DARWIN资产上显示风险管理投资属性。这是一个从110的分数,显示相关策略持续利用杠杆产生稳定风险的能力。分数越高,策略的可投资性也越高。

而且,因未执行行业最佳实践和风险管理监管,个人交易员通常会发现自身在执行层面进行风险建模(如止损和获利了结操作),对于投资组合层面(如同一账户中部署的资产或策略组合)则不会考虑很多。

请注意:我们为DARWIN的投资者解决了这一问题。我们的投资平台支持投资者在做出任何投资决策前先查看选择任何DARWIN资产组合的相关性和多元化优势。

交易技术

pastedGraphic_1.png

金融交易技术

就交易技术选择和策略制定而言,机构交易者所享有的灵活度不如散户交易员。

散户交易员能够在各种服务器、硬件、交易平台、编程语言和工具组合中进行选择,而不会有公司IT政策或预先确定的“许可系统”清单来影响他们的技术偏好。

虽然这一灵活度也伴随着一个唯一的不利因素,即购买硬件、软件和服务器订阅的成本会相当高(相对于散户交易员的可用资金而言)。这些成本必须由交易员自身承担,而机构中该成本很可能会通过所收取的管理费得到降低。

在下一博文中,我们将讨论算法/定量交易的关键优缺点。