Posts

Hidden Markov Models & Regime Change: S&P500

/0 Comments/in /by

In this post, we will employ a statistical time series approach using Hidden Markov Models (HMM), to firstly obtain visual evidence of regime change in the S&P500.

Detecting significant, unforeseen changes in underlying market conditions (termed “market regimes“) is one of the greatest challenges faced by algorithmic traders today. It is therefore critical that traders account for shifts in these market regimes during trading strategy development.

Why use Hidden Markov Models?

Hidden Markov Models for Detecting Market Regime Change (Source: Wikipedia)

Hidden Markov Models for Detecting Market Regime Change (Source: Wikipedia)

 

Hidden Markov Models infer “hidden states” in data by using observations (in our case, returns) correlated to these states (in our case, bullish, bearish, or unknown).

They are hence a suitable technique for detecting regime change, enabling algorithmic traders to optimize entries/exits and risk management accordingly.

We will make use of the depmixS4 package in R to analyse regime change in the S&P500 Index.

 

Hidden Markov Model - State Space Model (Source: StackExchange)

Hidden Markov Model – State Space Model (Source: StackExchange)

With any state-space modelling effort in quantitative finance, there are usually three main types of problems to address:

  1. Prediction – forecasting future states of the market
  2. Filtering – estimating the present state of the market
  3. Smoothing – estimating the past states of the market

We will be using the Filtering approach.

Additionally, we will assume that since S&P500 returns are continuous, the probability of seeing a particular return R in time t, with market regime M being in state m, where the model used has parameter-set P, is described by a multivariate normal distribution with mean μ and standard deviation σ [1].

Mathematically, this can be expressed as:

\(p(R_t | M_t = m, P) = N(R_t | μ_m, σ_m)\)

As noted earlier, we will utilize the Dependent Mixture Models package in R (depmixS4) for the purposes of:

  1. Fitting a Hidden Markov Model to S&P500 returns data.
  2. Determining posterior probabilities of being in one of three market states (bullish, bearish or unknown), at any given time.

We will then use the plotly R graphing library to plot both the S&P500 returns, and the market states the index was estimated to have been in over time.

You may replicate the following R source code to conduct this analysis on the S&P500.

Step 1: Load required R libraries

library(quantmod)
library(plotly)
library(depmixS4)

Step 2: Get S&P500 data from June 2014 to March 2017

getSymbols("^GSPC", from="2014-06-01", to="2017-03-31")

Step 3: Calculate differenced logarithmic returns using S&P500 EOD Close prices.

sp500_temp = diff(log(Cl(GSPC)))
sp500_returns = as.numeric(sp500_temp)

Step 4: Plot returns from (3) above on plot_ly scatter plot.

plot_ly(x = index(GSPC), y = sp500_returns, type="scatter", mode="lines") %>%

layout(xaxis = list(title="Date/Time (June 2014 to March 2017)"), yaxis = list(title="S&P500 Differenced Logarithmic Returns"))

S&P500 Differenced Logarithmic Returns
(June 2014 to March 2017)

S&P500 Differenced Logarithmic Returns (June 2014 to March 2017)

S&P500 Differenced Logarithmic Returns (June 2014 to March 2017)

Step 5: Fit Hidden Markov Model to S&P500 returns, with three “states”

hidden_markov_model <- depmix(sp500_returns ~ 1, family = gaussian(), nstates = 3, data = data.frame(sp500_returns=sp500_returns))

model_fit <- fit(hidden_markov_model)

Step 6: Calculate posterior probabilities for each of the market states

posterior_probabilities <- posterior(model_fit)

Step 7: Overlay calculated probabilities on S&P500 cumulative returns

sp500_gret = 1 + sp500_returns
sp500_gret <- sp500_gret[-1]
sp500_cret = cumprod(sp500_gret)

plot_ly(name="Unknown", x = index(GSPC), y = posterior_probabilities$S1, type="scatter", mode="lines", line=list(color="grey")) %>%

add_trace(name="Bullish", y = posterior_probabilities$S2, line=list(color="blue")) %>%

add_trace(name="Bearish", y = posterior_probabilities$S3, line=list(color="red")) %>%

add_trace(name="S&P500", y = c(rep(NA,1), sp500_cret-1), line=list(color="black"))

S&P500 Market Regime Probabilities
(June 2014 to March 2017)

S&P500 Hidden Markov Model States (June 2014 to March 2017)

S&P500 Hidden Markov Model States (June 2014 to March 2017)

Interpretation: In any one “market regime”, the corresponding line/curve will “cluster” towards the top of the y-axis (i.e. near a probability of 100%).

For example, during a brief bullish run starting on 01 June 2014, the blue line/curve clustered near y-axis value 1.0. This correlates as you can see, with movement in the S&P500 (black line/curve). The same applies to bearish and “unknown” market states.

An interesting insight one can draw from this graphic, is how the Hidden Markov Model successfully reveals high volatility in the market between June 2014 and March 2015 (constantly changing states between bullish, bearish and unknown).

 

References:
—
[1] Murphy, K.P. (2012) Machine Learning – A Probabilistic Perspective, MIT Press.
https://www.cs.ubc.ca/~murphyk/MLbook/
—

Influences:
—
The honourable Mr. Michael Halls-Moore. QuantStart.com
http://www.quantstart.com/
—

Additional Resource: Learn more about DARWIN Portfolio Risk (VIDEO)
* please activate CC mode to view subtitles.

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

Darwinex – The Open Trader Exchange

DARWIN Filters: A Practical Alternative to Markowitz Portfolio Theory

/0 Comments/in /by

In 1952 [1], the great Harry Markowitz published a paper on portfolio selection that essentially set the stage for modern portfolio theory in a mathematical context.

Harry Markowitz

Harry Markowitz – Nobel Prize Winning Economist

For those not familiar with this Nobel Prize winning economist [2], he devised a methodology whereby investors could mathematically evaluate the proportion of total available capital to allocate, to each constituent asset in a portfolio of assets.

His method was based on just the means and variances of asset returns.

For different choices of capital allocation per asset in a portfolio, different combinations of mean (μ) and variance (σ²) would materialize, collectively referred to as the attainable set.

As investors always want the highest possible return for the lowest possible risk, Markowitz termed all those combinations of μ and σ² where either:

1) σ² was the minimum possible value for a given μ, or

2) μ was the maximum possible value for a given σ²,

.. as the efficient set, or “efficient frontier” as it’s more popularly known.

Harry Markowitz - Efficient Frontier ModelHow did it benefit investors?

Markowitz Portfolio Theory (MPT) stated that investors should select a portfolio from the efficient set, depending on their risk appetite.

However,

The variances of asset returns in a portfolio do not fully explain the risk taken by an investor, and MPT is therefore not entirely applicable in practice.

For instance, MPT does not reveal the Value-at-Risk (VaR), extreme variations in an asset’s risk profile during times of high volatility, nor the Capacity of a given portfolio.

Darwinex’ Solution to Markowitz Portfolio Selection

Years of proprietary R&D at Darwinex, reliably addresses some of the inherent problems in traditional mean-variance portfolio construction & optimization.

All DARWIN (Dynamic Asset & Risk Weighted INvestment) assets listed on The Darwin Exchange are measured in terms of 12 Core Investment Attributes that go far beyond mean and variance.

These are:

  1. Experience
  2. Market Correlation
  3. Risk Stability (in terms of VaR)
  4. Risk Adjustment (in terms of intervention to stabilize VaR)
  5. Open Strategy
  6. Close Strategy
  7. Positive Return Consistency
  8. Negative Return Consistency
  9. Duration Consistency
  10. Loss Aversion
  11. Performance
  12. Capacity

With these robust behavioral analytics, DARWIN investors are able to iteratively filter assets in order to maximize expected returns and minimize standard deviation (risk), with zero mathematical optimization necessary to achieve desired allocations.

In fact, even an equally-weighted portfolio arrived at using DARWIN Filters presents a more statistically robust set of portfolio allocations, than mean-variance optimization where the possibility of overfitting to asset returns is a hidden risk.

DARWIN Filters

Creating custom combinations of the 12 investment attributes allows investors to analyse the behavioral machinery of assets they wish to include in their portfolios.

As Value-at-Risk (VaR), Excursion Analysis (+/- return consistency) and Capacity among others, become integral components of an investor’s selection criteria, the risks presented by traditional MPT (as discussed earlier), are effectively mitigated.

Perhaps the best way to demonstrate the effectiveness of this approach to portfolio construction, is through an example.

EXAMPLE: Real portfolio constructed using DARWIN Filters

A portfolio of 15 highly uncorrelated DARWIN assets (with an impressive Sharpe Ratio) was built using just DARWIN filters and zero mathematical optimization.

For inspiration, here are the actual realised returns of this portfolio between June 2014 and March 2017, both gross and net of performance fees:

DARWIN Portfolio Returns (June 2014 - March 2017)

DARWIN Portfolio Returns (June 2014 – March 2017)

And here is this DARWIN Portfolio’s performance against the S&P500 over the same time period:

DARWIN Portfolio vs. S&P500 (June 2014 - March 2017)

DARWIN Portfolio vs. S&P500 (June 2014 – March 2017)

Steps used in portfolio construction:

1) DARWIN Filters were first created using a combination of the 12 available Investment Attributes (as listed earlier), to define the investment criteria.

Create DARWIN Investment Attribute Filters

Create DARWIN Investment Attribute Filters

This filtered the initial full list of over 1,000 listed DARWIN assets, down to 15.

 

2) Monthly Returns listed publicly on each of the 15 DARWINs’ pages, were then used to construct a Variance-Covariance Matrix.

DARWIN Asset Returns

DARWIN Asset Returns

 

DARWIN Variance-Covariance Matrix

DARWIN Variance-Covariance Matrix

 

3) Assigning equal weights of 6.67% to all 15 assets, Expected Portfolio Returns and Standard Deviation were then duly calculated.

This led to a DARWIN portfolio with the following features:

DARWIN Portfolio Backtest Statistics

DARWIN Portfolio Backtest Statistics

 

 

 

 

4) For sake of exercise, here is a comparison of what MPT optimized allocations would be for the same portfolio:

Equal vs. MPT Optimized Portfolio Weights

Equal vs. MPT Optimized Portfolio Weights

 

 

 

 

 

 

 

 

 

 

 

 

MPT Optimized Portfolio Backtest Statistics

MPT Optimized Portfolio Backtest Statistics

 

 

 

 

An MPT optimized DARWIN portfolio would indeed have lead to a higher Sharpe Ratio (5.92 vs. 4.17), higher Expected Return (38.80% vs 28.61%), and a slightly higher Standard Deviation (5.71% vs. 5.66%)..

.. but at the risk of allocating a majority of available capital to only 4 out of 15 assets in the portfolio.

In this scenario – and as per Markowitz Portfolio Theory – a conservative investor would likely have opted for the equally-weighted portfolio, while a more aggressive investor may have opted for the MPT-optimized portfolio.

In both cases however, DARWIN Filters enabled both profiles of investor to consider the important attributes of Value-at-Risk (VaR), Capacity, Risk Stability and Consistency..

.. as opposed to traditional MPT mean-variance analysis where these would have been overlooked.

—
References

[1] Markowitz, H. (1952) Portfolio Selection. The Journal of Finance, Vol. 7, No. 1, 77-91. March. 1952.
www.jstor.org.proxy.lib.chalmers.se/stable/10.2307/2975974?origin=api (2012-10-30)

[2] The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 1990.
http://www.nobelprize.org/nobel_prizes/economic-sciences/laureates/1990/markowitz-facts.html
—

Watch this video to learn more about a DARWIN’s Investable Attributes:
* please activate CC mode to view subtitles.

Do you have what it takes? – Join the Darwinex Trader Movement!

Darwinex - The Open Trader Exchange

Darwinex – The Open Trader Exchange