An Introduction to Stock Market Data Analysis with Python (FW)

PART I

This post is the first in a two-part series on stock data analysis using Python, based on a lecture I gave on the subject for MATH 3900 (Data Science) at the University of Utah. In these posts, I will discuss basics such as obtaining the data from Yahoo! Finance using *pandas**, visualizing stock data, moving averages, developing a moving-average crossover strategy, backtesting, and benchmarking. The final post will include practice problems. This first post discusses topics up to introducing moving averages.

NOTE: The information in this post is of a general nature containing information and opinions from the author’s perspective. None of the content of this post should be considered financial advice. Furthermore, any code written here is provided without any form of guarantee. Individuals who choose to use it do so at their own risk.

Introduction

Advanced mathematics and statistics has been present in finance for some time. Prior to the 1980s, banking and finance were well known for being “boring”; investment banking was distinct from commercial banking and the primary role of the industry was handling “simple” (at least in comparison to today) financial instruments, such as loans. Deregulation under the Reagan administration, coupled with an influx of mathematical talent, transformed the industry from the “boring” business of banking to what it is today, and since then, finance has joined the other sciences as a motivation for mathematical research and advancement. For example one of the biggest recent achievements of mathematics was the derivation of the Black-Scholes formula, which facilitated the pricing of stock options (a contract giving the holder the right to purchase or sell a stock at a particular price to the issuer of the option). That said, bad statistical models, including the Black-Scholes formula, hold part of the blame for the 2008 financial crisis.

In recent years, computer science has joined advanced mathematics in revolutionizing finance and trading, the practice of buying and selling of financial assets for the purpose of making a profit. In recent years, trading has become dominated by computers; algorithms are responsible for making rapid split-second trading decisions faster than humans could make (so rapidly, the speed at which light travels is a limitation when designing systems). Additionally, machine learning and data mining techniques are growing in popularity in the financial sector, and likely will continue to do so. In fact, a large part of algorithmic trading is high-frequency trading (HFT). While algorithms may outperform humans, the technology is still new and playing in a famously turbulent, high-stakes arena. HFT was responsible for phenomena such as the 2010 flash crash and a 2013 flash crash prompted by a hacked Associated Press tweet about an attack on the White House.

This lecture, however, will not be about how to crash the stock market with bad mathematical models or trading algorithms. Instead, I intend to provide you with basic tools for handling and analyzing stock market data with Python. I will also discuss moving averages, how to construct trading strategies using moving averages, how to formulate exit strategies upon entering a position, and how to evaluate a strategy with backtesting.

DISCLAIMER: THIS IS NOT FINANCIAL ADVICE!!! Furthermore, I have ZERO experience as a trader (a lot of this knowledge comes from a one-semester course on stock trading I took at Salt Lake Community College)! This is purely introductory knowledge, not enough to make a living trading stocks. People can and do lose money trading stocks, and you do so at your own risk!

Getting and Visualizing Stock Data

Getting Data from Yahoo! Finance with pandas

Before we play with stock data, we need to get it in some workable format. Stock data can be obtained from Yahoo! Finance, Google Finance, or a number of other sources, and the pandas package provides easy access to Yahoo! Finance and Google Finance data, along with other sources. In this lecture, we will get our data from Yahoo! Finance.

The following code demonstrates how to create directly a DataFrame object containing stock information. (You can read more about remote data accesshere.)

Open High Low Close Volume Adj Close
Date
2016-01-04 102.610001 105.370003 102.000000 105.349998 67649400 103.586180
2016-01-05 105.750000 105.849998 102.410004 102.709999 55791000 100.990380
2016-01-06 100.559998 102.370003 99.870003 100.699997 68457400 99.014030
2016-01-07 98.680000 100.129997 96.430000 96.449997 81094400 94.835186
2016-01-08 98.550003 99.110001 96.760002 96.959999 70798000 95.336649

Let’s briefly discuss this. Open is the price of the stock at the beginning of the trading day (it need not be the closing price of the previous trading day),high is the highest price of the stock on that trading day, low the lowest price of the stock on that trading day, and close the price of the stock at closing time. Volume indicates how many stocks were traded. Adjusted close is the closing price of the stock that adjusts the price of the stock for corporate actions. While stock prices are considered to be set mostly by traders, stock splits (when the company makes each extant stock worth two and halves the price) and dividends (payout of company profits per share) also affect the price of a stock and should be accounted for.

Visualizing Stock Data

Now that we have stock data we would like to visualize it. I first demonstrate how to do so using the matplotlib package. Notice that the appleDataFrame object has a convenience method, plot(), which makes creating plots easier.

A linechart is fine, but there are at least four variables involved for each date (open, high, low, and close), and we would like to have some visual way to see all four variables that does not require plotting four separate lines. Financial data is often plotted with a Japanese candlestick plot, so named because it was first created by 18th century Japanese rice traders. Such a chart can be created with matplotlib, though it requires considerable effort.

I have made a function you are welcome to use to more easily create candlestick charts from pandas data frames, and use it to plot our stock data. (Code is based off this example, and you can read the documentation for the functions involved here.)

With a candlestick chart, a black candlestick indicates a day where the closing price was higher than the open (a gain), while a red candlestick indicates a day where the open was higher than the close (a loss). The wicks indicate the high and the low, and the body the open and close (hue is used to determine which end of the body is the open and which the close). Candlestick charts are popular in finance and some strategies in technical analysis use them to make trading decisions, depending on the shape, color, and position of the candles. I will not cover such strategies today.

We may wish to plot multiple financial instruments together; we may want to compare stocks, compare them to the market, or look at other securities such as exchange-traded funds (ETFs). Later, we will also want to see how to plot a financial instrument against some indicator, like a moving average. For this you would rather use a line chart than a candlestick chart. (How would you plot multiple candlestick charts on top of one another without cluttering the chart?)

Below, I get stock data for some other tech companies and plot their adjusted close together.

AAPL GOOG MSFT
Date
2016-01-04 103.586180 741.840027 53.696756
2016-01-05 100.990380 742.580017 53.941723
2016-01-06 99.014030 743.619995 52.961855
2016-01-07 94.835186 726.390015 51.119702
2016-01-08 95.336649 714.469971 51.276485

What’s wrong with this chart? While absolute price is important (pricy stocks are difficult to purchase, which affects not only their volatility but your ability to trade that stock), when trading, we are more concerned about the relative change of an asset rather than its absolute price. Google’s stocks are much more expensive than Apple’s or Microsoft’s, and this difference makes Apple’s and Microsoft’s stocks appear much less volatile than they truly are.

One solution would be to use two different scales when plotting the data; one scale will be used by Apple and Microsoft stocks, and the other by Google.

A “better” solution, though, would be to plot the information we actually want: the stock’s returns. This involves transforming the data into something more useful for our purposes. There are multiple transformations we could apply.

One transformation would be to consider the stock’s return since the beginning of the period of interest. In other words, we plot:

$\text{return}_{t,0} = \frac{\text{price}_t}{\text{price}_0}$

This will require transforming the data in the stocks object, which I do next.

AAPL GOOG MSFT
Date
2016-01-04 1.000000 1.000000 1.000000
2016-01-05 0.974941 1.000998 1.004562
2016-01-06 0.955861 1.002399 0.986314
2016-01-07 0.915520 0.979173 0.952007
2016-01-08 0.920361 0.963105 0.954927

This is a much more useful plot. We can now see how profitable each stock was since the beginning of the period. Furthermore, we see that these stocks are highly correlated; they generally move in the same direction, a fact that was difficult to see in the other charts.

Alternatively, we could plot the change of each stock per day. One way to do so would be to plot the percentage increase of a stock when comparing day $t$ to day $t + 1$, with the formula:

$\text{growth}_t = \frac{\text{price}_{t + 1} - \text{price}_t}{\text{price}_t}$

But change could be thought of differently as:

$\text{increase}_t = \frac{\text{price}_{t} - \text{price}_{t-1}}{\text{price}_t}$

These formulas are not the same and can lead to differing conclusions, but there is another way to model the growth of a stock: with log differences.

$\text{change}_t = \log(\text{price}_{t}) - \log(\text{price}_{t - 1})$

(Here, $\log$ is the natural log, and our definition does not depend as strongly on whether we use $\log(\text{price}_{t}) - \log(\text{price}_{t - 1})$ or $\log(\text{price}_{t+1}) - \log(\text{price}_{t})$.) The advantage of using log differences is that this difference can be interpreted as the percentage change in a stock but does not depend on the denominator of a fraction.

We can obtain and plot the log differences of the data in stocks as follows:

AAPL GOOG MSFT
Date
2016-01-04 NaN NaN NaN
2016-01-05 -0.025379 0.000997 0.004552
2016-01-06 -0.019764 0.001400 -0.018332
2016-01-07 -0.043121 -0.023443 -0.035402
2016-01-08 0.005274 -0.016546 0.003062

Which transformation do you prefer? Looking at returns since the beginning of the period make the overall trend of the securities in question much more apparent. Changes between days, though, are what more advanced methods actually consider when modelling the behavior of a stock. so they should not be ignored.

Moving Averages

Charts are very useful. In fact, some traders base their strategies almost entirely off charts (these are the “technicians”, since trading strategies based off finding patterns in charts is a part of the trading doctrine known astechnical analysis). Let’s now consider how we can find trends in stocks.

A $q$-day moving average is, for a series $x_t$ and a point in time $t$, the average of the past $q$ days: that is, if $MA^q_t$ denotes a moving average process, then:

$MA^q_t = \frac{1}{q} \sum_{i = 0}^{q-1} x_{t - i}$latex

Moving averages smooth a series and helps identify trends. The larger $q$ is, the less responsive a moving average process is to short-term fluctuations in the series $x_t$. The idea is that moving average processes help identify trends from “noise”. Fast moving averages have smaller $q$ and more closely follow the stock, while slow moving averages have larger $q$, resulting in them responding less to the fluctuations of the stock and being more stable.

pandas provides functionality for easily computing moving averages. I demonstrate its use by creating a 20-day (one month) moving average for the Apple data, and plotting it alongside the stock.

Notice how late the rolling average begins. It cannot be computed until 20 days have passed. This limitation becomes more severe for longer moving averages. Because I would like to be able to compute 200-day moving averages, I’m going to extend out how much AAPL data we have. That said, we will still largely focus on 2016.

You will notice that a moving average is much smoother than the actua stock data. Additionally, it’s a stubborn indicator; a stock needs to be above or below the moving average line in order for the line to change direction. Thus, crossing a moving average signals a possible change in trend, and should draw attention.

Traders are usually interested in multiple moving averages, such as the 20-day, 50-day, and 200-day moving averages. It’s easy to examine multiple moving averages at once.

The 20-day moving average is the most sensitive to local changes, and the 200-day moving average the least. Here, the 200-day moving average indicates an overall bearish trend: the stock is trending downward over time. The 20-day moving average is at times bearish and at other timesbullish, where a positive swing is expected. You can also see that the crossing of moving average lines indicate changes in trend. These crossings are what we can use as trading signals, or indications that a financial security is changing direction and a profitable trade might be made.

Visit next week to read about how to design and test a trading strategy using moving averages.

Update: An earlier version of this article suggested that algorithmic trading was synonymous as high-frequency trading. As pointed out in the comments by dissolved, this need not be the case; algorithms can be used to identify trades without necessarily being high frequency. While HFT is a large subset of algorithmic trading, it is not equal to it.

PART II

This post is the second in a two-part series on stock data analysis using Python, based on a lecture I gave on the subject for MATH 3900 (Data Mining) at the University of Utah (read part 1 here). In these posts, I will discuss basics such as obtaining the data from Yahoo! Finance using*pandas, visualizing stock data, moving averages, developing a moving-average crossover strategy, backtesting, and benchmarking. This second post discusses topics including divising a moving average crossover strategy, backtesting, and benchmarking, along with practice problems for readers to ponder.

NOTE: The information in this post is of a general nature containing information and opinions from the author’s perspective. None of the content of this post should be considered financial advice. Furthermore, any code written here is provided without any form of guarantee. Individuals who choose to use it do so at their own risk.

Call an open position a trade that will be terminated in the future when a condition is met. A long position is one in which a profit is made if the financial instrument traded increases in value, and a short position is on in which a profit is made if the financial asset being traded decreases in value. When trading stocks directly, all long positions are bullish and all short position are bearish. That said, a bullish attitude need not be accompanied by a long position, and a bearish attitude need not be accompanied by a short position (this is particularly true when trading stock options).

Here is an example. Let’s say you buy a stock with the expectation that the stock will increase in value, with a plan to sell the stock at a higher price. This is a long position: you are holding a financial asset for which you will profit if the asset increases in value. Your potential profit is unlimited, and your potential losses are limited by the price of the stock since stock prices never go below zero. On the other hand, if you expect a stock to decrease in value, you may borrow the stock from a brokerage firm and sell it, with the expectation of buying the stock back later at a lower price, thus earning you a profit. This is called shorting a stock, and is a short position, since you will earn a profit if the stock drops in value. The potential profit from shorting a stock is limited by the price of the stock (the best you can do is have the stock become worth nothing; you buy it back for free), while the losses are unlimited, since you could potentially spend an arbitrarily large amount of money to buy the stock back. Thus, a broker will expect an investor to be in a very good financial position before allowing the investor to short a stock.

We will call a plan that includes trading signals for prompting trades, a rule for deciding how much of the portfolio to risk on any particular strategy, and a complete exit strategy for any trade an overall trading strategy. Our concern now is to design and evaluate trading strategies.

We will suppose that the amount of money in the portfolio involved in any particular trade is a fixed proportion; 10% seems like a good number. We will also say that for any trade, if losses exceed 20% of the value of the trade, we will exit the position. Now we need a means for deciding when to enter position and when to exit for a profit.

Here, I will be demonstrating a moving average crossover strategy. We will use two moving averages, one we consider “fast”, and the other “slow”. The strategy is:

• Trade the asset when the fast moving average crosses over the slow moving average.
• Exit the trade when the fast moving average crosses over the slow moving average again.

A long trade will be prompted when the fast moving average crosses from below to above the slow moving average, and the trade will be exited when the fast moving average crosses below the slow moving average later. A short trade will be prompted when the fast moving average crosses below the slow moving average, and the trade will be exited when the fast moving average later crosses above the slow moving average.

We now have a complete strategy. But before we decide we want to use it, we should try to evaluate the quality of the strategy first. The usual means for doing so is backtesting, which is looking at how profitable the strategy is on historical data. For example, looking at the above chart’s performance on Apple stock, if the 20-day moving average is the fast moving average and the 50-day moving average the slow, this strategy does not appear to be very profitable, at least not if you are always taking long positions.

Let’s see if we can automate the backtesting task. We first identify when the 20-day average is below the 50-day average, and vice versa.

Open High Low Close Volume Adj Close 20d 50d 200d 20d-50d
Date
2016-08-26 107.410004 107.949997 106.309998 106.940002 27766300 106.940002 107.87 101.51 102.73 6.36
2016-08-29 106.620003 107.440002 106.290001 106.820000 24970300 106.820000 107.91 101.74 102.68 6.17
2016-08-30 105.800003 106.500000 105.500000 106.000000 24863900 106.000000 107.98 101.96 102.63 6.02
2016-08-31 105.660004 106.570000 105.639999 106.099998 29662400 106.099998 108.00 102.16 102.60 5.84
2016-09-01 106.139999 106.800003 105.620003 106.730003 26643600 106.730003 108.04 102.39 102.56 5.65

We will refer to the sign of this difference as the regime; that is, if the fast moving average is above the slow moving average, this is a bullish regime (the bulls rule), and a bearish regime (the bears rule) holds when the fast moving average is below the slow moving average. I identify regimes with the following code.

The last line above indicates that for 1005 days the market was bearish on Apple, while for 600 days the market was bullish, and it was neutral for 54 days.

Trading signals appear at regime changes. When a bullish regime begins, a buy signal is triggered, and when it ends, a sell signal is triggered. Likewise, when a bearish regime begins, a sell signal is triggered, and when the regime ends, a buy signal is triggered (this is of interest only if you ever will short the stock, or use some derivative like a stock option to bet against the market).

It’s simple to obtain signals. Let $r_t$ indicate the regime at time $t$, and $s_t$ the signal at time $t$. Then:

$s_t = \text{sign}(r_t - r_{t - 1})$

$s_t \in \{-1, 0, 1\}$, with $-1$ indicating “sell”, $1$ indicating “buy”, and $0$ no action. We can obtain signals like so:

Open High Low Close Volume Adj Close 20d 50d 200d 20d-50d Regime Signal
Date
2016-08-26 107.410004 107.949997 106.309998 106.940002 27766300 106.940002 107.87 101.51 102.73 6.36 1.0 0.0
2016-08-29 106.620003 107.440002 106.290001 106.820000 24970300 106.820000 107.91 101.74 102.68 6.17 1.0 0.0
2016-08-30 105.800003 106.500000 105.500000 106.000000 24863900 106.000000 107.98 101.96 102.63 6.02 1.0 0.0
2016-08-31 105.660004 106.570000 105.639999 106.099998 29662400 106.099998 108.00 102.16 102.60 5.84 1.0 0.0
2016-09-01 106.139999 106.800003 105.620003 106.730003 26643600 106.730003 108.04 102.39 102.56 5.65 1.0 -1.0

We would buy Apple stock 23 times and sell Apple stock 23 times. If we only go long on Apple stock, only 23 trades will be engaged in over the 6-year period, while if we pivot from a long to a short position every time a long position is terminated, we would engage in 23 trades total. (Bear in mind that trading more frequently isn’t necessarily good; trades are never free.)

You may notice that the system as it currently stands isn’t very robust, since even a fleeting moment when the fast moving average is above the slow moving average triggers a trade, resulting in trades that end immediately (which is bad if not simply because realistically every trade is accompanied by a fee that can quickly erode earnings). Additionally, every bullish regime immediately transitions into a bearish regime, and if you were constructing trading systems that allow both bullish and bearish bets, this would lead to the end of one trade immediately triggering a new trade that bets on the market in the opposite direction, which again seems finnicky. A better system would require more evidence that the market is moving in some particular direction. But we will not concern ourselves with these details for now.

Let’s now try to identify what the prices of the stock is at every buy and every sell.

Price Regime Signal
Date
2010-06-11 253.509995 -1.0 Sell
2010-07-22 259.020000 -1.0 Sell
2011-03-30 348.630009 0.0 Sell
2011-03-31 348.510006 -1.0 Sell
2011-05-27 337.409992 -1.0 Sell
2011-11-17 377.410000 -1.0 Sell
2012-05-09 569.180023 -1.0 Sell
2012-10-17 644.610001 -1.0 Sell
2013-06-26 398.069992 -1.0 Sell
2013-10-03 483.409996 -1.0 Sell
2014-01-28 506.499977 -1.0 Sell
2014-04-22 531.700020 -1.0 Sell
2014-06-11 93.860001 -1.0 Sell
2014-10-17 97.669998 -1.0 Sell
2015-01-05 106.250000 -1.0 Sell
2015-04-16 126.169998 -1.0 Sell
2015-06-25 127.500000 -1.0 Sell
2015-12-18 106.029999 -1.0 Sell
2016-05-05 93.239998 -1.0 Sell
2016-07-08 96.680000 -1.0 Sell
2016-09-01 106.730003 1.0 Sell
End Date Price Profit
Date
2010-03-16 2010-06-11 224.449997 29.059998
2010-06-18 2010-07-22 274.070011 -15.050011
2010-09-20 2011-03-30 283.230007 65.400002
2011-05-12 2011-05-27 346.569988 -9.159996
2011-07-14 2011-11-17 357.770004 19.639996
2011-12-28 2012-05-09 402.640003 166.540020
2012-06-25 2012-10-17 570.770020 73.839981
2013-05-17 2013-06-26 433.260010 -35.190018
2013-07-31 2013-10-03 452.529984 30.880012
2013-10-16 2014-01-28 501.110001 5.389976
2014-03-26 2014-04-22 539.779991 -8.079971
2014-04-25 2014-06-11 571.939980 -478.079979
2014-08-18 2014-10-17 99.160004 -1.490006
2014-10-28 2015-01-05 106.739998 -0.489998
2015-02-05 2015-04-16 119.940002 6.229996
2015-04-28 2015-06-25 130.559998 -3.059998
2015-10-27 2015-12-18 114.550003 -8.520004
2016-03-11 2016-05-05 102.260002 -9.020004
2016-07-01 2016-07-08 95.889999 0.790001
2016-07-25 2016-09-01 97.339996 9.390007

Above, we can see that on May 17th, 2013, there was a massive drop in the price of Apple stock, and it looks like our trading system would do badly. But this price drop is not because of a massive shock to Apple, but simply due to a stock split. And while dividend payments are not as obvious as a stock split, they may be affecting the performance of our system.

We don’t want our trading system to be behaving poorly because of stock splits and dividend payments. How should we handle this? One approach would be to obtain historical stock split and dividend payment data and design a trading system for handling these. This would most realistically represent the behavior of the stock and could be considered the best solution, but it is more complicated. Another solution would be to adjust the prices to account for stock splits and dividend payments.

Yahoo! Finance only provides the adjusted closing price of a stock, but this is all we need to get adjusted opening, high, and low prices. The adjusted close is computed like so:

$\text{price}^{\text{adj}}_t = m_t \times \text{price}_t$

where $m_t$ is the multiplier used for the adjustment. Solving for $m_t$ requires only division and thus we can use the closing price and the adjusted closing price to adjust all prices in the series.

End Date Price Profit
Date
2010-03-16 2010-06-10 29.355667 3.408371
2010-06-18 2010-07-22 35.845436 -1.968381
2010-09-20 2011-03-30 37.043466 8.553623
2011-05-12 2011-05-27 45.327660 -1.198030
2011-07-14 2011-11-17 46.792503 2.568702
2011-12-28 2012-05-09 52.661020 21.781659
2012-06-25 2012-10-17 74.650634 10.019459
2013-05-17 2013-06-26 57.882798 -4.701326
2013-07-31 2013-10-04 60.457234 4.500835
2013-10-16 2014-01-28 67.389473 1.122523
2014-03-11 2014-03-17 72.948554 -1.272298
2014-03-24 2014-04-22 73.370393 -1.019203
2014-04-25 2014-10-17 77.826851 16.191371
2014-10-28 2015-01-05 102.749105 -0.028185
2015-02-05 2015-04-16 116.413846 6.046838
2015-04-28 2015-06-26 126.721620 -3.184117
2015-10-27 2015-12-18 112.152083 -7.897288
2016-03-10 2016-05-05 100.015950 -7.278331
2016-06-23 2016-06-27 95.582210 -4.038123
2016-06-30 2016-07-11 95.084904 1.372569
2016-07-25 2016-09-01 96.815526 9.914477

As you can see, adjusting for dividends and stock splits makes a big difference. We will use this data from now on.

Let’s now create a simulated portfolio of \$1,000,000, and see how it would behave, according to the rules we have established. This includes:

• Investing only 10% of the portfolio in any trade
• Exiting the position if losses exceed 20% of the value of the trade.

When simulating, bear in mind that:

• Trades are done in batches of 100 stocks.
• Our stop-loss rule involves placing an order to sell the stock the moment the price drops below the specified level. Thus we need to check whether the lows during this period ever go low enough to trigger the stop-loss. Realistically, unless we buy a put option, we cannot guarantee that we will sell the stock at the price we set at the stop-loss, but we will use this as the selling price anyway for the sake of simplicity.
• Every trade is accompanied by a commission to the broker, which should be accounted for. I do not do so here.

Here’s how a backtest may look:

End Date Price Profit Low
Date
2010-03-16 2010-06-10 29.355667 3.408371 26.059775
2010-06-18 2010-07-22 35.845436 -1.968381 31.337127
2010-09-20 2011-03-30 37.043466 8.553623 35.967068
2011-05-12 2011-05-27 45.327660 -1.198030 43.084626
2011-07-14 2011-11-17 46.792503 2.568702 46.171251
2011-12-28 2012-05-09 52.661020 21.781659 52.382438
2012-06-25 2012-10-17 74.650634 10.019459 73.975759
2013-05-17 2013-06-26 57.882798 -4.701326 52.859502
2013-07-31 2013-10-04 60.457234 4.500835 60.043080
2013-10-16 2014-01-28 67.389473 1.122523 67.136651
2014-03-11 2014-03-17 72.948554 -1.272298 71.167335
2014-03-24 2014-04-22 73.370393 -1.019203 69.579335
2014-04-25 2014-10-17 77.826851 16.191371 76.740971
2014-10-28 2015-01-05 102.749105 -0.028185 101.411076
2015-02-05 2015-04-16 116.413846 6.046838 114.948237
2015-04-28 2015-06-26 126.721620 -3.184117 119.733299
2015-10-27 2015-12-18 112.152083 -7.897288 104.038477
2016-03-10 2016-05-05 100.015950 -7.278331 91.345994
2016-06-23 2016-06-27 95.582210 -4.038123 91.006996
2016-06-30 2016-07-11 95.084904 1.372569 93.791913
2016-07-25 2016-09-01 96.815526 9.914477 95.900485
End Date End Port. Value Profit per Share Share Price Shares Start Port. Value Stop-Loss Triggered Total Profit Trade Value
2010-03-16 2010-06-10 1.011588e+06 3.408371 29.355667 3400.0 1.000000e+06 0.0 11588.4614 99809.2678
2010-06-18 2010-07-22 1.006077e+06 -1.968381 35.845436 2800.0 1.011588e+06 0.0 -5511.4668 100367.2208
2010-09-20 2011-03-30 1.029172e+06 8.553623 37.043466 2700.0 1.006077e+06 0.0 23094.7821 100017.3582
2011-05-12 2011-05-27 1.026536e+06 -1.198030 45.327660 2200.0 1.029172e+06 0.0 -2635.6660 99720.8520
2011-07-14 2011-11-17 1.031930e+06 2.568702 46.792503 2100.0 1.026536e+06 0.0 5394.2742 98264.2563
2011-12-28 2012-05-09 1.073316e+06 21.781659 52.661020 1900.0 1.031930e+06 0.0 41385.1521 100055.9380
2012-06-25 2012-10-17 1.087343e+06 10.019459 74.650634 1400.0 1.073316e+06 0.0 14027.2426 104510.8876
2013-05-17 2013-06-26 1.078880e+06 -4.701326 57.882798 1800.0 1.087343e+06 0.0 -8462.3868 104189.0364
2013-07-31 2013-10-04 1.086532e+06 4.500835 60.457234 1700.0 1.078880e+06 0.0 7651.4195 102777.2978
2013-10-16 2014-01-28 1.088328e+06 1.122523 67.389473 1600.0 1.086532e+06 0.0 1796.0368 107823.1568
2014-03-11 2014-03-17 1.086547e+06 -1.272298 72.948554 1400.0 1.088328e+06 0.0 -1781.2172 102127.9756
2014-03-24 2014-04-22 1.085120e+06 -1.019203 73.370393 1400.0 1.086547e+06 0.0 -1426.8842 102718.5502
2014-04-25 2014-10-17 1.106169e+06 16.191371 77.826851 1300.0 1.085120e+06 0.0 21048.7823 101174.9063
2014-10-28 2015-01-05 1.106140e+06 -0.028185 102.749105 1000.0 1.106169e+06 0.0 -28.1850 102749.1050
2015-02-05 2015-04-16 1.111582e+06 6.046838 116.413846 900.0 1.106140e+06 0.0 5442.1542 104772.4614
2015-04-28 2015-06-26 1.109035e+06 -3.184117 126.721620 800.0 1.111582e+06 0.0 -2547.2936 101377.2960
2015-10-27 2015-12-18 1.101928e+06 -7.897288 112.152083 900.0 1.109035e+06 0.0 -7107.5592 100936.8747
2016-03-10 2016-05-05 1.093921e+06 -7.278331 100.015950 1100.0 1.101928e+06 0.0 -8006.1641 110017.5450
2016-06-23 2016-06-27 1.089480e+06 -4.038123 95.582210 1100.0 1.093921e+06 0.0 -4441.9353 105140.4310
2016-06-30 2016-07-11 1.090989e+06 1.372569 95.084904 1100.0 1.089480e+06 0.0 1509.8259 104593.3944
2016-07-25 2016-09-01 1.101895e+06 9.914477 96.815526 1100.0 1.090989e+06 0.0 10905.9247 106497.0786

Our portfolio’s value grew by 10% in about six years. Considering that only 10% of the portfolio was ever involved in any single trade, this is not bad performance.

Notice that this strategy never lead to our stop-loss order being triggered. Does this mean we don’t need stop-loss orders? There is no simple answer to this. After all, if we had chosen a different level at which a stop-loss would be triggered, we may have seen it triggered.

I have provided links both for and “against” the use of stop-loss orders, but from now on I’m not going to require our backtesting system to account for them. While less realistic (and I do believe an industrial-strength system should account for a stop-loss rule), this simplifies the backtesting task.

A more realistic portfolio would not be betting 10% of its value on only one stock. A more realistic one would consider investing in multiple stocks. Multiple trades may be ongoing at any given time involving multiple companies, and most of the portfolio will be in stocks, not cash. Now that we will be investing in multiple stops and exiting only when moving averages cross (not because of a stop-loss), we will need to change our approach to backtesting. For example, we will be using one pandasDataFrame to contain all buy and sell orders for all stocks being considered, and our loop above will have to track more information.

I have written functions for creating order data for multiple stocks, and a function for performing the backtesting.

Price Regime Signal
Date Symbol
GOOG 282.318173 -1.0 Sell
MSFT 24.677580 -1.0 Sell
SNY 16.360001 -1.0 Sell
YHOO 16.360001 -1.0 Sell
2010-05-03 GOOG 265.035411 -1.0 Sell
2010-05-10 HPQ 19.435830 -1.0 Sell
2010-05-14 NTDOY 35.799999 -1.0 Sell
2010-05-17 SNY 16.270000 -1.0 Sell
YHOO 16.270000 -1.0 Sell
2010-05-19 AMZN 124.589996 -1.0 Sell
MSFT 23.835187 -1.0 Sell
2010-05-21 IBM 108.322991 -1.0 Sell
2010-06-10 AAPL 32.764038 0.0 Sell
2010-06-11 AAPL 33.156405 -1.0 Sell
2010-07-01 IBM 105.861499 -1.0 Sell
2010-07-20 IBM 109.298956 -1.0 Sell
2016-06-27 AAPL 91.544087 -1.0 Sell
FB 108.970001 -1.0 Sell
2016-06-28 SNY 36.040001 -1.0 Sell
YHOO 36.040001 -1.0 Sell
NFLX 91.480003 0.0 Sell
2016-07-01 NFLX 96.669998 -1.0 Sell
2016-07-11 AAPL 96.457473 -1.0 Sell
2016-09-01 AAPL 106.730003 1.0 Sell
2016-09-02 AMZN 772.440002 1.0 Sell
FB 126.510002 1.0 Sell
GOOG 771.460022 1.0 Sell
HPQ 14.490000 1.0 Sell
IBM 159.550003 1.0 Sell
MSFT 57.669998 1.0 Sell
NFLX 97.379997 1.0 Sell
NTDOY 28.840000 1.0 Sell
SNY 43.279999 1.0 Sell
TWTR 19.549999 1.0 Sell
YHOO 43.279999 1.0 Sell

475 rows × 3 columns

End Cash Portfolio Value Profit per Share Share Price Shares Start Cash Total Profit Trade Value Type
Date Symbol
2010-03-16 AAPL 9.001907e+05 1.000000e+06 0.000000 29.355667 3400.0 1.000000e+06 0.0 99809.2678 Buy
AMZN 8.079377e+05 1.000000e+06 0.000000 131.789993 700.0 9.001907e+05 0.0 92252.9951 Buy
GOOG 8.079377e+05 1.000000e+06 0.000000 282.318173 0.0 8.079377e+05 0.0 0.0000 Sell
HPQ 7.084706e+05 1.000000e+06 0.000000 20.722316 4800.0 8.079377e+05 0.0 99467.1168 Buy
IBM 6.089637e+05 1.000000e+06 0.000000 110.563240 900.0 7.084706e+05 0.0 99506.9160 Buy
MSFT 6.089637e+05 1.000000e+06 0.000000 24.677580 0.0 6.089637e+05 0.0 0.0000 Sell
NFLX 5.090727e+05 1.000000e+06 0.000000 10.090000 9900.0 6.089637e+05 0.0 99891.0000 Buy
NTDOY 4.126127e+05 1.000000e+06 0.000000 37.099998 2600.0 5.090727e+05 0.0 96459.9948 Buy
SNY 4.126127e+05 1.000000e+06 0.000000 16.360001 0.0 4.126127e+05 0.0 0.0000 Sell
YHOO 4.126127e+05 1.000000e+06 0.000000 16.360001 0.0 4.126127e+05 0.0 0.0000 Sell
2010-03-17 SNY 3.136127e+05 1.000000e+06 0.000000 16.500000 6000.0 4.126127e+05 0.0 99000.0000 Buy
YHOO 2.146127e+05 1.000000e+06 0.000000 16.500000 6000.0 3.136127e+05 0.0 99000.0000 Buy
2010-03-22 GOOG 1.310711e+05 1.000000e+06 0.000000 278.472004 300.0 2.146127e+05 0.0 83541.6012 Buy
2010-03-23 MSFT 3.315733e+04 1.000000e+06 0.000000 25.106096 3900.0 1.310711e+05 0.0 97913.7744 Buy
2010-05-03 GOOG 1.126680e+05 9.959690e+05 -13.436593 265.035411 0.0 3.315733e+04 -0.0 79510.6233 Sell
2010-05-10 HPQ 2.059599e+05 9.897939e+05 -1.286486 19.435830 0.0 1.126680e+05 -0.0 93291.9840 Sell
2010-05-14 NTDOY 2.990399e+05 9.864139e+05 -1.299999 35.799999 0.0 2.059599e+05 -0.0 93079.9974 Sell
2010-05-17 SNY 3.966599e+05 9.850339e+05 -0.230000 16.270000 0.0 2.990399e+05 -0.0 97620.0000 Sell
YHOO 4.942799e+05 9.836539e+05 -0.230000 16.270000 0.0 3.966599e+05 -0.0 97620.0000 Sell
2010-05-19 AMZN 5.814929e+05 9.786139e+05 -7.199997 124.589996 0.0 4.942799e+05 -0.0 87212.9972 Sell
MSFT 6.744502e+05 9.736573e+05 -1.270909 23.835187 0.0 5.814929e+05 -0.0 92957.2293 Sell
2010-05-21 IBM 7.719409e+05 9.716411e+05 -2.240249 108.322991 0.0 6.744502e+05 -0.0 97490.6919 Sell
2010-06-10 AAPL 8.833386e+05 9.832296e+05 3.408371 32.764038 0.0 7.719409e+05 0.0 111397.7292 Sell
2010-06-11 AAPL 8.833386e+05 9.832296e+05 3.800738 33.156405 0.0 8.833386e+05 0.0 0.0000 Sell
2010-06-18 AAPL 7.865559e+05 9.832296e+05 0.000000 35.845436 2700.0 8.833386e+05 0.0 96782.6772 Buy
2010-06-28 IBM 6.974378e+05 9.832296e+05 0.000000 111.397697 800.0 7.865559e+05 0.0 89118.1576 Buy
2010-07-01 IBM 7.821270e+05 9.788006e+05 -5.536198 105.861499 0.0 6.974378e+05 -0.0 84689.1992 Sell
2010-07-06 IBM 6.861598e+05 9.788006e+05 0.000000 106.630175 900.0 7.821270e+05 0.0 95967.1575 Buy
2010-07-09 NTDOY 5.900898e+05 9.788006e+05 0.000000 36.950001 2600.0 6.861598e+05 0.0 96070.0026 Buy
2010-07-20 IBM 6.884589e+05 9.812025e+05 2.668781 109.298956 0.0 5.900898e+05 0.0 98369.0604 Sell
2016-06-23 AAPL 3.951693e+05 1.863808e+06 0.000000 95.582210 1900.0 5.767755e+05 0.0 181606.1990 Buy
TWTR 2.094333e+05 1.863808e+06 0.000000 17.040001 10900.0 3.951693e+05 0.0 185736.0109 Buy
2016-06-27 AAPL 3.833670e+05 1.856135e+06 -4.038123 91.544087 0.0 2.094333e+05 -0.0 173933.7653 Sell
FB 5.795130e+05 1.862921e+06 3.770004 108.970001 0.0 3.833670e+05 0.0 196146.0018 Sell
2016-06-28 SNY 7.885450e+05 1.880959e+06 3.110001 36.040001 0.0 5.795130e+05 0.0 209032.0058 Sell
YHOO 9.975770e+05 1.898997e+06 3.110001 36.040001 0.0 7.885450e+05 0.0 209032.0058 Sell
2016-06-30 AAPL 8.169157e+05 1.898997e+06 0.000000 95.084904 1900.0 9.975770e+05 0.0 180661.3176 Buy
NFLX 9.907277e+05 1.893981e+06 -2.640000 91.480003 0.0 8.169157e+05 -0.0 173812.0057 Sell
2016-07-01 NFLX 9.907277e+05 1.893981e+06 2.549995 96.669998 0.0 9.907277e+05 0.0 0.0000 Sell
SNY 8.045767e+05 1.893981e+06 0.000000 37.990002 4900.0 9.907277e+05 0.0 186151.0098 Buy
YHOO 6.184257e+05 1.893981e+06 0.000000 37.990002 4900.0 8.045767e+05 0.0 186151.0098 Buy
2016-07-11 AAPL 8.016949e+05 1.896589e+06 1.372569 96.457473 0.0 6.184257e+05 0.0 183269.1987 Sell
NTDOY 6.133349e+05 1.896589e+06 0.000000 27.700001 6800.0 8.016949e+05 0.0 188360.0068 Buy
2016-07-14 MSFT 4.264099e+05 1.896589e+06 0.000000 53.407133 3500.0 6.133349e+05 0.0 186924.9655 Buy
2016-07-25 AAPL 2.424604e+05 1.896589e+06 0.000000 96.815526 1900.0 4.264099e+05 0.0 183949.4994 Buy
FB 6.001543e+04 1.896589e+06 0.000000 121.629997 1500.0 2.424604e+05 0.0 182444.9955 Buy
2016-07-26 GOOG -8.766857e+04 1.896589e+06 0.000000 738.419983 200.0 6.001543e+04 0.0 147683.9966 Buy
2016-08-18 NFLX -2.703726e+05 1.896589e+06 0.000000 96.160004 1900.0 -8.766857e+04 0.0 182704.0076 Buy
2016-09-01 AAPL -6.758557e+04 1.915427e+06 9.914477 106.730003 0.0 -2.703726e+05 0.0 202787.0057 Sell
2016-09-02 AMZN 1.641464e+05 1.979327e+06 213.000000 772.440002 0.0 -6.758557e+04 0.0 231732.0006 Sell
FB 3.539114e+05 1.986647e+06 4.880005 126.510002 0.0 1.641464e+05 0.0 189765.0030 Sell
GOOG 5.082034e+05 1.993255e+06 33.040039 771.460022 0.0 3.539114e+05 0.0 154292.0044 Sell
HPQ 7.081654e+05 2.006030e+06 0.925746 14.490000 0.0 5.082034e+05 0.0 199962.0000 Sell
IBM 8.996254e+05 2.015652e+06 8.018727 159.550003 0.0 7.081654e+05 0.0 191460.0036 Sell
MSFT 1.101470e+06 2.030572e+06 4.262865 57.669998 0.0 8.996254e+05 0.0 201844.9930 Sell
NFLX 1.286492e+06 2.032890e+06 1.219993 97.379997 0.0 1.101470e+06 0.0 185021.9943 Sell
NTDOY 1.482604e+06 2.040642e+06 1.139999 28.840000 0.0 1.286492e+06 0.0 196112.0000 Sell
SNY 1.694676e+06 2.066563e+06 5.289997 43.279999 0.0 1.482604e+06 0.0 212071.9951 Sell
TWTR 1.907771e+06 2.093922e+06 2.509998 19.549999 0.0 1.694676e+06 0.0 213094.9891 Sell
YHOO 2.119843e+06 2.119843e+06 5.289997 43.279999 0.0 1.907771e+06 0.0 212071.9951 Sell

475 rows × 9 columns

A more realistic portfolio that can invest in any in a list of twelve (tech) stocks has a final growth of about 100%. How good is this? While on the surface not bad, we will see we could have done better.

Benchmarking

Backtesting is only part of evaluating the efficacy of a trading strategy. We would like to benchmark the strategy, or compare it to other available (usually well-known) strategies in order to determine how well we have done.

Whenever you evaluate a trading system, there is one strategy that you should always check, one that beats all but a handful of managed mutual funds and investment managers: buy and hold SPY. The efficient market hypothesis claims that it is all but impossible for anyone to beat the market. Thus, one should always buy an index fund that merely reflects the composition of the market. SPY is an exchange-traded fund (a mutual fund that is traded on the market like a stock) whose value effectively represents the value of the stocks in the S&P 500 stock index. By buying and holding SPY, we are effectively trying to match our returns with the market rather than beat it.

I obtain data on SPY below, and look at the profits for simply buying and holding SPY.

Open High Low Close Volume Adj Close
Date
2010-01-04 112.370003 113.389999 111.510002 113.330002 118944600 99.292299
2016-09-01 217.369995 217.729996 216.029999 217.389999 93859000 217.389999

Buying and holding SPY beats our trading system, at least how we currently set it up, and we haven’t even accounted for how expensive our more complex strategy is in terms of fees. Given both the opportunity cost and the expense associated with the active strategy, we should not use it.

What could we do to improve the performance of our system? For starters, we could try diversifying. All the stocks we considered were tech companies, which means that if the tech industry is doing poorly, our portfolio will reflect that. We could try developing a system that can also short stocks or bet bearishly, so we can take advantage of movement in any direction. We could seek means for forecasting how high we expect a stock to move. Whatever we do, though, must beat this benchmark; otherwise there is an opportunity cost associated with our trading system.

Other benchmark strategies exist, and if our trading system beat the “buy and hold SPY” strategy, we may check against them. Some such strategies include:

• Buy SPY when its closing monthly price is aboves its ten-month moving average.
• Buy SPY when its ten-month momentum is positive. (Momentum is the first difference of a moving average process, or $MO^q_t = MA^q_t - MA^q_{t - 1}$.)

(I first read of these strategies here.) The general lesson still holds: don’t use a complex trading system with lots of active trading when a simple strategy involving an index fund without frequent trading beats it. This is actually a very difficult requirement to meet.

As a final note, suppose that your trading system did manage to beat any baseline strategy thrown at it in backtesting. Does backtesting predict future performance? Not at all. Backtesting has a propensity for overfitting, so just because backtesting predicts high growth doesn’t mean that growth will hold in the future.

Conclusion

While this lecture ends on a depressing note, keep in mind that the efficient market hypothesis has many critics. My own opinion is that as trading becomes more algorithmic, beating the market will become more difficult. That said, it may be possible to beat the market, even though mutual funds seem incapable of doing so (bear in mind, though, that part of the reason mutual funds perform so poorly is because of fees, which is not a concern for index funds).

This lecture is very brief, covering only one type of strategy: strategies based on moving averages. Many other trading signals exist and employed. Additionally, we never discussed in depth shorting stocks, currency trading, or stock options. Stock options, in particular, are a rich subject that offer many different ways to bet on the direction of a stock. You can read more about derivatives (including stock options and other derivatives) in the bookDerivatives Analytics with Python: Data Analysis, Models, Simulation, Calibration and Hedging, which is available from the University of Utah library (for University of Utah students).

Another resource (which I used as a reference while writing this lecture) is the O’Reilly book Python for Finance, also available from the University of Utah library.

Remember that it is possible (if not common) to lose money in the stock market. It’s also true, though, that it’s difficult to find returns like those found in stocks, and any investment strategy should take investing in it seriously. This lecture is intended to provide a starting point for evaluating stock trading and investment, and I hope you continue to explore these ideas.

Problems

Problem 1

Devise a trading strategy as described in lecture based on moving-average crossovers (you do not need a stop-loss). Pick a list of at least 15 stocks that have existed since January 1st, 2010. Backtest your strategy with the stocks chosen and benchmark the performance of your portfolio against the performance of SPY. Are you able to beat the market?

Problem 2

Realistically, with every trade a commission is applied. Read about howcommission works, and modify the backtest() function in the lecture to allow multiple commission structures (flat fee, percentage of portfolio, etc.) to be simulated.

Additionally, our current moving average crossover strategy results in a trading signal triggering the moment two moving averages cross. We would like to make sure signals are more robust, either by:

1. Triggering a trade when the moving averages differ by a fixed amount
2. Triggering a trade when the moving averages differ by some amount of *(rolling) standard deviations*, which are defined by:

$SD^n_t = \sqrt{\frac{1}{n - 1} \sum_{i = 0}^{n - 1} (x_{t - 1} - MA^n_t)^2}$

(*pandas* does have means for computing rolling standard deviations.) Regarding the latter, if the moving averages differ by $p \times SD^n_t$, a trading signal is sent. Modify the function ma_crossover_orders() so that these restrictions can be implemented. Specifically, you should have the ability to set how many days are in the window of the rolling standard deviation (it need not be the same as either the fast or slow moving average windows), and how many standard deviations the moving averages must differ by in order for a signal to be sent. (The current behavior of these functions should still be possible; in fact, it should be the default behavior.)

Once these changes have been made, repeat problem 1, including a realistic commission scheme (consider looking up one from a brokerage firm) when simulating the performance of the portfolio, and requiring the moving averages differ by some fixed number or standard deviations in order for signals to be sent.

Problem 3

We did not set up our trading system to allow for shorting stocks. Short selling is much trickier, since losses from short selling are unlimited (a long position, on the other hand, limits losses to the total value of the assets purchased). Read about short selling here. Then modify the functionbacktest() to allow for short selling. How will the function decide how to conduct short sales, including how many shares to short and how to account for shorted stocks when conducting other trades? We leave this up to you to decide. As a hint, the number of shares being shorted can be represented internally in the function by a negative number.

Once this is done, repeat Problem 1, perhaps also using features implemented in Problem 2.

Origin Part I

Origin Part II