No. of Recommendations: 148
May, 2006

What is MI? part 1 (of 6): Intro, Math, Problems

---- Intro
What is “Mechanical Investing”?
What is stock screening? Is it a new phenomenon?
Does it work?
---- Math
What's up with all the math?
What is XIRR?
What is GSD?
What is drawdown?
What is a Sharpe Ratio?
What is a Correlation Coefficient?
What is friction?


"In [all] cases the advent of popularity marked almost the exact moment when the system ceased to work well. … The moral seems to be that any approach to moneymaking in the stock market which can be easily described and followed by a lot of people is by its terms too simple and too easy to last."
– Benjamin Graham, The Intelligent Investor, 1949


What is “Mechanical Investing”?

Mechanical Investing as practiced here is the use of pre-defined procedures or algorithms for choosing financial securities to invest in: deciding what to buy, how long to hold, when to sell, what to buy next. This definition is intentionally broad, meant to encompass a range of investments: mutual funds, bonds, options, futures, options on futures, ETFs, REITs, etc. Someone who is using a timing system which tells him when to go long on NDX index futures and when to go short, is practicing Mechanical Investing. Someone who is using the Fed Ratio to tell her when to move from equity funds to bonds, is practicing Mechanical Investing.

The features of a “mechanical” investing system are that the pre-defined process tells the investor:

exactly what to buy
• when (under what circumstances) to sell
• what to buy next

Someone who is Dollar Cost Averaging by putting $100 every month into SPY is practicing Mechanical Investing: the implied holding period is “forever”, or “sell never” (or at least not until you need the money, or are old enough that your recommended asset allocation changes).

A Mechanical Investing program is specified with enough precision that anyone could execute it the same way. It is objective and repeatable. If you described your mechanical system to me and then went on vacation, I could (if I had access to the same dataset) make exactly the same trades for you that you would yourself; if I described my system to you, I could go on vacation and you could make exactly the same trades I would. The opposite of Mechanical Investing would be Discretionary Trading.

It is customary in investing essays to disparage “un-rigorous” methods of trading: taking a stock tip from your uncle or barber, following voodoo technical analysis guidelines etc. However, there are some quite successful and quite serious discretionary traders out there. Examples at TMF of “discretionary” approaches to investing include the Rule Breaker and Hidden Gems stock recommendations. It's not that the guidelines to those approaches are poorly defined or insufficiently explained: really, they're explained at great length. But there is subjectivity or judgment (discretion) in their application. How big a moat does this Rule Breaker really have around it? How explosive is the growth potential of this company? How gem-like is this tiny company, really? Two informed investors could sit down with the exact same Hidden Gems guidelines and the same dataset, pore thru company fundamentals and interview the same CEOs, and emerge with two different sets of stock recommendations. And in fact these recommendations might be quite good; but they are not mechanical in the sense of being objective & replicable.

I should also mention that there are quite successful Technical Analysis traders out there: most of the mega-successful “market wizard” types seem to be avowed technical analysts. Much of TA could be replicated “mechanically”: for example, Moving Average crossovers and other indicators; also basic trending, which could for example be defined using a regression on stock prices. Some other aspects of TA seem more subjective: for example, visual chart analysis looking for “head and shoulders” or “cup and handle” or “double bottom” formations. Those aspects of TA which could be defined explicitly enough to be backtested would certainly fit in with MI; other aspects might not be mechanical in the sense of being objective & replicable. I'm not trying to say those methods aren't useful: just that until they can be defined explicitly, it's tough to find a place for them within MI.

“Objective” and “repeatable” define part of the MI backbone: the other major component is “backtested”. You don't pluck a system out of thin air and start trading it. You trade it because you have some indication that it will outperform the market. Else why trade it? You could just buy SPY and have done. You need a reason to expect that the system will give you better long-term results than just Long Term Buy & Hold on an index, and that reason comes from looking at backtested results.

Why trust an algorithm over your own judgment?

Does TA Work? MI # 110056 10/28/2001
One of my favorite articles ever was a piece in the March 30, 1998 New Yorker, by a surgical resident at the Mass General Hospital named Atul Gawande. It should be mandatory reading for anyone interested in this board. …
[Gawande later wrote a great book called “Complications,” about his experiences in residency.]
The medical establishment has started to recognize that some form of automation can produce the best results in medical treatment, it has been reluctant to apply the same insight to the area of medical diagnosis … doctors are convinced that they'd better stick with their well-honed instincts when they're making a diagnosis…. How often does my intuition lead me astray? The radical implication of the Swedish study is that the individualized, intuitive approach that lies at the center of modern medicine is flawed – that our attempt to acknowledge human complexity causes more mistakes than it prevents. There's ample support for this conclusion from studies outside medicine. Over the past four decades, cognitive psychologists have shown repeatedly that a blind algorithmic approach usually trumps human judgment in making predictions and diagnoses. The psychologist Paul Mochi, in his classic 1954 treatise, “Clinical Versus Statistical Prediction,” described a study of Illinois parolees which compared estimates given by prison psychiatrists that a convict would violate parole with estimates derived from a rudimentary formula that weighed such factors as age, number of previous offenses, and type of crime. Despite the formula's crudeness, it predicted the occurrence of parole violations far more accurately than the psychiatrists did. In recent articles, Mochi and the social scientists David Faust and Robyn Dawes have reviewed more than a hundred studies comparing computers or statistical formulas with human judgment in predicting everything from the likelihood that a company will go bankrupt to the life expectancy of liver-disease patients. In virtually all cases, statistical thinking equaled or surpassed human judgment. You might think that a human being and a computer working together would make the best decisions. But, as the researchers point out, this claim makes little sense. If opinions agree, no matter. If they disagree, the studies show that you're better off sticking with the computer's judgment.

Maybe we don't have a lot of confidence in our own investing judgment. Most investors would see better results just from buying an index fund than from picking and choosing individual stocks: you've probably seen that information. A good algorithm might be more trustworthy than our individual judgment.

The definition of MI so far given is intentionally broad, to emphasize the wide range of activities that really are “mechanical investing”. All of these are touched on in the discussions on this board. However, the vast majority (like over 85%) of what we talk about on this board is mechanical screening for stocks.

What is stock screening? Is it a new phenomenon?

Stock screening is a very old idea. Benjamin Graham discusses it in his classic The Intelligent Investor, first published in 1949. There has been an absolute explosion in stock screening these last several years, as computer usage and the internet have become more basic to our society. There are online stock screeners, for example at MSN and at Zacks. It may seem more reasonable to the “average investor” these days to select stocks off a mechanical screen, than it might have ten or even five years ago. The ease with which an investor can run different stock screens has changed enormously.
But stock screening itself is an old practice. Here's an excerpt from chapter 7 of Graham's The Intelligent Investor (5th edition, 1973):

Our experience and study leads us to recommend three investment approaches [for the enterprising investor]…
[1] The Relatively Unpopular Large Company
In studies of the price behavior of the unpopular issues in the Dow-Jones Industrial Average… it was assumed that an investment was made each year in either the six or the ten issues in the DJIA which were selling at the lowest multipliers of their current or previous year's earnings. These could be called the “cheapest” stocks in the list… It was assumed further that these purchases were sold out at the end of holding periods ranging from one to five years. The results of these investments were then compared with the results shown in either the DJIA as a whole or in the highest multiplier… An original investment of $10,000 made in the low-multiplier issues in 1936, and switched each year in accordance with the principle, would have grown to $66,900 by 1962. The same operations in high-multiplier stocks would have ended with a value of only $25,300; while an operation in all thirty stocks would have increased the original fund to $44,000. The concept of buying “unpopular large companies” and its execution on a group basis, as described above, are both quite simple.
[2] Purchase of Bargain Issues
The type of bargain that can be most readily identified is a common stock that sells for less than the company's net working capital alone, after deducting all prior obligations. … A compilation made in 1957, when the market's level was by no means low, discloses about 150 of such common stocks. In [the table below] we summarize the results of buying, on December 31, 1957, one share of each of the 85 companies in that list for which data appeared in Standard & Poor's Monthly Stock Guide, and holding them for two years. …each of the groups advanced in the two years to somewhere in the neighborhood of the aggregate net-current-asset value. The gain for the entire “portfolio” in that period was 75%, against 50% for Standard & Poor's 425 industrials… none of the issues showed significant losses, seven held about even, and 78 showed appreciable gains. Our experience with this type of investment selection – on a diversified basis – was uniformly good for many years prior to 1957. It can probably be affirmed without hesitation that it constitutes a safe and profitable method of determining and taking advantage of undervalued situations.
[3] Special Situations, or “Workouts”
The exploitation of special situations is a technical branch of investment which requires a somewhat unusual mentality and equipment. Probably only a small percentage of our enterprising investors are likely to engage in it, and this book is not the appropriate medium for expounding its complications.

Mr Graham has just presented us with 2 stock screens:

1. List the 30 stocks in the Dow Jones Industrial Average, buy the six (or ten) with the lowest PE; hold for one year.

2. Buy 1 share each of the stocks you find in the S&P Monthly Stock Guide, that are selling for less than their net-current-asset value; hold for two years or until the stock price reaches the net-current-asset value.

Mr Graham produces backtests for these screens, and he compares the result to some benchmark: he justifies the use of the screen by its outperformance of the benchmark over the lookback period. Hello, welcome to Mechanical Investing.

Notice something else: Mr Graham isn't closely examining each company on this list of candidates, with an eye toward picking that one great stock. He's recommending these stocks as a basket. Buy all 6 (or 10) of the low-PE DJIA stocks and hold for a year. Mr Graham doesn't claim to know which ones are going to do better and which are going to do less well: his contention is that as a group this basket of stocks is likely to do better over the coming year than the other DJIA stocks. There's an essential humility in this approach, in not claiming to know which stock is going to do what. This is central to MI. The mechanical investor does not necessarily know anything about a particular company. Instead, he is trying to buy baskets of stocks which have certain characteristics, so that the odds of these “baskets” doing well are in his favor. With a defined holding period (eg 1 year), the mechanical investor gets to apply these criteria over and over during his investing lifetime; so that if the odds really are in his favor with these screening criteria, then the more chances he gets to apply the criteria, the more likely he is to beat the benchmark over the long haul. That's if the screening criteria tilt the odds in the mechanical investor's favor.

Let's examine that first screen of Mr Graham's. It shows some characteristics that are common to nearly all of the stock screens we use here:

A base universe against which the screening criteria are applied.
In this case, the Dow Jones Industrials. Another universe we commonly use is the set of stocks included in the ValueLine investment survey. In general you don't want to apply automated screening criteria to all publicly traded stocks, because there's a lot of garbage out there that you don't want to own. In this screen of Graham's, we're looking at low PE stocks: in the “all stocks” universe, many of the lowest-PE stocks represent companies that are going out of business, or have some other problems. Stocks which are plummeting for very good reasons, eg the WorldComs and Enrons of the world, will eventually show up on any low-PE or low Price-to-Book screen: the price will eventually fall low enough that its ratio would seem attractive, especially if the data provider hasn't had a chance to update its values for that company. If we restrict ourselves to the DJIA, we have some assurance that the stocks selected will represent large & stable companies. On this board we often use the phrase “crap filter” for this concept: some way to keep the riffraff off the screen.

A filter using price in some way.
PE in this case. The stock price is low vs the company's earnings-per-share. Other stock screens we use look for stock prices that are low vs other things, like earnings growth or sales or dividend yield. Still other screens look for price momentum, which will probably seem counter-intuitive at first. More on this in the section on types of stock screens: but note that it's very rare that a screen not include some price-related criteria.

A final sort.
In this case the final sort is the filter mentioned above, low PE. Mr Graham tells us to buy the 6 or 10 with the lowest PE; this indicates that the stocks are sorted by PE (ascending). The lowest-PE stock would be #1 on the screen, 2nd lowest #2, and so forth; the 7th stock will be the one with the 7th-lowest PE. A stock screen generally needs a final sort: and we want the sort to be supported by backtest results, so that the top 3 stocks (as sorted) generally perform better than the 4-5-6 stocks, which generally perform better than the 7-8-9 stocks. Stock returns are uneven of course, so the data will show a lot of noise: but what we want to see is that the higher-ranked stocks tend to do better than the lower-ranked stocks; else we will want to look at alternative final sorts.

A defined holding criterion.
In this case time-based, 1 year. Time-based holds are what we use most often with our stock screens on this board: so we usually describe screens as “annuals”, or “semis” (6-month hold), or “quarterlies” (3-month hold), or “monthlies” (1-month hold). (A 1-month hold is probably not what Mr Graham had in mind for investing: he would doubtless describe that as “trading”, or “speculating”.) Another common type of holding criteria is a “hold until” some condition is met: for example, in that second Graham screen, you're to hold for 2 years or until the stock price reaches the net-current-asset value (the price “advances to somewhere in the neighborhood of the aggregate net-current-asset value”). Deep value trading often has a criterion like this, where you buy an undervalued company and hold it until it becomes a fairly-valued company, or even a slightly over-valued company: then you take profits and find the next opportunity. This is distinct from Long Term Buy and Hold (LTBH), where you don't have a defined exit: implicitly you're keeping forever, or until something fundamental changes.

A backtest.
We'll have a lot more to say about backtesting below: but notice above that Mr Graham tells us that an investment of $10,000 made in the low-multiplier stocks in 1936, and rebalanced every year to follow the rule, would have grown to $66,900 by 1962; while just buying the base set of thirty DJ stocks would have grown to only $44,000. He produces a backtest result to show how this investing method outperforms the base set of stocks.

Mr Graham developed more stock screens than just the two discussed above. For example, if you look on Google, you can find his famous “seven criteria for stock selection for the defensive investor”. After discussing those criteria in The Intelligent Investor, he had this to say about baskets of stocks:

Our application of specific criteria to this select group of industrial stocks indicates that the number meeting every one of our tests will be a relatively small percentage of all listed industrial issues. We hazard the guess that about 100 issues of this sort could have been found in the Standard & Poor's Stock Guide at the end of 1970, just about enough to provide the investor with a satisfactory range of personal choice.

A range of “personal choice”. What's interesting is that Mr Graham doesn't seem to attach any importance to which individual stocks are selected; just that they be from the group that passes the tests. This is pretty much how we operate here. To such an extent, that sometimes we don't know much more about a company than its ticker symbol.

More even than Graham, the work of James P. O'Shaughnessy provides much of the foundation for what we do here. In his classic book What Works On Wall Street, from 1996, O'Shaughnessy reported on his research from taking 40 years of data from Standard & Poor's Compustat database and empirically testing many stock-picking methods. O'Shaughnessy put together 50-stock portfolios of certain kinds of stocks (for example, the 50 lowest Price-to-sales ratio stocks, the highest Price-to-sales ratio stocks, etc.): these paper portfolios were rebalanced annually. He examined all the measures that investors commonly rely on, including P/E's, Price-to-cash-flow, Price-to-Book etc, to determine what works. He found that certain criteria were consistently successful in mechanically screening for stocks. We don't typically put together 50-stock portfolios, nor endorse putting 100% of one's investment portfolio into a single stock-selection strategy: but O'Shaughnessy's use of meticulous research to formulate explicit rules for selecting stocks that have a good chance of beating the market, and holding for a defined time period, speaks directly to us. Notice that it shows the same humility in the face of individual stock selection that Graham's approaches above do. The investor doesn't rely on his own ability to perform in-depth analyses of companies & industries, or to pick great story stocks: in fact, he may think he's not capable of going that successfully. Instead, he relies on data to tell him what kinds of stocks have done well in the past: what criteria have worked. Then he picks baskets of stocks that conform to those criteria.

Backtesting, MI # 60083 2/29/2000
I am a Property and Casualty Actuary and my analogy stems from insurance risk classification. Let's take automobile liability insurance, for example. Given the entire population of drivers, there is an expected cost of insuring them. Risk Classification is the attempt to segregate all drivers into groups, based on some indicator variable. We hope that drivers within any group will have similar loss experience, and any group's experience will differ from other groups. If we can identify some variables that “predict” these differences, we can price each group differently. Companies like Allstate, State Farm, and Farmer's know that being a male age 19 doesn't cause you to be a riskier driver, but they have "backtested" results that show this group, as a whole, contains riskier drivers. They have been very successful using these risk screens for many years. I see mechanical investing as a way to segregate stocks in the same way. I don't believe that a price-to-earnings growth ratio causes a stock to outperform, but given historical data that shows groups of them do outperform, more often then not, I tend to believe that that indicator variable might be segregating a group of “better” stocks.

MrToast in MI # 184619, 2/28/2006
“We pick strategies and procedures here, not stocks.”

warrl, MI # 184801, 3/2/2006
we don't have watch lists. We don't look at stocks. Instead we look at HOW TO look at stocks. And make the computers do the actual looking. Many of our screens start with the ValueLine database of 1700 stocks. Others go to Stock Investors Pro / AAII / Reuters which starts with circa 8800 stocks.

What are the major stock screening criteria?

LAPropDoc addressed this in an excellent series on stock screening, a few years ago. Paraphrasing from the link:
The Major Types of Screen Criteria
Screens and their criteria can be divided into some general types:
• Value/Contrarian
• Income/Yield
• Institutional or Insider Ownership (relatively new category,
doesn't appear in Doc's post)
• Growth at a reasonable price
• Momentum or Relative Strength

Some more reading on this topic is available here:

A taxonomy of MI screens, 1/5/2002
(Note that was written before we had added the yield and ownership families of screens to our repertoire.)

One example of a Value or Contrarian screen is Mr Graham's first screen referenced above: out-of-favor Dow Jones stocks, representing The Relatively Unpopular Large Company. Note I'm possibly the only here one who insists there really is a difference between a “Value” screen and a “Contrarian” screen: but I'm writing this, so you get to hear about it. The next paragraph is for splitting the semantic hair to distinguish “Value” from “Contrarian”: you can definitely skip it if you want to.

Graham's Dow screen is clearly a Contrarian screen, because it explicitly looks for stocks that are “unpopular” as measured by PE, without any consideration for what the underlying company should “really” be worth. Most mechanical “Value” screens are really Contrarian in this way: they look for stocks that are cheaper than other stocks, by some measurement such as Price to Book ratio or Price to Sales or something, without calculating what a “fair” price would be for a stock. By contrast, Graham's Bargain Issues / net-current-asset screen actually is a Value screen, because it involves a calculation of what price you'd be willing to pay. This lets you follow his famous dictum to “buy a dollar's worth of a company for less than a dollar”: you formulate an idea of what a dollar's worth of the company is. The Benchmark Investing screens, which have their own board devoted to them, also involve a calculation of a “fair” price for the stock, which should also make them Value screens in the same way: however, I have an idea that the fair price is calculated strictly off of the company's historical price action, rather than its fundamentals (I'm not intimately familiar with BI). The BMW screening method, which also has its own board devoted to it at TMF, is strictly Contrarian in the sense that it's looking for out-of-favor stocks of big, safe companies, and it's waiting for the market to return to them. Note that BMW traders see this differently: they stress that they're calculating a “fair price” for the company's shares, based on long-term market valuation, in accord with Buffett's famous dictum that “In the short run the stock market is a voting mechanism, and in the long run it is a weighing mechanism.” So by their lights the screen is more Value than Contrarian. Eh. I'm not sure I agree. In general it's much easier to create a Contrarian screen than a true Value screen: real Value screens involve intense analysis of a company's balance sheets and/or income statements, while Contrarian screens can be whipped up using relatively quick measurements, like PE ratios and the like. And this highlights the thing that really separates Value investors from Contrarian investors. Both investors look for companies that are relatively cheaply valued compared to other companies: but Contrarian investors don't know if it might be cheap for a very good reason. Just about any Contrarian screen should pick up companies like Enron and Worldcom, as their stocks spiral toward zero: at some point the stock price is low enough for the PE or price-to-book or whatever to look really attractive. Is this plummeting stock a great buying opportunity, or a falling knife? Whereas the classical Value investor has done some deep analysis of the company, reassured himself of the strength of its business, looked at the discounted cash flows and all that stuff, and has a fairly clear idea of what the company ought to be “worth”. Faced with a plummeting stock, the Value investor has a very firm opinion about what he's looking at: he'll be decisive in either jumping in to buy all the shared he can, or sidestepping the falling knife.

(Not that real “value” investors never make mistakes: corporate fraud hurts everyone. And in practice both sets of investors share a lot in terms of philosophy and technique.)

Anyway: Value and Contrarian screens are usually lumped together here. The stocks which show up on these screens usually need some time for the market to turn toward them, so they are typically for longer holds. Mr Graham recommended two years for the Bargain Issues / net-current-asset screen, or until certain the stock price moved to your target. He recommended one year for the low-PE Dow screen. The BMW Method stocks are typically held for periods like one to three years, or until certain price criteria are met (unless your notion of the company's stability changes). The Benchmark Investing screens are typically held for a year. Our older screens in this category were originally looked-at with annual or semi-annual holds. Some Value/Contrarian screens backtest well with shorter holds.

Benchmark Investing:


Income/Yield criteria are a classic way of evaluating stocks. Above I referenced Benjamin Graham's famous “seven criteria for stock selection for the defensive investor”. One of them was a strong dividend record: Graham recommended uninterrupted payments of at least the past 20 years. The “Dogs of the Dow” is another well-known stock-picking strategy that emphasizes yield. The stock market has changed a great deal over the last several decades: nowadays investors do not demand dividend payouts to the same extent they once did, and many companies use excess cash in different ways (for example, repurchasing outstanding shares). However, dividend-paying issues have done very well in recent years: currently the YieldYear family of screens are among the favorite screens used on this board. These screens were originally thought of as for longer-term holds, like annuals: however, many of them seem to backtest very well with shorter holding periods, like quarterly or even monthly.

Institutional or Insider Ownership is a relatively new thing for us to be looking at, though if you read books by professional stock traders, it's something they've looked at for a long time. The notion is that institutional ownership of a stock (eg by a mutual fund) indicates a high level of price support; and indirectly, a company that has been well-scrutinized. If a mutual fund is actually taking a position in a stock, then the stock would likely go up. There may be other factors at work here: this isn't something we've studied in great detail. But it seems a promising criterion. We have some “PIH” screens, for “percent institutional holdings”, that seem to do very well with short-term holds like quarterly or even monthly. Insider Ownership indicates corporate leadership with favorable view of the stock's prospects. We have an “Optimistic Management” screen, which filters on dividend growth and then sorts by insider purchases, that backtests well as an annual.

Growth at a reasonable price, or GARP, is a widespread goal of investment advisors. Stock screens on this board which fit into this category often have a filter on PEG, or one of its variants: POG (price to growth in operating income), PSG (price to sales growth) etc. The stocks that show up on these screens are “pricier,” in terms of standard ratios like PE or price-to-book, than stocks which show up on Value or Yield screens. Typically these prices have gone up recently. In fact, our GARP screens don't always filter on company growth: some of them are filtered primarily on price growth, with some expectation that the growth is “reasonable”, or sustainable. An example of this would be the H52EarnPS screen, which first filters for stocks trading at 90% or greater of their 52-week high, but then requires EPS growth from the last quarter, and then sorts on a low price-to-sales ratio. The low PS ratio suggests that even though the stock is trading near its recent high, it's not necessarily “overextended”: it still has some room to grow (in price). Another example of this would be the Cornerstone Growth screen, from O'Shaughnessy's What Works On Wall Street. This screen requires stocks have a price/sales ratio below 1.5, positive earnings growth over the last 12-months, and then sorts by 12-month relative strength descending. O'Shaughnessy backtested his screen with annual holds. Most of the GARP variants we've backtested on this board seen to work best with shorter holds: semi-annual or quarterly, or even monthly.

Momentum, or Relative Strength, is far and away the most consistently powerful screening criterion we've seen. This is probably the single most counter-intuitive finding, for people new to mechanical stock screening. Stocks that have gone up a lot are due to go down, aren't they? You'd think. But over and over we've seen that stocks which have gone up, tend to go up a little more, at least for a little while. This effect is consistent, over a variety of market conditions. However, it is not durable, at least not where individual picks are concerned: momentum stocks require frequent trading. Picks need to be rotated: you generally need to get out of them every month, or at least quarterly. So there are potentially sizeable trading costs incurred. Also, stock screens based on momentum tend to be tremendously volatile, so that you have a very bumpy ride. But the potential rewards are great, even with a very simple momentum screen. For example, our backtesting indicates that if you had adopted this simple momentum screen in 1969:

• start with the 100 stocks rated Timeliness 1 by ValueLine.
• sort them by 26-week Relative Strength (gain over the past 26 weeks, compared to the market), descending
• buy the top 12
• hold for 1 month, rebalance

then over the 37 years from 1969, you would have achieved a 27% annual return, vs an 11% annual return for the S&P 500. In concrete terms? $10,000 invested in the S&P 500 would have turned into almost $411,000 over the 37 years. That's nice. The same amount invested in the RS26 screen could have turned into more than $78 million.

Sweet! Does it really work?

Great question.


If you don't mind, I'd like to hold off addressing that, until we've hit some other topics.


What's up with all the math?

Let's ask a question. Consider these two reported investment performances:

• A: $10,000 invested at the beginning of 1989 would have turned into $48,920 by the end of 2000!!!!
• B: $5,000 invested at the beginning of 2000 would have turned into $10,784 by the end of 2005!!!!

Which performance is the more impressive?

C'mon, it's a simple question, right? Should have a simple answer. But what is the answer? So let me first point out the hyperbole in the language describing the performances of these two investments. This is a pet peeve of mine: it annoys me so much, I'm having trouble editing all the cussing from this paragraph. Isn't this crap the standard crap you see hyping every investment? Everything on the internet, every scheme, every scam, every newsletter, every premium investment service is touted with this intentional obfuscation. “X dollars invested in year 1 would have turned into Y dollars at the end of year Z!!!!” Yes, intentional obfuscation: because it hides information rather than reveals it, because it uses an eye-popping number to catch the reader's attention while taking advantage of the reader's inability to do nth-roots in their head. The truth is that every compounding investment will produce an eye-popping number at the end of the investment period: that's the miracle of compounding. And the other truth is that small differences in annual percentage return can cause big differences in the number at the end of the investment period, or at least differences that seem very big.

If you're going to compare two investments, the very first thing to do is put them on the same footing. We like to see them annualized: the Compounded Annual Gross Return, abbreviated CAGR. At least then we can compare one annual return to another annual return, which can be the beginning of a conversation. And here is why we start to do some math. Going back to our original question:

• A: $10,000 invested at the beginning of 1989 would have turned into $48,920 by the end of 2000!!!!
• B: $5,000 invested at the beginning of 2000 would have turned into $10,784 by the end of 2005!!!!

Investment A: the beginning of 1989 thru the end of 2000 is 12 years. $48,920/$10,000 = 4.892. So this investment multiplied its original stake by 4.892 over 12 years. The average annual multiple would be the 12th root of 4.892, or 4.892 ^ (1/12), which equals 1.141451. An annual multiple of 1.141 is an annual return of 14.1%.

Investment B: the beginning of 2000 thru the end of 2005 is 6 years. $10,784/$5,000 = 2.1568. This investment multiplied its original stake by 2.1568 over 6 years. The average annual multiple would be the 6th root of 2.1568, or 2.1568 ^ (1/6), which equals 1.13667. An annual multiple of 1.137 is an annual return of 13.7%.

These two investments had performance results very close to one another, a CAGR of 14.1% vs a CAGR of 13.7%. That's virtually indistinguishable; and in case you haven't noticed, we just started doing math. To compare the results of different investment strategies (or stock screens), we will almost always put them on an annual basis, and that will involve either taking the nth root of a cumulative return or taking a geometric mean of a series of annual returns. For example, consider the following series:
S&P 500 annual returns
Year Pct

1989 36
1990 -5
1991 31
1992 7
1993 10
1994 2
1995 38
1996 24
1997 27
1998 33
1999 20
2000 -11

To calculate the CAGR, the first step is to convert that percentage return in the second column into a return multiple. Create a third column: the value in there is 1+(Pct/100). So you'll have 1.36, .95, 1.31, 1.07 etc. Now there are 3 different methods to calculate a CAGR. The methods are mathematically equivalent.

1. Take the product of the column of return multiples (6.6967) and then take the 12th root of that figure. That gives you an annual multiple.
2. If you're using Excel, use the GEOMEAN() function on the column of multiples. That will give you the average annual multiple.
3. Create an additional column, with each value being the log of that row's return multiple. Take the average of these logs, then take the antilog (or EXP) of that average. This will give you an average multiple.

Each of those three calculations should give you this figure: 1.11835. So the S&P 500 had an average annual return of 11.8% during the period 1989-2005. We say its CAGR was 11.8%.

By the way, that raises an interesting point. Earlier we said that Investment A had a CAGR of 14.1% and Investment B had a CAGR or 13.7%. So Investment A looks marginally better than Investment B. But the S&P 500 returned 11.8% per year during the time period that Investment A was running; whereas it returned only -0.85% per year over the period that Investment B was running. So how you like me now? Relative to the market, the performance of Investment B was absolutely the more impressive. The lesson to be drawn is, you can't directly compare investments over different time periods: you need apples-to-apples, an annualized return figure and the same performance period, in order to directly compare investments.

A great summary of CAGR and GSD and the Sharpe Ratio and how to calculate them can be found here:

Once you dip your toe into the pool of mathematics, it only gets deeper. From figuring out an average return multiple, it's only a matter of time before you get to standard deviations and tests of statistical significance and linear regressions and on and on and on. We sling around a fair amount of math. But don't get bogged down in the computations: keep your eye on the ball. We did logs or geometric means above in order to figure out what our annual return is. When we do standard deviations it's for purposes of assessing volatility of an investment, or how risky it is. When we look at linear regressions it's in order to find stocks that are growing “smoothly”. The tools we use for these things are mathematical and occasionally abstract; the questions we're trying to answer are all about trying to find good investments.

Of course some people here just really like math, like LorenCobb:

Optimizing Screen Parameters, MI # 100450 5/3/2001
Warning: canary genocide with vector calculus follows…

What is XIRR?

Excel's function for calculating the Internal Rate of Return. It's used for getting a CAGR from an investment where you're putting money in at various times. For example, suppose that you're investing your Roth IRA in a blend of MI screens, and at the same time you're making your max allowed annual contributions. Figuring your CAGR is not as simple as taking the starting value and ending value and seeing what annual return multiple gets you there, because of the added monies. You'd need use XIRR() or some calculation like it to get an accurate annualized return.

If you have Excel, see their help for the XIRR() function.

What is GSD?

Geometric Standard Deviation, a measure of the standard deviation of multipliers. It's the way we measure return volatility in stock screens. The lower the GSD, the lesser the volatility and the more tolerable an investment is. Screens with GSDs over 25 and 30 will give you wild fluctuations in account value: big swings up and down, huge gains and hard-to-stomach losses. The high-return screens we've developed here, particularly the momentum screens, are very often high-GSD screens, and thus have high volatility. Check out the Wikipedia article on standard deviation. Look at the graph they show, under the heading “Geometric interpretation”. Our GSD is a standard deviation on logs of return multiples, annualized.

The area under the bell curve is of interest. If you assume a normal distribution, then that means some 68% of the time the return you get will be within one standard deviation of the observed average return; some 95% of the time the return you get will be within two s.d.'s. Looking at it the other way, that means that about 1/3 of the time you can expect a return outside the 1 sd range, either above it or below it; and about 5% of the time you can expect a return outside the 2 sd range. That's if you assume a normal distribution, which is unreasonably optimistic: the accumulated evidence overwhelmingly suggests that the distribution of stock market returns is “fat tailed”, meaning that values outside of the sd ranges occur far more often than a normal distribution would imply.

Standard deviation is often referred to as “sigma”. A “3 sigma event” would be one that falls outside the 3 standard deviation range around the average: in other words, an event that should be rare according to the statistical analysis. 3 sigma events aren't anywhere near as rare in investing as the statistical analysis would lead us to believe: they happen a lot more often, which is part of the fat tails argument.

A great summary of GSD and the Sharpe Ratio and how to calculate them can be found here:

What is drawdown?

A measurement of the maximum loss an investment portfolio sustains over any single period during the lookback time frame. It's related to GSD, in the sense that it's another way to estimate portfolio risk, and also in that GSD and drawdown tend to be correlated: an investment with high GSD will tend to show a large drawdown. If you were invested 100% in the S&P 500, then from Sept 2000 to October 2002 your portfolio experienced a drawdown of around -45%, almost half your money.

Another way of looking at portfolio risk:

Ulcer Index, MI # 21725 4/1/1999

Risk Measures – Ulcer Index, MI # 26000 5/8/1999

Yet another:

Re: Binding the Bear: Simply & Easily, MI # 185357 3/14/2006
I've tuned for a new risk metric: I calculate the drawdown % (current port value / high water mark to date - 1) at every daily close, then take the standard deviation of all those daily numbers. A perfect score is zero, which would be for something which rose or stayed the same every day, and higher numbers give a sleeplessness metric.

What is the Sharpe Ratio?

A standard measurement used within the financial industry to evaluate the risk/reward profile of an investment. Dr William Forsyth Sharpe won a Nobel Prize for this stuff. For additional background you'll also want to read about the Capital asset pricing model and Modern portfolio theory.

It starts with the assumption that there is a “risk-free investment” that you could put money into. That risk-free investment is usually assumed to be US Treasury bonds. You can get a certain rate of return from Treasuries, given by whatever the current yield is. If you want a higher return (and who doesn't?), then you generally have to take on more risk. For example, stocks give you a chance at a higher return, but you assume the risk of losing some of your investment. The excess return you get from your investment (say stocks) comes at the cost of the extra risk. The Sharpe ratio is an attempt to measure that cost: it's the ratio of excess return (over the risk-free return) to the extra risk, where the figure used for risk is the volatility of the investment. Conceptually:

Sharpe ratio ~ (excess return) / volatility

A great summary of the Sharpe Ratio and how to calculate it can be found here:

The conclusion is that investments are most usefully ranked according to their risk-adjusted return: how much return did they give in exchange for any excess risk they took?

This is pretty profound stuff. (They don't hand out Nobel Prizes for nothing.) Yet it seems to me that there are weaknesses with it. The first thing that jumps out at me is, observed volatility isn't the real risk that was assumed. Volatility is a decent guesstimate for risk, and we know that high volatility is painful. But the volatility that occurred during the backtest only tells part of the risk story. Just because the stock market crash of 1929 and the subsequent Great Depression didn't occur during your investment period doesn't mean there was no risk of such an event. The actual volatility you observed during your trading is always smaller than the volatility that was possible. The other thing that jumps out at me is, this is a ratio, so you could get a high value by reducing the denominator. Theoretically there could be an investment that gave you a return 1.5% higher than the risk-free rate every sixth months. Such an investment would have extremely low volatility; so it could have a nice high Sharpe Ratio. Yet its total annual return could be in the neighborhood of 7%: less than the annual returns of the S&P 500 over the last few decades, and not an investment I'm interested in. That last concern is probably a little bogus: we are usually blending high-CAGR stock screens, so it's not like a low-CAGR investment will sneak in there. But it seems at least theoretically possible, so I usually keep half an eye on the CAGR when I sort investments by the Sharpe Ratio.

There's another weakness with some of the theoretical underpinnings of the Sharpe Ratio, perhaps a fundamental weakness. Volatility is used as a proxy for risk, and it's a decent one: but volatility (beta) is not the same thing as real risk. This is a favorite topic of Warren Buffett's, so let's hear it from him:

(The quote in the link is from Buffett's essay “The Superinvestors of Graham-and-Doddsville”, which first appeared in the Columbia Business School magazine in 1984, and is now Appendix 1 to Graham's The Intelligent Investor.)

Buffett's reminder on statistics, MI # 5138 8/19/1998
“I would like to say one important thing about risk and reward. Sometimes risk and reward are correlated in a positive fashion. … The exact opposite is true with value investing. If you buy a dollar bill for 60 cents, it's riskier than if you buy a dollar bill for 40 cents, but the expectation of reward is greater in the latter case. The greater the potential for reward in the value portfolio, the less risk there is.
One quick example: The Washington Post Company in 1973 was selling for $80 million in the market. At that time, that day, you could have sold the assets to any one of ten buyers for not less than $400 million, probably appreciably more. The company owned the Post, Newsweek, plus several television stations in major markets. Those same properties are worth $2 billion now, so the person who would have paid $400 million then would not have been crazy. Now if the stock had declined even further to a price that made the valuation $40 million instead of $80 million, its beta would have been greater. And to people who think that beta measures risk, the cheaper price would have made it look riskier. This is truly Alice In Wonderland. I have never been able to figure out why it's riskier to buy $400 million worth of properties for $40 million than $80 million.”

So. What I take away from that is this: do not be soothed by the beautiful precision of the Sharpe Ratio. It's a great-looking number out to a couple of decimal places, and it's important, but it is not the last word on risk vs reward.

The Sharpe Ratio has been such an investment standard that we've traditionally used it too. When people on this board construct “optimized blends” of stock screens, they are most often trying to maximize the Sharpe Ratio. In general, no one here will settle for a Sharpe Ratio of less than one: that would mean you're assuming more risk (in the form of volatility) than you're getting back in investment return. In fact most of our better screen blends show (backtested) Sharpe's over 1.20 or even much higher.

The Sharpe can be a little tricky computationally (I've screwed it up a few times when posting backtest results to the board), so we sometimes proxy it by using the ratio CAGR/GSD. That's a quick-&-dirty shortcut calculation, which gives you a return-vs-volatility number. It will usually rank investments in about the same order as the Sharpe Ratio will (see MI # 105158 ), but it will occasionally give a slightly different ranking (see MI # 105537 ).

Some recent alternatives to the Sharpe Ratio have been proposed in the academic community, including the Sortino Ratio and the Upside potential ratio.

The Sharpe Ratio has one other very significant quality. It is designed in such a way that, if you select from a number of investments based on the Sharpe Ratio, and the investment you choose does not have quite as high a CAGR as you're shooting for, then you can use margin to “amplify” the CAGR up to where you want it, while keeping the same risk/reward profile you choose the investment for. That result is to some extent theoretical: you can't use margin in an IRA, and I personally can't borrow at the risk-free rate (maybe you can). But if you are able to use margin in your trading account, it's an important, even profound, result.

Why pay so much attention to GSD? I like volatility to the upside; it's downside volatility that bothers me. Max drawdown is the thing to be concerned with.

So you'd think.

It's very easy for a newbie to get very excited about the high CAGRs, and decide that high volatility just doesn't matter in pursuit of high returns. “If I'm holding for 20 or 30 years, then who cares how I got there if in the end I have a high return?” But in fact it matters a lot. Take that RS26 we mentioned in passing earlier. If we reduce the holding to 5 stocks, that screen shows a backtested CAGR of 25 and a GSD of 42. The Sharpe Ratio doubled that of the S&P 500 over the 37-year backtest. But that GSD of 42, that's a big deal. The mean return was +25%, but if you look at the returns 1 and 2 sd's out you get these:

CAGR+2sd +152
CAGR+1sd +77
CAGR-1sd -12
CAGR-2sd -38
CAGR-3sd -56

The 1sd range is from -12% to +77%; the 2sd range is from -38% to +152%. Remember we said that assuming a normal distribution, you have approx a one in three chance to get a return outside the 1sigma range, and about a one in twenty (=5%) chance of a return outside the 2sigma range. Over a 37-year backtest, that's about 12 occurrences of a return outside the 1sigma range, and about 2 occurrences of a return outside the 2sigma range. That's assuming a normal distribution: remember we also said that the distributions of stock market returns have fat tails, even very fat tails, which means that values outside the sd ranges are much more likely than we might think. You're going to get the multi-sigma years, and more of them than you think.

Now I know what you're thinking, seriously I do. You're thinking that one or two +152% returns cover a multitude of sins. If that happens twice, those two years multiply your account value by 6.35. Sweet. But the problem is threefold:

1. You never know which sigma you're going to get, the plus one or the minus one. If you did, this whole stock market thing would be a lot easier. Market surprises are often rude.

2. It is actually easier for the market to go down than it is for the market to go up. Think about it. For stock prices to go dramatically up, the money to buy them has to be coming from somewhere. You had to scrounge to put together your trading account: this stuff doesn't grow on trees. The huge bull market of the 90s was fueled in part by liquidity: there was a lot of cash lying around. Is there a lot more right now? Whereas for stock prices to fall dramatically, all that needs to happen is for a significant chunk of people to sell. If everyone suddenly said “I want my money back,” we have a problem.

3. The minus ones hurt more than the plus ones help. This is huge, and easy to miss when you're looking at a string of multipliers.

Consider two consecutive returns: one year you are lucky and have a +50% return, and the next year you are unlucky and have a -50% return. Wild ride. So tell me: after two years, are you even? You are not. After two years, you are down 25%.
Year   return    account
0 - $10,000
1 +50% $15,000
2 -50% $7,500

Returns that appear symmetrical when described as percentage gains and losses really are not symmetrical. The inverse of -50% is not +50%, it's +100%: if you lost half your money, you'd have to then double your money to get back to even. This has far-reaching ramifications that are not captured in backtest results. Suppose you were originally holding 10 stocks at a thousand apiece, with an annual rebalance. Now that you're down 25%, what happens to your portfolio allocation? Do you hold 7 stocks at a thousand each, planning to open your eighth position when you get some gains? Ok: but now are you adequately diversified? Or do you instead hold 10 stocks at $750 each? If so, what happens to your trading costs? Now that your position size is smaller, each commission and fee eats up a larger percentage of your account: this acts like a drag on your potential gains, and may make your intended plan impractical.

Note also that the backtester assumes the purchase of fractional shares: all your money is 100% invested. But in fact you can't achieve that in real life. Suppose the individual stock you want to buy trades at $150. If you're down to $750 per position, you can't even buy 5 shares: you pay some commission so you can only buy 4 shares, and now you've got something like $140 of that position sitting on the sidelines earning you almost nothing.

It's a lot harder to win back 33% of your account (the amount needed to turn $7,500 into $10,000) than it is to lose 25%.

The fact is that a multi-sigma event can put you out of business. And volatility does not respect a plus sign or a minus sign. High volatility can go in any direction it wants. I know it feels like there is, but in mathematical terms there is no difference between upside and downside volatility: volatility is volatility. When you look at backtest results, you want to keep volatility (as measured by GSD) low. Trust me on that.

Emintz lectures us about this in 2002:

New Backtest: RS4 > 0 Filter, MI # 117584 2/8/2002
Remember the discussions at the end of 1999 and beginning of 2000? Some folks suggesting that you could find a screen that had upside volatility but no downside volatility? Boy, didn't they get a rude awakening? There were exceptions, of course, but this was the overall trend. People wanted to believe that you could separate out the upside and downside variance of a screen, and that they could be different. The results of 2000-2001 showed them wrong. If you want upside potential, you have to accept downside risk. As the physicists would say, there's no such thing as a free lunch.

Home Run, MI # 123220 4/19/2002
I have yet to see any evidence that the distribution of returns is not symmetrical in log-space. Individual screens may show some skew, but I'm talking about the population of screens as a whole. Therefore I don't see a basis for distinguishing between upside and downside variation. This is what got a lot of people in trouble in early 2000 - they became convinced that a screen could have high upside volatiliy without high downside volatility.

Is max drawdown a useful measure? MI # 123275 4/19/2002
The fact that GSD is a better predictor of max drawdown than Sharpe ratio just supports the idea that max drawdown is largely a random event - and the more volatility there is in your stocks, the greater the chance of a catastrophic drawdown. It also points out something I've said repeatedly - those who think that there's a difference between upside and downside volatility are fooling themselves.

Zeelotes examines the issue more recently:

drawdown vs. Stdev, MI # 184441 2/24/2006

drawdown vs. Stdev, MI # 184448 2/25/2006

Duh: why don't you just cut your losers and let your winners run?

Good thinking.

As it happens, we've looked at this issue again and again over the life of this board. And thus far, in any systematic study of the issue what seems to be the case is, any use of stop losses systematically hurts your returns. It seems that the stocks chosen from our momentum screens are volatile enough that you'll get stopped out of your big gainers, to such a degree that many screens are no longer attractive investments. This has kept most of us from using stop-losses in our regular MI trading. Interestingly, some of the research on the topic indicated that selling winners early, and promptly investing the proceeds into other screen stocks, had positive results. Check out these old posts:

PST, MI # 36617 9/4/1999
increase in return when the PST was set at about +20%. Locking in these high returns in the middle of the month and preventing a 'return to the mean' counterbalances the months when the returns are 25-30%.

Re: A simple sell strategy, MI # 41203 10/12/1999
Selling when each pick hit 20% increased the average monthly return from 5.06% to 5.82% and the cumulative monthly for the period from 76.34% to 91.93%.

What we know so far about Stop Loss / Sell Early, MI # 28080 6/19/2001

Protecting Profits / Stop-Gain, MI # 116119 1/27/2002
The traditional wisdom from many stock gurus supports the stop-loss argument -- they say to let your winners run and to cut your losses early. However, I've taken a look at stop-loss approaches in the context of some MI screens before, and concluded they did more harm than good for the situations I looked at. There's another approach to “protecting profits” that's the opposite of stop-loss… I called it stop-gain. … I've recently taken a look at using this same approach on longer holds using standard MI screens, simply to see if the idea can be extended to our more traditional environment. This post looks specifically at RS26-Quarterly and leads me to the tentative conclusion that the idea has merit. RS26 results over the 32-year period 1970-2001 yielded 34% - 40% when using the stop-gain approach, versus 28% with a straight quarterly hold.

There's also this article, from an outside source:
the frequency of the losses for the investor will increase as a function of the proximity of the stop loss. … The probability of deriving a profit declines as the stop loss point becomes closer to the present asset price</span>

The discussions of stop gains / sell early have recently resurfaced:

Highlight Timing: Applied to Screens, 11/21/2005
have mainly used Screamers or one of the variations: (gentle screamers, GS-Voom)but I don't always hold for a month-I usually put in a sell @ a 20% gain (limit order) so I don't have to follow thing too closely. Though I know that this sort of strategy has not been shown to work in past posts on this board-It allows me to sleep & although I kick myself when one goes up 25% or more & I sold at 20% - I am thankful for the gains when I can take them.

20% profit latch with NHS, 11/29/2005
I was curious what this method would do to returns, so I got out my SIPro data disks and ran through… we get a reduction in volatility and the CAGR goes up as well. This is because while we average returns within each month, the monthly returns are multiplied [more frequent compounding] … It's not often that you get a 50% increase in CAGR (30% to 47%) while cutting GSD by 40% (60 to 35).

Screamers w/ Profit Latch, 12/1/2005

Profit Latch: A Word of Caution, 12/2/2005

I don't think we've heard the last word yet on this subject: but the simplistic notion of cutting your losers does not seem applicable in a completely straightforward way to MI.

What is a Correlation Coefficient?

It's a statistical measure of the degree to which two series of numbers move together. It's a decimal value ranging from -1.0 to 1.0. A value of 1.0 would mean that the two series are perfectly correlated: as one set of values moves up or down, the other will as well, to the same degree. Consider two series:


1 2
2 4
3 6
4 8
5 10

Series A and series B have a correlation coefficient of 1: they move together. Now look at these:


1 10
2 8
3 6
4 4
5 2

Series C and series D move exactly opposite one another: as one goes up the other goes down. They have a perfect inverse correlation: the coefficient is -1. But these:


1 6
2 4
3 10
4 2
5 8

These two series do not have a clear-cut relationship. As one goes up, the other might go up or down or sideways. The coefficient is near zero: the actual value is 0.1. We say they are not correlated.

You can find some mathematically rigorous treatments of the Correlation Coefficient on the internet:

This stat becomes relevant for investing when you start to look at diversification. Check out this graph, of the S&P 500 Index and the Dow Jones Industrial Average since 1950:

I won't bother to download the values and calculate the coefficient: I think we can confidently say that those two are pretty tightly correlated, with a coefficient approaching 1. When one of them goes up, the other does too: and when one goes down, so does the other. Now look at this, the CBOE Gold Index vs the S&P 500 since 1997:

We can see that they don't generally tend to move together. I won't bother to calculate it, but I'm pretty sure that their correlation coefficient is negative.

Degree of correlation has important consequences for diversifying, when it comes to investments. The purpose of “diversification” in investing is to spread out your money so that your eggs are not all in the same basket: the idea is that you'll have some degree of protection in adverse market conditions. If a segment of the stock market drops, you won't lose too much of your money, because you'll be “diversified” into other areas. Classically you would diversify by asset allocation: you'd have some stocks, and some bonds, and some real estate, and so forth. But even within the stock portion of your assets, traditional advice is to diversify, to protect against market risk. This all ties in to modern portfolio theory; check out some of the links under the Sharpe Ratio question, above. I should note that Warren Buffet is an outspoken critic of the traditional idea of stock “diversification”: he counsels focus. However, the degree of focus Mr Buffet can bring to bear on his investments, to the extent that he knows everything there is to know about the businesses he buys, is perhaps not achievable for the average part-time investor / 401k participant. Even Buffet owns a number of businesses, not just one: and some research suggests that adequate diversification against market risk starts to be achieved when you own about 12 different stocks.

Some further discussion of the correlation coefficient:

Elan's correlation coefficient article, FW # 22299 12/8/2000

Uncorrelated investments/rebalancing, FW # 29431 12/17/2001

Pearson Correllation Coefficient? MI # 74674 7/18/2000

More texture on the Correlation Coefficient later, when we get to Blending; including some remarkable work Emintz did to group our VL screens into families.

What is friction?

Friction is what slows you down. . Most backtests show pure stocks gains, ie on the date you bought it closed at price A, and on the date you sold it closed at price B. This gives you as estimate of the raw stock return: but for various reasons a real trader would probably have been unable to realize the raw stock return. Commissions & fees, spreads, taxes if not in a tax-deferred account: these all act to make your real-world returns less than the “idealized” returns you see in reports of backtest results.

Commissions & fees
BrownCo charges you $5 per stock trade, but they've been swallowed up by eTrade and their fee structure seems likely to change; Scottrade charges $7; Ameritrade $10.99; eTrade $12.99, although they have tiered pricing which gives lower commissions to very active traders or to large accounts. FolioFn will let you trade their “window stocks” free, if you pay a $200 annual fee; otherwise it's $4 per trade on window stocks and $14.95 per trade on non-window stocks, with a minimum of $14.95 a quarter in fees (which is not bad). Trading costs money. And this cost is a drag on portfolio performance.

Suppose you are looking at trading a screen which backtests to a 36% CAGR with monthly trading of 5 stocks. 36% is a great annual return, you'd double your money every 27 months if you could get that year-in and year-out: 10 grand would turn into more then two mil in 25 years. Further suppose you are doing this at Ameritrade, paying $11 a pop to trade, and you have just $5,000. By the way, don't try this at home: 5 grand is nowhere near enough to support monthly trading. Anyway: let's further assume there is no volatility to this strategy at all: you get 2.6% return every month. (If you find this investment, let me know, I want in.) Here's what you'd look at with no commissions:

Start Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec End
$5,000 $5,130 $5,263 $5,400 $5,541 $5,685 $5,832 $5,984 $6,140 $6,299 $6,463 $6,631 $6,804 $6,804

Nice. Now here's what you're looking at with commissions:

Start Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec End
$5,000 $5,017 $5,035 $5,053 $5,071 $5,090 $5,110 $5,130 $5,150 $5,171 $5,193 $5,215 $5,238 $5,128

You've turned a 36% annual gain, which you really did get on the stock you purchased, into a 2.5% realized gain in your portfolio. You'd have been better off buying bonds. The problem is that you paid $110 every month to trade stocks: $11 times 5 stocks to buy, then $11 times 5 stocks to sell. You would have gained $130 on your stocks in January ($5000 times 2.6%): commissions whittled that down to $20 (actually less than that, since commissions eroded your initial purchasing power). The lesson: commissions matter.

When you get a quote from your broker, there are two prices, a bid and an ask. The market maker will take either side of the trade, either selling shares to you or buying shares from you. The bid is what the market maker is offering you to buy a share of the stock, the ask is what he wants to sell a share. Guess what? The ask is higher than the bid. I just went online to Interactive Brokers and checked the quote for SPY, the S&P 500 ETF, one of the most liquid issues there is: the bid was $129.66 and the ask $129.68. The market maker is scalping 2¢ per share off of each order: which can add up to a fair amount of money, given the tremendous volume in SPY traded every day. However, the volume in SPY acts to keep the spread pretty tight. If a market maker tries to create too wide a spread, some other market maker will step in with a tighter spread and the trading action will go to him. You're generally not going to get rooked buying or selling SPY.

However, low volume, thinly traded stocks like some of the small cap stuff that pops up on our screens, can sometimes have large spreads. The market makers are scalping more than just a couple cents: and they can do this because there isn't enough action on these stocks to force a competitive market. Entering a market order on stocks like these can be an invitation to having your lunch money stolen. You'll pay a little too much to buy the stock, and then receive a little too little when you eventually sell. Above we made a big deal out of spending $55 to enter positions in 5 stocks: now suppose you want to buy $5,000 worth of a thinly-traded stock, with a bid/ask of 12.40/12.90: a huge 50¢ spread. Five grand will buy you 387 shares of that stock. How much will it cost you if you buy those 387 shares at market price and then sell them right back, assuming you get a free ride on commissions and there's no change in the bid/ask quotes? Yes, you just lost $193, which is 3.8% of your investment. That's assuming you actually got your whole order filled at the bid/ask, and did not encounter slippage. The spread represents a huge hurdle in making money trading that stock. The most basic thing you need to do is use a limit order for your trades; but sometimes there just aren't enough to go around, and you'll sit unfilled for a while.

Spreads can be a very serious challenge, when trading low-volume stocks: more so even than commissions. Some MI traders will just skip a stock, if the bid/ask spread is too great a pct of the stock price; others will go ahead, but use limit orders. Pay attention and be careful.

Most MI practitioners trade in an IRA, so this is generally a non-issue. But if you're trading in a taxable account, then there are some things to pay attention to. Gains on holds over a year are taxed at a (comparatively low) capital gains rate: but gains on short-term holds (less than a year) are taxed as ordinary income. Our quarterly and monthly screens can create a big tax bill for you. Tax bills can be interesting: this was particularly true at the end of the tech bubble.

Gains and losses, MI # 67041 4/20/2000
this brings to the forefront one of the hidden thorns of monthly trading. From October to January, a monthly portfolio racked up huge gains--all short term, all in 1999, for which the taxes must be paid in April 2000. January to April took back all those profits -- and then some. The 6-7 months from Oct to April were probably a wash profit-wise. But the profits and losses fall into different tax years. So you have to pay the full tax on the gain portion… Oh yes, one more thing. The money that was going to be used to pay the taxes disappeared the week before you had to write the check.

As Ray would say, “It's never the bus you see coming that squashes you flat.” Here's another feel-good story:

Monthlies + margin + dump = Disaster, MI # 72247 6/21/2000

And another post full of interesting discussion:

When to Go to Monthly in a Taxable Account? MI # 45249 11/12/1999

I am not a tax advisor and have no useful advice for you on these issues, except to pay attention and be careful. Also, as you keep track of your portfolio performance, make sure you track the impact of taxes which need to be paid, whether you actually pay them from your account or from another source: performance net of taxes is an important point of comparison between trading strategies and LTBH strategies.

Fractional shares
This is not really a source or friction; but it is a way in which real-world results can differ from backtest results, so I thought I'd mention it. Consider a screen that backtests to a 36% CAGR for a 5-stock monthly hold. The backtest probably assumes 100% investment: that is, if you had $20,000, then you'd have an even $4,000 in each of the five stocks on the screen this month. But in real life the stocks you're going to trade might have prices like $37, $42, $17, $96 and $22. Assuming $7 commissions at Scottrade, then the actual purchases you're going to make are:

amt/stock price quotient actual invested amt

$4,000 $37 107.92 107 $3,959
$4,000 $42 95.07 95 $3,990
$4,000 $17 234.88 234 $3,978
$4,000 $96 41.59 41 $3,936
$4,000 $22 181.50 181 $3,982
$20,000 $19,845

You wanted to be “fully” invested, but this month the closest you could get is 99.2% invested, just because of the way the stock prices fell out. In general you've never going to be able to really be 100% invested (unless you're using margin). As sources of friction go, this is minor compared to spreads and commissions; but it's worth being aware of.

Jamie's backtester page has a trade simulator which will show results of trading a screen with friction. It's worth a look.

Print the post  


What was Your Dumbest Investment?
Share it with us -- and learn from others' stories of flubs.
When Life Gives You Lemons
We all have had hardships and made poor decisions. The important thing is how we respond and grow. Read the story of a Fool who started from nothing, and looks to gain everything.
Community Home
Speak Your Mind, Start Your Blog, Rate Your Stocks

Community Team Fools - who are those TMF's?
Contact Us
Contact Customer Service and other Fool departments here.
Work for Fools?
Winner of the Washingtonian great places to work, and Glassdoor #1 Company to Work For 2015! Have access to all of TMF's online and email products for FREE, and be paid for your contributions to TMF! Click the link and start your Fool career.