The Motley Fool Discussion Boards

Previous Page

Investing/Strategies / Retirement Investing

URL:  https://boards.fool.com/kinda-hard-to-argue-with-backtests-the-argument-30890353.aspx

Subject:  Re: Community Question Date:  9/26/2013  11:40 PM
Author:  spinning Number:  73346 of 100395

Kinda hard to argue with backtests.

The argument against backtests is data mining. If you test enough strategies with enough tunable parameters you will get something that matches the past but has no skill in predicting the future. It doesn't matter if the person developing the strategy did all this testing. Imagine a million people each testing one strategy. If one looks great they will think that the first strategy they tested was successful and since that is incredibly unlikely by random chance, it must have something to it.

Here is a silly illustration. I believe that people buy stocks based on whether the letters in the stock symbol seem exciting and profitable. Because more people buy these stocks, they perform better. It has nothing to do with the companies themselves, it is just the human psychology of letters. I did a backtest over the past 10 years, testing all 17,576 three letter combinations and found that purchasing equal amounts of all stocks whose symbols begin with the letters AAP resulted in a 53% CAGR over ten years.

This strategy is unconvincing despite the wonderful backtest. The theory behind it is nonsense. And there are too many tunable parameters, the 17,576 three letter combinations.

Faber's strategy looks more promising for four reasons:

- It is simple and has few tunable parameters.

- It makes sense at a fundamental level.

- It was re-evaluated twice after being first proposed in 2006.

- It has a real person who has successfully used it, assuming of course that Ray is real person telling us the truth and not a bot written by Faber in 1997 in a devious 16 year strategy to sell his timing system.

The reason to reject or accept a strategy is more complex than just the fact of backtesting.

And here is xkcd's take on the issue:
http://xkcd.com/882/
Copyright 1996-2020 trademark and the "Fool" logo is a trademark of The Motley Fool, Inc. Contact Us