Skip to main content
No. of Recommendations: 2
Kinda hard to argue with backtests.

The argument against backtests is data mining. If you test enough strategies with enough tunable parameters you will get something that matches the past but has no skill in predicting the future. It doesn't matter if the person developing the strategy did all this testing. Imagine a million people each testing one strategy. If one looks great they will think that the first strategy they tested was successful and since that is incredibly unlikely by random chance, it must have something to it.

Here is a silly illustration. I believe that people buy stocks based on whether the letters in the stock symbol seem exciting and profitable. Because more people buy these stocks, they perform better. It has nothing to do with the companies themselves, it is just the human psychology of letters. I did a backtest over the past 10 years, testing all 17,576 three letter combinations and found that purchasing equal amounts of all stocks whose symbols begin with the letters AAP resulted in a 53% CAGR over ten years.

This strategy is unconvincing despite the wonderful backtest. The theory behind it is nonsense. And there are too many tunable parameters, the 17,576 three letter combinations.

Faber's strategy looks more promising for four reasons:

- It is simple and has few tunable parameters.

- It makes sense at a fundamental level.

- It was re-evaluated twice after being first proposed in 2006.

- It has a real person who has successfully used it, assuming of course that Ray is real person telling us the truth and not a bot written by Faber in 1997 in a devious 16 year strategy to sell his timing system.

The reason to reject or accept a strategy is more complex than just the fact of backtesting.

And here is xkcd's take on the issue:
Print the post  


The Retirement Investing Board
This is the board for all discussions related to Investing for and during retirement. To keep the board relevant and Foolish to everyone, please avoid making any posts pertaining to political partisanship. Fool on and Retire on!
What was Your Dumbest Investment?
Share it with us -- and learn from others' stories of flubs.
When Life Gives You Lemons
We all have had hardships and made poor decisions. The important thing is how we respond and grow. Read the story of a Fool who started from nothing, and looks to gain everything.
Contact Us
Contact Customer Service and other Fool departments here.
Work for Fools?
Winner of the Washingtonian great places to work, and Glassdoor #1 Company to Work For 2015! Have access to all of TMF's online and email products for FREE, and be paid for your contributions to TMF! Click the link and start your Fool career.