No. of Recommendations: 1
Sharpe ratio during the most recent in-sample year is best (but still weak) predictor of out-of-sample results.

Max drawdown and standard deviation, while not predictors of out-of-sample success, were predictors of those metrics in the out-of-sample periods.

http://www.valuewalk.com/2016/04/comparing-backtest-sample-p...
Print the post Back To Top
No. of Recommendations: 3
I hit a paywall at valuewalk. Is the valuewalk posting different than the quantopian March 9, 2016 paper "All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms"?

https://www.quantopian.com/posts/q-paper-all-that-glitters-i...

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2745220

Abstract:
When automated trading strategies are developed and evaluated using backtests on historical pricing data, there exists a tendency to overfit to the past. Using a unique dataset of 888 algorithmic trading strategies developed and backtested on the Quantopian platform with at least 6 months of out-of-sample performance, we study the prevalence and impact of backtest overfitting. Specifically, we find that commonly reported backtest evaluation metrics like the Sharpe ratio offer little value in predicting out of sample performance (R² < 0.025). In contrast, higher order moments, like volatility and maximum drawdown, as well as portfolio construction features, like hedging, show significant predictive value of relevance to quantitative finance practitioners. Moreover, in line with prior theoretical considerations, we find empirical evidence of overfitting – the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance. Finally, we show that by training non-linear machine learning classifiers on a variety of features that describe backtest behavior, out-of-sample performance can be predicted at a much higher accuracy (R² = 0.17) on hold-out data compared to using linear, univariate features. A portfolio constructed on predictions on hold-out data performed significantly better out-of-sample than one constructed from algorithms with the highest backtest Sharpe ratios.
Print the post Back To Top
No. of Recommendations: 0
Paper is the same; rest was commentary.
Print the post Back To Top
No. of Recommendations: 6
I can see one major flaw in the study right off the bat.
It's "all that glisters", not "all that glitters "
It's from Shakespeare.

Not just an old fashioned spelling...they are two different words with somewhat different meanings.

OK, I'll go back to my geek cave now.

Jim
Print the post Back To Top
No. of Recommendations: 4
Shakespeare: "To be or not to be".
Sartre: "To be is to do".
Sinatra: "Do be do be do".
Print the post Back To Top