No. of Recommendations: 3
This flashes my curve fitting warning light.

I agree that this process adds a bit more room for curve fitting, but there aren't
really a lot of degrees of freedom, since this is (as I understand it)
entirely a step-forward test. At each month you're using only screen
performance data for periods prior to that date. This doesn't increase
the degree to which the blend is overfit, because it isn't an
optimization on the whole test era. (two master blends, one bear and
one bull, created with the optimizer and hindsight would definitely have that problem).
Instead, subdividing the past and doing separate bull and bear blends
is really reducing the statistical support for the blends which are
created at each trading period. They will fit their smaller training
sets better, but that doesn't matter, as the results don't use those
numbers, they use the results in the period after that training.
You would expect that to reduce their predictive power. But, in this
case, it seems there was still enough information to create useful
blends for each case, as the fits were predictive enough to give good results.

The main degrees of freedom rest only in (a) the choice and
tuning of the bull/bear signal, and (b) the sort metric used to pick
the screens at each period, which implicitly includes things like the
lookback period for the screen results, the screen depth to use, and the
number of screens to use. That doesn't really seem like a whole lot to me.
To me it seems that by far the biggest issue is that when you're looking
at "only" the results up to 1998, you're using a screen whose 1999
results were already taken into account when the screen was being tuned.

FWIW, as for the timing signal, which is the other big tuning risk,
I have duplicated a signal almost identical to Zee's, and it has
nothing but a lookback length and a single cutoff value, so it's not a
particularly overwrought timing model as these things go.

So, as always I'm on the side of the dubious, but this procses doesn't
seem particularly overfit to me. Obviously I wouldn't expect the same
results going forward because the screens themselves are mostly in their
backtest periods, but the screen selection process seems very sensible to me.

Print the post  


Live Video Event Monday!
The GP team is hosting a live video event on Monday at 4 p.m. ET. Don't worry if you can't make it — we'll have a replay and a transcript. Click for more!
When Life Gives You Lemons
We all have had hardships and made poor decisions. The important thing is how we respond and grow. Read the story of a Fool who started from nothing, and looks to gain everything.
Contact Us
Contact Customer Service and other Fool departments here.
Work for Fools?
Winner of the Washingtonian great places to work, and Glassdoor #1 Company to Work For 2015! Have access to all of TMF's online and email products for FREE, and be paid for your contributions to TMF! Click the link and start your Fool career.