The Motley Fool Discussion Boards

Previous Page

Investing/Strategies / Mechanical Investing

URL:  http://boards.fool.com/this-flashes-my-curve-fitting-warning-light-i-26843500.aspx

Subject:  Re: Blending at a Whole New Level Date:  7/24/2008  6:16 PM
Author:  mungofitch Number:  211475 of 253076

This flashes my curve fitting warning light.

I agree that this process adds a bit more room for curve fitting, but there aren't
really a lot of degrees of freedom, since this is (as I understand it)
entirely a step-forward test. At each month you're using only screen
performance data for periods prior to that date. This doesn't increase
the degree to which the blend is overfit, because it isn't an
optimization on the whole test era. (two master blends, one bear and
one bull, created with the optimizer and hindsight would definitely have that problem).
Instead, subdividing the past and doing separate bull and bear blends
is really reducing the statistical support for the blends which are
created at each trading period. They will fit their smaller training
sets better, but that doesn't matter, as the results don't use those
numbers, they use the results in the period after that training.
You would expect that to reduce their predictive power. But, in this
case, it seems there was still enough information to create useful
blends for each case, as the fits were predictive enough to give good results.

The main degrees of freedom rest only in (a) the choice and
tuning of the bull/bear signal, and (b) the sort metric used to pick
the screens at each period, which implicitly includes things like the
lookback period for the screen results, the screen depth to use, and the
number of screens to use. That doesn't really seem like a whole lot to me.
To me it seems that by far the biggest issue is that when you're looking
at "only" the results up to 1998, you're using a screen whose 1999
results were already taken into account when the screen was being tuned.

FWIW, as for the timing signal, which is the other big tuning risk,
I have duplicated a signal almost identical to Zee's, and it has
nothing but a lookback length and a single cutoff value, so it's not a
particularly overwrought timing model as these things go.

So, as always I'm on the side of the dubious, but this procses doesn't
seem particularly overfit to me. Obviously I wouldn't expect the same
results going forward because the screens themselves are mostly in their
backtest periods, but the screen selection process seems very sensible to me.

Jim
Copyright 1996-2014 trademark and the "Fool" logo is a trademark of The Motley Fool, Inc. Contact Us