Message Font: Serif | Sans-Serif
UnThreaded | Threaded | Whole Thread (14) | Ignore Thread Prev | Next
Author: TMFKnave Three stars, 500 posts Old School Fool CAPS All Star Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: of 8173  
Subject: Change in accuracy ranking methodology Date: 9/11/2006 12:39 PM
Post New | Post Reply | Reply Later | Create Poll Report this Post | Recommend it!
Recommendations: 23
Thanks to several users on this board such as Eldrehad, LoneIguana, and HelicalZz, for helping us devise a new accuracy ranking.

Here's what we plan to implement at public launch:

Players are ranked in terms of accuracy by looking at comparing the probabilities of achieving their respective accuracy ratings.

Before getting into the tricky math, here's the problem we've solved.

In the current BETA version, players are ranked for accuracy solely on the percentage right they have. So a player with 7 out of 7 correct will be ranked ahead of someone who has 99 out of 100. This seems absurd.

So, now, we look at the probabilities of getting X correct and compare those numbers. And the probability of getting 7 out of 7 is about 1% (assuming around a 50% success chance), while the probability of getting 99 out of 100 right is virtually 0%. So, if a player managed to actually make 99 out of 100 correct, he would pretty much be guaranteed to be the most accurate player in CAPS.

99 out of 100 is far too extreme and would likely never happen. More realistically, in our new method, a player who has picked 70 correct out of 100 will still beat a player who has 7 of 7, for example.

That's the gist of it, and now the geeky math stuff follows:

Let's compare players who have 7 picks versus players who have 20 picks.

First, we look at the probability of getting X correct out of 7, which uses the binomial formula (P = probability of) :

P(X correct) = P(correct pick)^(# Correct Picks) * P incorrect pick)^(#Incorrect Picks) * Number of Ways to get X of 7 picks.

This yields the following:

Picks Correct Prob of X correct
7 0 0.781%
7 1 5.469%
7 2 16.406%
7 3 27.344%
7 4 27.344%
7 5 16.406%
7 6 5.469%
7 7 0.781%

Now because it's better to get more correct, we need to make this cumulative.

Picks Correct Prob of X or MORE correct
7 0 100.00%
7 1 99.22%
7 2 93.75%
7 3 77.34%
7 4 50.00%
7 5 22.66%
7 6 6.25%
7 7 0.781%

So, the probability of getting 5 or more correct is sum of all the probabilities with more correct, or P(5 or more) = P(5)+P(6)+P(7)

This assumes that the probability of making a correct pick is 50%. We y may bump this number a little higher or even base it on the actual average accuracy % of CAPS players, which as of last week was around 57%.

Now, let's look at the table for someone with 20 picks:

Picks Correct Prob of X or MORE correct
20 0 100.0000000%
20 1 99.9999046%
20 2 99.9979973%
20 3 99.9798775%
20 4 99.8711586%
20 5 99.4091034%
20 6 97.9305267%
20 7 94.2340851%
20 8 86.8412018%
20 9 74.8277664%
20 10 58.8098526%
20 11 41.1901474%
20 12 25.1722336%
20 13 13.1587982%
20 14 5.7659149%
20 15 2.0694733%
20 16 0.5908966%
20 17 0.1288414%
20 18 0.0201225%
20 19 0.0020027%
20 20 0.0000954%

So, look at our player who has 16 of 20 correct. The likelihood of achieving 16 or more correct is .59%, which is less likely than achieving 7 of 7 (probability of .78%), so the 16 of 20 player would be ranked higher in terms of accuracy.

With this change implemented on our test system, we see players with lots of picks and very high scores like TMF BreakerCharly and TMFEldrehad (who have compiled accuracies of hovering atound 69%) jump back up to our top spots.

We are happy with this (not because they are TMFers... grin), but because it brings the players that are most frequently adding to our community intelligence higher up in the rankings. However, the players with few picks can still do well, but they have a much lower margin for error.

Comments are welcome.

Post New | Post Reply | Reply Later | Create Poll Report this Post | Recommend it!
Print the post  
UnThreaded | Threaded | Whole Thread (14) | Ignore Thread Prev | Next


Kiddie Card Board
Should kids have their own credit card?
When Life Gives You Lemons
We all have had hardships and made poor decisions. The important thing is how we respond and grow. Read the story of a Fool who started from nothing, and looks to gain everything.
Post of the Day:
Berkshire Hathaway

Taking Candy From Strangers
What was Your Dumbest Investment?
Share it with us -- and learn from others' stories of flubs.
Community Home
Speak Your Mind, Start Your Blog, Rate Your Stocks

Community Team Fools - who are those TMF's?
Contact Us
Contact Customer Service and other Fool departments here.
Work for Fools?
Winner of the Washingtonian great places to work, and Glassdoor #1 Company to Work For 2015! Have access to all of TMF's online and email products for FREE, and be paid for your contributions to TMF! Click the link and start your Fool career.