Algorithm for Finding Good, Reliable Players

Question

I've the following players, each value corresponds to a result in percentage of right answers in a given game.

$players = array
(
    'A' => array(0, 0, 0, 0),
    'B' => array(50, 50, 0, 0),
    'C' => array(50, 50, 50, 50),
    'D' => array(75, 90, 100, 25),
    'E' => array(50, 50, 50, 50),
    'F' => array(100, 100, 0, 0),
    'G' => array(100, 100, 100, 100),
);

I want to be able to pick up the best players but I also want to take into account how reliable a player is (less entropy = more reliable), so far I've come up with the following formula:

average - standard_deviation / 2

However I'm not sure if this is a optimal formula and I would like to hear your thoughts on this. I've been thinking some more on this problem and I've come up with a slightly different formula, here it is the revised version:

average - standard_deviation / # of bets

This result would then be weighted for the next upcoming vote, so for instance a new bet from player C would only count as half a bet.

I can't go into specifics here but this is a project related with the Wisdom of Crowds theory and the Delphi method and my goal is to predict as best as possible the next results weighting past bets from several players.

I appreciate all input, thanks.

Answer 1

Check out http://blog.stackoverflow.com/2009/10/alternate-sorting-orders/

The formula in there is to sort voting, but if you consider the score to be similar to voting (0-whatever) you should be able to use it to calculate which players are more consistently scoring higher.

Answer 2

Have you considered just using the median? It's considered a more robust statistic (less affected by outliers) than the mean. In your data, you get medians of: 0, 25, 50, 82.5, 50, 50, 100.

Does that seem to be what you intuitively want? I agree with others that there's no "right answer" here.