[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[computer-go] Statistical Significance (was: SlugGo v.s. Many Faces, newest data)

On Sep 6, 2004, at 8:38 PM, Douglas Ridgway wrote:

On Fri, 3 Sep 2004, David G Doshay wrote:
I found it interesting that Many Faces won 10 of the first 13, and then
SlugGo won 11 of the next 14. This was what I meant in an earlier
email when I wrote about how easy it is to be fooled by small samples.
I wrote something for gnugo-devel on March 2, 2004 on essentially this.
Fortunately, neither 10/13 nor 11/14 is quite long enough to meet a 5%
test, so even if you got only one of these streaks then stopped, standard
statistics avoids being misled. My brain wants to draw an inference long
earlier, however.
The paper "Statistical Significance of a Match" by Rémi Coulom, posted
here on 4 Sept and available at
http://remi.coulom.free.fr/WhoIsBest.zip
contains a table claiming that with 3 losses and 10 wins the confidence
level is 97% that the winner of 10 is "better." This result is based upon
a formula from Bayes. I am not enough of a statistician to know.

So many statistics, so little ... (must be something good to put here)

It looks to me like we would need at least 200 games to know what
is going on at 4 stones. All of these games involved huge groups and
large strings of cut off stones.
Even 200 might not be enough. I estimated a sample size of 260 or so to
show a difference for a player who wins 60% (see the Mar 2 email), which
is at the outside of the likely differential at 4 stones. For 3 or 5
stones, it'd be okay.
His table also claims 97% Cl at 40 losses and 59 wins, essentially the 60%
you mention.

I must say that the swings I see lead me to think that the Bayesian formula
is too optimistic for this Go playing application, and that Doug's Cl's at least
feel about right.

Rémi, do you have any further thoughts?

Thanks,
David


_______________________________________________
computer-go mailing list
computer-go@xxxxxxxxxxxxxxxxx
http://www.computer-go.org/mailman/listinfo/computer-go/