Re: [computer-go] Statistical Significance (was: SlugGo v.s. ManyFaces, new...

[David G Doshay <ddoshay@xxxxxxx>]

On Sep 8, 2004, at 9:52 AM, Harry Wang wrote:

> well, I agree that there is only one outcome - porgram
> either win or loses. However, when I look at the
> strength of two programs - such as SlugGo and GnuGO,
> variance is a good indicator to show strength of a Go
> program.
>
> given 80 games played between two programs, if I have
> one program winning 70% of the time with a small
> variance. Then I am very skeptical about the result.
> Then I play more matches using two programs - say
> another 300 games. Usually the winning percentage gets
> smaller when more games are played.
>
> With larger variance, my personal confidence level is
> higher that one program is stronger than another.
I think Harry has confused variance and winning margin.

I do think it is important to look at the scores, not just the
win/loss numbers or ratios. Additionally, I think it is very
important to pay attention to large variances in the scores.
When the winning margins are small and the variance is
large I want to see many more games. In these 4 stone
games we have wins by 91 and 76, and losses by 107 and
93. It can be argued what this means, but I think we have
to pay attention to it.

Here is a table of data for our games against Many Faces,
counting SlugGo's score as positive and MFG's as negative:

h = 1 	25 games
SlugGo 20, MFG 5
mean = 32.5 	s.d. = 45.3

h = 2		42 games
SlugGo 31, MFG 11
mean = 18.9 	s.d. = 34.3

h = 3		31 games
SlugGo 15, MFG 16
mean = -13.5 	s.d. = 55.4

h = 4 	45 games
SlugGo 23, MFG 21, one tie
mean = 3.25 	s.d. = 47.1

h = 5 	6 games
SlugGo 2, MFG 4
mean = -6.8 	s.d. = 21.7

h = 6 	5 games
SlugGo 2, MFG 3
mean = 2.6 	s.d = 22.8


Cheers,
David

_______________________________________________
computer-go mailing list
computer-go@xxxxxxxxxxxxxxxxx
http://www.computer-go.org/mailman/listinfo/computer-go/