Re: [computer-go] Computer Go Test Collection 2.0

[Gunnar Farnebäck <gunnar@xxxxxxxxxxxxxx>]

Markus wrote:
> Version 2.0 of the Computer Go Test Collection has been released.
> [...]
> We welcome feedback on the test set: modifications and additions to the test
> cases, alternative good (or bad) moves, and comments about which types of
> tests are most useful.

I won't comment too much on this specific collection. I have looked at
it quickly but for GNU Go it is somewhat redundant since we already
have our own very big test collection. Let's note however that the
format used in this collection is exactly the same as GNU Go is using
and that the analysis tool in GoGui is interchangeable with the tools
in GNU Go.

If you don't already have a testing framework in place for your
engine, I warmly recommend using this one. For GNU Go it has been
invaluable and pretty much a prerequisite for the strength increase
over the last four years.

When it comes to which types of tests are *most* useful, my experience
is that those can be found in two categories:
1. Systematic and serious errors your own engine insists on doing.
2. Systematic collections of specific types of problems, preferrably
   in artificial settings without any distractions.

The reason why I prefer artificial settings over real-game positions
in the latter category is that it allows you to concentrate on a
specific part of the engine and makes debugging/tuning as efficient as
possible. They also tend to be faster to run, which does become an
issue once your test suite becomes big. The reason for a systematic
collection is that you can get a good idea of what is needed to solve
a whole class of problems by analyzing the failures in many more or
less closely related situations.

The main difficulty with category 2 is that it takes a lot of effort
to create such collections. It is much easier to identify mistakes in
games your engine is playing and adding those to the test suite. In
GNU Go the following two test files are (mostly) of category 2:

ld_owl.tst:
Life and death tests, mostly basic corner shapes, including a big
section with variations on the tripod shape. A drawback of this
collection is that it uses GNU Go specific GTP commands to query life
and death status of groups, which are not necessarily easy to emulate
in other engines.

seki.tst:
Corner formations which can be invaded to make seki, plus followup
moves. Uses only standardized commands except for a few tests at the
end.

You can find those files in the regression directory of e.g. the GNU
Go 3.5.8 distribution.

I know that there is ongoing work (outside of GNU Go) on a semeai test
suite of this kind, which I hope will be released to the public. If
somebody knows of other collections of a similar nature, I would be
very interested in seeing them.

/Gunnar
_______________________________________________
computer-go mailing list
computer-go@xxxxxxxxxxxxxxxxx
http://www.computer-go.org/mailman/listinfo/computer-go/