Re: [computer-go] Computer Olympiad photo

[Don Dailey <drd@xxxxxxx>]

> From the last 3 tournaments diep played, the DECISIVE errors not long after
> diep had left book, they took place at depths of around 10-13 ply, when i
> searched them at 24 hours a move, so getting depths of up to 20 ply, the
> vaste majority of moves still were played. Only 1 of the moves another 2
> ply was helping (i simply got too small of a search depth with my slow
> knowledge oriented chessprogram which is not so selective like most top
> programs are).
> 
> So where overwhelming brute force power would fix 1 move,
> all of the errors i could easily fix by some improvements 
> in evaluation function.


This is a very misleading use of statistics.  Let me point out the problem:

  1. Your program  already has  a reasonably well  balanced evaluation
     function compared to most chess  programs.  That means it is hard
     to  make improvements.  Even  if you  hand coded  some evaluation
     fixes  to make  the program  play all  the right  moves  where it
     failed,  the actual  improvement  to the  program  would be  very
     negligible.

     Usually, changes to fix  problems create other problems.  That is
     because chess  evaluation is hard  and we code up  heurstics that
     are based on  general principles, not hard facts.   Once you have
     achieved a  reasonable balance you  reach a point  of diminishing
     returns  where   it's  simply  very   hard  to  make   even  tiny
     improvements.   Each heuristic  interacts with  other heuristics,
     and it's a big mess.

     If you add new ideas to the evaluation function, you slow it down
     a little.   It becomes very  strong in dealing with  the problems
     you  just fixed, but  you have  effectively weakened  the program
     just a  little in  every other position  because you  have slowed
     down the search.   I once made a number  of very clear positional
     improvements  to my  chess  program and  discovered  that it  was
     playing weaker.  After a lot of study, I learned that the program
     was  now 1/2 the  speed of  the previous  version and  was losing
     tactically.   In reality  all  the improvements  affected only  a
     fraction of all the positions it was required to search.

  2. It  doesn't take  a  lot of  different  moves to  make a  program
     stronger.   You mentioned  3  tournaments, but  Diep didn't  lose
     every game, it is well above average strength of tournament chess
     programs.   So  it probably  won  most  of  it's games.   We  are
     interested only in the few games  it lost (or draw when a win was
     called  for.)  But  an extra  win out  of a  very few  games will
     register as a substantial  rating improvement.  At any rate, this
     is a pathetically small sample to base conclusions on.

     I can make my program play  any move I want by tinkering with the
     evaluation function.  But can I  do this without affecting ANY of
     the other moves it played negatively?

My own experience in computer  chess tournaments is different.  I have
won  my share  of  tournaments  including one  Dutch  open where  your
program nearly beat  mine.  One thing I always notice  is how often my
program NEARLY  made a losing  move but changes  near the end  of it's
thinking cycle.  This happens so often that I am always thankful I had
a little bit  faster computer.  I can't help but  believe that with an
even faster computer, I would have salvaged a few losses.

On the  other hand, some of our  bad moves would have  been fixed with
evaluation tuning or adjustments.  We  played a terrible move one year
against KING and  lost miserably in a slaughter.   That one move could
have been fixed  with a small evaluation change, OR  a small amount of
extra  thinking  time.   Extra   thinking  time  is  a  general  cure,
evaluation is a specific cure.

I think chess  programs are so strong now that games  are won and lost
almost by accident.  If your  program couldn't see the winning plan in
20  ply, chances  are the  other program  didn't know  it  was winning
either.  YOU  obviously realized the move  was losing and if  a 20 ply
search didn't  reveal this I think  it proves the point  I have always
tried to make which is that computers still suck at chess.

In another computer game we  played, our program had to choose between
2 completely different plans.   The plans were radically different and
led  to  radically  different   kinds  of  games.   The  program  kept
alternating between the 2 moves.  It  settled on the plan that let the
opponent have a big juicy center, but giving us 2 connected and passed
pawns  but which  were not  very well  advanced.  The  only difference
between these  plans was  a point or  two in the  evaluation function.
Neither me nor the computer had  any way of really knowing whether the
plan  was  good  because  there   was  no  immediate  danger  and  the
consequences  were far  off into  the future.   This is  so  common in
computer chess that I laugh when people say all you need in chess is a
simple evaluation that counts heads  and deep search.  They just don't
have a  clue.  As it turns out,  we won that game  probably because we
signficantly outsearched our opponent.   Not because we made the right
decision at that point.  In fact, I suspect we made the wrong decision
because our pieces were really tied down after losing the center.

Don
_______________________________________________
computer-go mailing list
computer-go@xxxxxxxxxxxxxxxxx
http://computer-go.org/mailman/listinfo/computer-go