Re: [computer-go] Weights are very important

[William Harold Newman <william.newman@xxxxxxxxxxx>]

On Tue, Jan 13, 2004 at 04:38:41AM +0100, Vincent Diepeveen wrote:
> At 20:06 12-1-2004 +0100, Nicol N. Schraudolph wrote:
> >On Mon, 12 Jan 2004, Vincent Diepeveen wrote:
> >
> >> Yet a NN when trained will pick some random value from between -36000 and
> >> +36000 and start tuning with that.
> >> 
> >> As a human we know very well that it should be something between 1 and 40.
> >
> >If you know a priori that the value should be between 1 and 40, why
> >on earth would you not initialize your neural net within that range
> >and let learning refine it from there?  When I seek optima in a design
> 
> For a very simple reason. If you believe that NN's work, it should figure
> that out itself of course. If you don't believe in it, you don't use them
> at all.

"of course"? It doesn't sound obvious to me.

I spent a number of years working in theoretical chemistry, doing
numerical simulations (mostly Monte Carlo simulations) of quantum
mechanics. There was no serious question of "believing that [the
numerical techniques] worked" -- of course they worked. But they were
very computationally expensive, often unaffordably expensive. It was
well worth figuring out how to put in as much analytical information
as possible before you started trying to beat the problem to death
with your computer. As a trivial example of this sort of thing, try
numerically integrating exp(P(abs(x-6/17))), for some polynomial P,
and some interval in x which includes the singularity at 6/17. If you
don't tell your integrator about the singularity, it will still
probably converge to the correct answer, but much more slowly than you
can make the calculation converge by using a smart integration
algorithm and making sure it's in some sense forewarned about the
specialness of the point x=6/17.

Machine learning is an optimization problem rather than a
simulation/calculation problem, and I have less experience with such
problems, but I have *some* experience, and I've read a fair amount
about such problems. In many hard optimization problems it's very
helpful to apply prior knowledge about the general properties of the
solution. The kinds of prior knowledge that you want to have are
somewhat different than what you want in problems similar to
integration; in particular, almost any optimization method will almost
certainly appreciate it very very much if it doesn't need to tunnel
out of too many local minima in order to find the global minimum you
care about. One good and classic way to help is to give it the kind of
information that Nicol Schraudolph described.

> If you believe in it, giving it the accurate range, without the chance to
> also put a parameter at 0, then you are later fooling yourself when you
> claim it works.

Do you believe that if people who figure out efficient orbits for
space missions give their programs initial hints then they are later
fooling themselves when they says the programs work, that they
shouldn't give hints if they believe in the programs, and that if they
don't believe in the programs they shouldn't use the programs at all?

-- 
William Harold Newman <william.newman@xxxxxxxxxxxxxxxxx>
In examining the tasks of software development versus software maintenance,
most of the tasks are the same -- except for the additional maintenance
task of "understanding the existing product". -- Robert L. Glass, _Facts
and Fallacies of Software Engineering_
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C  B9 25 FB EE E0 C3 E5 7C
_______________________________________________
computer-go mailing list
computer-go@xxxxxxxxxxxxxxxxx
http://computer-go.org/mailman/listinfo/computer-go