But algorithms cannot be random. Given an initial weight, and the algorithm, (and the other information needed to compute the algorithm) it is plain to see what the newly computed weight will be. By induction, the weights can be predicted arbitrarily far into the future.
But you are missing the biggest piece. Say I'm designing a supervised ANN with two output neurons, one hidden layer composed over half a dozen neurons, and an input layer composed of roughly the same amount (this is a very simple set up). I want it to recognize rom letters. Not just letters like those one would see typed, but hand-written as well, or letters which come close to the shape they are supposed to but were written by a child. The initial weights and the algorithms only provide a starting place. As soon as I began to "train" the network, some ANNs begin to alter in unpredictable ways (which doesn't make them non-deterministic). For example, say I input something that looks like te letter "H." Initially, let's say certain "paths" to the output neurons result in one output neuron firing (a "yes") and another not. At this point, everything is pretty-straightforward, but even now I would have to do a fair amount of work to figure out how the network would respond to that input.
This is where things start to get more complex. I tell the network that the output which fired "yes" was correct. The network uses the algorithm to determine which neurons led it to the correct answer. But this isn't a binary process. It isn't that some neurons were right and others were wrong, or that some weights were right and others were wrong. Just that certain weights were involved, to varying degrees (some were weak, others strong, and everywhere in between). So the network uses the algorithms to adjust the weights across the network based on the weights that led it to the correct response and those that did not. Again, however, some weights that were involved in the correct response didn't actually "help" and vice versa. Now I show it a square. Theoretically, the network will tend to have more connections between input neurons and the hidden layer with lower weights, so it is more likely that the output neurons won't fire. But once again, a lot of the adjustment the network made was out of error. Whatever the case, I continue to tell the network when an output neuron fires correctly, and it continues to adjust based on my responses.
One part of the complexity of even this simple network is the input. Squares have a lot of straight lines, like an "H" or and "L" or an "N" but not so much an "n" and certainly not an "O." So weights which tend to lead a neuron to fire for an "H' might also do the same for a square, but not for an "O." It takes a lot of training for the neuron to "remember" the patterns of weights which allowed it to distinguish between an "H" and a square but also recognize an "O." It doesn't take a lot of time before the network has changed its weights in unpredictable ways.
Say I give the network an "H," an "A," and a "V." The networks memory (the weights) will be attuned to that type of shape, and with a fair amount of work I can predict, if I know the EXACT shape of the letters I show (because we aren't talking merely about typed letters), exactly how the weights will change. If I then show the network an "O," most likely both output neurons will fail to fire. I then tell the network they are both wrong. This is tricky, because a lot of the weights which led to the wrong answer were correct before, and the network remembers this. This is where certain parameters in the algorithm become really important: if the network adjusts too fast, then showing it an "O" here is basically like going back to ground zero.
Instead, the network will probably try to select weights which contributed less to the correct answers before, along with some which contributed less to wrong answers. Now it won't be as good at recognizing an "A," but will be better prepared for a "g" or a "C."
Again, this is a simple network, but it is a dynamical system, so small perturbations can mean a lot. Make an "H" a little more curvy, or an "O" a little more like a square, and the whole network adjusts its memory.
Yes, but it is still computable. It will remain computable in principle regardless of the complexity of the network.
One issue is that that the algorithm doesn't specify HOW the weights will change given stimuli 1, 2, 3,...n, only how it adapts, remembers, and learns. Given a sufficiently complex ANN, predicting the evolution of the network is a bit like predicting the weather. Even knowing how it will change from simulus 15 to 16 and why becomes hard or impossible. One of the big areas of work in this field is coming up with tools which allow the programmer to figure out how certain answers were achieved, not just predicting future ones (although the two issues are very related).
Certainly in a software simulation of a neural network, they have to; they are programs. In hardware, I am relatively sure that there will still be a well-defined ordering of events, it will just be harder to track.
A basic artificial neuron has a threshold value (some nonlinear function) and is connected to other neurons by a series of weights. The neuron fires (perhaps sending information to a series of neurons, perhaps to an output neuron) if the weights cause the neuron to reach its threshold, it fires (given different set-ups, like feedforward vs. backpropagation, this may alter connections only upward, or also backward). Now, I can write the program so that it tells me whether or not each neuron fired and what the weights were, but tracking isn't the issue. Interpreting is much more difficult. Knowing that a neuron fired is pretty useless. The eventual output relies on the networks memory, or certain weight patterns, and I can only know how the weight pattern changes from one state from another after it has done so. Tracking the firing of each neuron won't enable me to do that.
Also it is important to note that the functions which determine whether a neuron fires and (sometimes) what that means are nonlinear functions in
n-dimensional space. That means the behavior of a given neuron is complex enough. Determining how a series of multi-dimensional nonlinear functions interact in response to a stimulus resulting in a final pattern of weight distributions is a nightmare, and proceeding "step-by-step" either defeats the whole process (it becomes way too slow) and/or is useless, because I can't tell, given that neurons 3, 5,6, & 7 fired on
x hidden layer but the others didn't, what that really means.
The output is the pattern as a whole, and tracking it often just provides you with useless information. In order to determine what caused the network to respond the way it did involves a series of differential equations, not tracking.
Only in the event that the machine runs for an infinitely long time, which is impossible as mentioned.
The point of Turing's original paper was to prove that a procedure was possible to determine the truth value of any mathematical proposition. The proof (which showed that this was impossible) relied on infinity. But yes, one can build a turing machine with finite tape. It's been done.
They can't be computed by Turing machines? So how do software simulations of neural nets work? Modern CPUs are equivalent to Turing machines.
Modern CPUs aren't exactly equivalent to Turing machines, but I think I see your point. With some exceptions, most neural networks only simultate the "massive parallelism" of the brain. However, increasingly a number of researchers aren't running ANNs on a CPU, but are implementing ANN algorithms using
truly parallel ANNs. In other words, they are using parallel hardware (for a simple and straightforward example, see
http://cactuscode.org/media/news/ncur20/Cactus_WesleySmith06.pdf).
However, even when only simulating parallism, again the the linear nature computing may allow one to runn ANN algorithms on an actual turing maching (tape and all), but the problem is the same as running it on a CPU. Being able to "track" changes simply doesn't give you the information you need to understand whether the system is heading.
