Neural nets
Back-propagation algorithm

One of the most used algorithms for neural network learning is the algorithm of back propagation. After twenty years of relative stagnation of neural networks, this algorithm started a new wave of research in this scientific field.

This algorithm is suitable for learning of multi-layer networks with feed forward propagation. The learning is of a supervised type. It is an iteration algorithm according to the method of computation. The energetic function is minimized based on gradient; therefore it is sometimes classified as a gradient learning method.

The principle of back propagation algorithm is similar to common learning method used in education. If a student finds deficiency in the schoolwork, he has to study more. This is similar in back propagation. Neural network can be considered as a student, testing may be considered as a mechanism for testing, whether the neural network answers to the input vectors according to a training set. Training set is a schoolwork that the student has to study. If we find out that the network does not respond correctly, we must change its weight coefficients, until it starts to react correctly.

Fig. 7.9: Graphical representation of neural network learning

Back propagation is an iteration process, wherein we proceed from an initial state to the state of full knowledge.

The choice of weights from the training set does not need to be sequential, very often we use e.g. random order. The choice of strategy is not given strictly, it rather belongs to a category "art of learning neural networks". In the course of learning, we may make some patterns more frequent, some less etc. A suitable strategy may significantly influence the speed and success of learning. A global error in the algorithm can be represented by e.g. mean square error calculated over all patterns from the training set. This error represents the extent of knowledge of the network. The criterion is a limit, where the learning process is stopped. The algorithm of back propagation is based on minimization of energy of a neural network. Energy represents the extent of knowledge, i.e. deviation between real and required values of outputs of a neural network for given training set. It substantially the same as the global error, but for calculation of the global error, we use rather statistic criteria.

For a neural network with back propagation, the energy function is defined as follows:

Where in d is the number of inputs, yi is i-th output and di is i-th required output. This function is nothing but a sum of squares of deviations. The back propagation learning algorithm has to be considered as an optimization method, which is able to find correct weights and thresholds for a given neural network.