Neural nets
Perceptron

First of all, we will introduce the simplest model consisting of a single neuron, usually called perceptron. There are n connections leading to the neuron that represent either outputs of other neurons or stimuli from the exterior. Each of the input carries an information xi in the form of a real number every moment. The numbers represents values of some attributes; if we consider input information from the surrounding world – input space – the entire vector x = [x1‚ ..., xn] will characterize certain object of study. The attributes may be represented by data of temperature, pressure, color, Boolean, etc.

Fig. 7.7: Scheme of neuron with weights of respective connections

Every connection leading to the neuron is characterized by another real number wi which represents a so called synaptic weight (importance) of a connection, and every neuron is characterized by a threshold θ. A weighted sum ξ = ∑ wi xi – υ (i= 1, ...‚ n) represents the overall stimulus, so called potential of a neuron. The neuron reacts on this potential by an output response z = S(ξ), wherein S is a predetermined non-linear transfer function, usually in the form of a sigmoid (monotonously rising between two asymptotic values, e.g. 0 and 1, with largest derivative in the point 0).

Networks with multiple neurons may be classified according to various aspects; we will concern on two of them. The first one is given by network topology, the other one by mode of operation. According to the first aspect, we may introduce recurrent networks (their graph contains cycles, i.e. outputs of some neurons are introduced back to the network as stimuli) and other networks, wherein multilayer networks are most important; they have neurons divided into layers, where the outputs of neurons of one layer are inputs of all neurons proximate to the "higher" layer; there is no other connection between neurons of such network.

## Perceptron paradigm

Perceptron was invented by F. Rosenblatt in 1958. He was inspired by the model of human eye and tried to develop its model. He proceeded from the fact that retina comprises light sensitive sensors arranged into a matrix. Their outputs lead to specialized cells, so called daemons, genetically predetermined to recognize specific types of patterns. Outputs of daemons are further processed in cells with threshold behavior, so their output is active from a certain level of input stimulus.

Fig. 7.8: Recognition with perceptron

Perceptron network is also three-layered, according to its physiological model. The input layer works as a compensating or branching layer. Its objective is to map a two-dimensional field of sensors onto a one-dimensional vector of processor elements. This second layer is formed by stimuli detectors. Each of them is randomly connected to elements of the input layer. The weights assigned to inputs are constant. The objective of the daemons is to detect specific stimuli. The last, third layer contains pattern recognizers (or perceptron). While weights of the first and second layer are fixed, the weights of the inputs of the output layers may be set up during training. For learning of a perceptron network, Rosenblatt suggested a so called perceptron learning algorithm.

The elements differ according to layers. The neurons have one input (threshold) fixed to a constant of 1. Other inputs are randomly connected to daemon outputs of the middle layer and their weights may be set up. The transfer characteristic of a processor element of a perceptron layer is as follows: the output is zero if the weighted sum of all its inputs is zero or negative. If not, the output is one. Other non-linear transfer function is sometimes used. In such a case, if the weighted sum of all its inputs is zero or negative, the output is -1, otherwise +1.

The weights are set up randomly. If the output is correct, the weights do not change. If the output has to be 1, but it is 0, weights on all active inputs are incremented. If the output has to be 0, but it is 1, weights on all active inputs are decremented.

The inputs are active if their value is > 0. The value of weight change depends on chosen option.

- Fixed increments or decrements are always applied.
- The increments (decrements) vary according to the value of error. It is advantageous to have larger increments in the case of large errors and vice versa. However, such acceleration of convergence may result in learning instability.
- Fixed and variable increments (decrements) are combined with respect to the value of error.

Besides the classic perceptron that has just been introduced, there are other types of perceptron – Minsky and Papert (MP-perceptron) and Adaline (or Madaline) by Widrow.